A combined device scheduling optimization method based on GNN and DRL

Abstract

The application discloses a kind of based on GNN and DRL combination equipment scheduling optimization method, comprising: according to combination equipment and wafer processing information, capture the dependent relationship and processing state of combination equipment wafer processing;According to the dependent relationship and processing state of combination equipment wafer processing, construct state variable space and action variable space;According to state variable space and action variable space, construct processing state diagram, and utilize the feature of processing state diagram extracted by graph neural network GNN;To the feature of extracting processing state diagram, the feature is learned using the deep reinforcement learning DRL scheduling decision model constructed, and the trained DRL scheduling decision model is obtained by the training of a large number of processing instances;For real-time combination equipment wafer processing task, the action decision is output by the trained DRL scheduling decision model;According to action decision, combination equipment wafer processing is scheduled.The application can realize the effective scheduling of combination equipment wafer processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of semiconductor processing, in particular to a combined equipment scheduling optimization method based on GNN and DRL. BACKGROUND

[0002] With the rapid development of the electronic information industry, the semiconductor industry is experiencing rapid growth. Semiconductor manufacturing covers wafer production, processing and testing stages. In this process, combined equipment is widely used in various semiconductor manufacturing processes, including photolithography, etching and chemical vapor deposition, etc.

[0003] In the combined equipment, the wafers unloaded from the load lock go through a series of processing steps in multiple processing modules, and finally return to the load lock after completion. Usually, a batch of wafers (25) follow a predetermined flow to visit the processing modules one by one. The visiting order of each wafer is the processing route of the wafer in the combined equipment. Due to the trend of product customization and small batch wafers, cluster tools are needed to process multiple types of wafers at the same time.

[0004] Currently, the diameter of wafers has increased from 200mm to 300mm, and even to 450mm. Especially due to the rapid progress of integrated circuit manufacturing technology, the circuit width is continuously reduced to 14-22nm, and even recently to below 10nm, so that a wafer can accommodate more chips. The batch size is often significantly reduced, even to 5-8. As the diversification and individualization of consumer demand lead to the reduction of wafer batch size, this makes manufacturers want a production line to produce multiple types of wafers at the same time. However, this faces two major challenges: (a) different types of wafers require different processing routes, which requires the algorithm to have strong versatility so that different types of wafers can be combined on the same production line; (b) batch wafer production must complete the processing of all wafers. However, the current polynomial time complexity algorithm based on manual design relies heavily on human experience, and it is difficult to quickly design a new suitable polynomial time complexity algorithm when some conditions change in the face of multiple wafer types. Therefore, it is of great significance to automatically learn to solve the problem of processing multiple types of wafers in combined equipment. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art and provide a combined equipment scheduling optimization method based on GNN and DRL, which can realize effective scheduling of wafer processing in combined equipment.

[0006] To achieve the above purpose, the technical solution provided by the present application is: a combined equipment scheduling optimization method based on GNN and DRL, comprising the following steps:

[0007] 1) Capture the dependency relationship and processing state of the combined device wafer processing according to the combined device and wafer processing information;

[0008] 2) Construct a state variable space and an action variable space according to the dependency relationship and processing state of the combined device wafer processing;

[0009] 3) Construct a processing state graph according to the state variable space and the action variable space, and extract features of the processing state graph using a graph neural network GNN;

[0010] 4) Learn the features of the extracted processing state graph using a constructed deep reinforcement learning DRL scheduling decision model, and obtain a trained DRL scheduling decision model through training of a large number of processing instances;

[0011] 5) Output an action decision, i.e., a scheduling decision result, for a real-time combined device wafer processing task through the trained DRL scheduling decision model;

[0012] 6) Schedule the combined device wafer processing according to the scheduling decision result.

[0013] Further, in step 1), the dependency relationship includes a processing route γ corresponding to each wafer type and a processing module PM in the combined device corresponding to the processing route, assuming that there is a kth wafer with l processing steps, i.e., the length of the corresponding processing route is l, then the kth wafer processing route γ k is represented as follows:

[0014] γ k = [γ k,1 , γ k,2 , γ k,3 , …, γ k,l ]

[0015] In the formula, γ k,l represents the lth processing step;

[0016] Each processing step in each processing route corresponds to a processing module, γ k,l corresponds to the mth processing module PM m , so γ k,l = PM m ;

[0017] The processing state includes the occupancy of the combined device processing module and the robot, wherein the processing module can only process one wafer at a time, and when the wafer is processed and transported to the next processing module by the robot, a new wafer can be processed. The robot can only transport one wafer at a time, and after transporting one wafer, the next wafer can be transported.

[0018] Further, in step 2), the state variable space includes the number of wafers n, each wafer corresponds to a processing route length l, when constructing the matrix, each row of the matrix represents the same wafer, and each column represents a processing step in the processing route of the wafer in the corresponding row, that is, for the i-th row and the j-th column of the matrix p i,j represents the j-th processing step of the i-th wafer, where i∈n and j∈l, and the to-be-processed matrix p satisfies:

[0019]

[0020] wherein γ n,l represents the l-th processing step of the n-th wafer;

[0021] The state variable space further includes the processing time α of each processing module, the mechanical arm moving time μ, and the wafer loading and unloading time β. Each time a wafer is transported, the mechanical arm needs to move to the processing module where the wafer processing is completed, then unload the wafer, transport the wafer to the next processing module, and load it into the next processing module for processing. Then the corresponding task duration П is calculated as follows:

[0022] П = 2 · (μ + β) + α

[0023] For the matrix p, each to-be-processed wafer step corresponds to a corresponding task duration П, when constructing the matrix, each row of the matrix represents the same wafer, and each column represents the processing duration corresponding to the processing step of the wafer in the corresponding row, that is, for the i-th row and the j-th column of the matrix p i,j represents the processing duration of the j-th processing step of the i-th wafer, and the task duration matrix r is represented as:

[0024]

[0025] wherein П n,l represents the processing duration of the l-th processing step of the n-th wafer;

[0026] The action variable space includes selecting candidate to-be-processed wafer steps, and the total decision step number S satisfies S = l × n. For each operation in the step t∈S, the operation is a qualified operation in the decision step t.

[0027] Further, in step 3), the graph features are learned using a graph neural network GNN, and the node set V = S and the edge set E of the graph are connected arcs of the node set of the decided processing. Then for the node x, when x∈V, the multi-layer perceptron The result after iteration is represented as:

[0028]

[0029] wherein Represents the result of the (y-1)th iteration, ∈ (y) Let N(x) represent the learnable parameters in the y-th iteration, and let N(x) represent the set of neighboring nodes of node x, calculating the case of all neighboring nodes. The features of the processing state map are output after the final iteration.

[0030] Furthermore, in step 4), the constructed DRL scheduling decision model includes a policy network and a value network. The policy network is used to determine the action to be taken in the current state. During training, the policy network adjusts the probability distribution of these actions by minimizing the total processing time. The value network is used to evaluate the value of the state, i.e. the long-term reward expected to be obtained in a given state, so as to help decision-making in reinforcement learning.

[0031] Furthermore, in step 4), the Markov decision process of the constructed DRL scheduling decision model is as follows:

[0032] Status: For each step t, including the processed step L f and unprocessed steps L w Then the current state processing state diagram G(t) is represented as G(t) = (S,L) f ,L w When L f empty set hour, This indicates that processing has just begun, when L w empty set hour, This indicates that processing has been completed; for each processing step, calculate the corresponding completion time after its selection. Assume that in step t, the j-th processing step γ of the i-th wafer is selected. i,j The corresponding processing module completion time PM(γ) i,j Add the corresponding task duration П i,j Achieve PM(γ) i,j The update of ) is PM(γ) i,j ) = PM(γ i,j )+П i,j ;

[0033] Action: For step t, because wafer fabrication has a fixed process, the continuous process corresponds to the wafer fabrication route. The next processing step can only be performed after the previous processing step is completed. Therefore, each decision requires selecting a qualified operation A. t In this process, each wafer can have at most one wafer process step selected at step t, and as more and more steps are completed, A... t It will get smaller and smaller;

[0034] State transition: For step t, each time a qualified operation A is selected.t Then, G(t) = (S, L f , A w ) is updated, and the action selection process is repeated until the processing is completed.

[0035] Reward: The reward goal is to minimize the completion time, so when a qualified operation A t is selected at step t, the reward R t-1 (A t ) for selecting each qualified operation is calculated according to the completion time PM(A t ) of the current processing module and the completion time PM(A t ) after selecting A t , that is, R t (A t ) = PM(A t ) - PM(A t-1 ), and the smaller R t (A t ) means that the completion time of the operation selected at the current step is the smallest, that is, the reward is minimized.

[0036] Policy: For the current state processing state graph G(t) at step t, the random policy θ(G(t)) outputs the action probability distribution P(A t ) of the qualified operation A t .

[0037] Further, in step 5), for the real-time combined equipment wafer processing task, the current state graph features are extracted by GNN, the DRL scheduling decision model outputs the policy θ(G(t)) and the action probability distribution P(A t ), and the processing scheduling is performed according to the policy θ(G(t)) and the action probability distribution P(A t ).

[0038] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0039] 1. The present application can better understand the dynamic changes in the equipment processing process by capturing the dependency between the combined equipment processing states. This helps to more accurately predict the changes in the processing state and the interaction between the equipment, thereby improving the effect of scheduling decisions.

[0040] 2. The present application uses graph neural network (GNN) to extract the features of the processing state graph, which can effectively convert the information in the graph structure into a form that can be understood by the deep learning model. This helps to improve the model's understanding of the processing state graph, thereby better making decisions.

[0041] 3、The deep reinforcement learning (DRL) of the present application can autonomously learn and adjust the decision-making strategy according to the feedback of the environment, thereby adapting to different combined equipment processing instances. This means that the model can handle processing problems of various scales and complexities, and continuously improve performance during the training process.

[0042] 4、The present application realizes the automatic management of the processing of wafers by the combined equipment through the trained DRL scheduling decision-making model, which automatically makes scheduling decisions for new processing instances. This can greatly reduce the workload of manual intervention and improve processing efficiency and utilization of combined equipment processing modules BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a flowchart of the method of the present application.

[0044] Figure 2 is a network structure diagram of the method of the present application. DETAILED DESCRIPTION

[0045] The present application will be further described below in conjunction with specific embodiments.

[0046] The present embodiment provides a combined equipment scheduling optimization method based on GNN and DRL, as shown in Figure 1 , mainly including six processes: A, capturing the dependency relationship and processing state of the combined equipment wafer processing according to the combined equipment and wafer processing information; B, constructing the state variable space and action variable space according to the dependency relationship and processing state of the equipment wafer processing; C, constructing a processing state diagram according to the state variable space and action variable space, and extracting the features of the processing state diagram using a graph neural network GNN; D, learning the features of the extracted processing state diagram using the constructed deep reinforcement learning DRL scheduling decision-making model, obtaining the trained DRL scheduling decision-making model through the training of a large number of processing instances; E, outputting the action decision, i.e. the scheduling decision result, of the real-time combined equipment wafer processing task through the trained DRL scheduling decision-making model; F, scheduling the combined equipment wafer processing according to the scheduling decision result.

[0047] In A, the dependency relationship includes the processing route γ corresponding to each wafer type and the processing module PM in the combined equipment corresponding to the processing route, assuming that there is a kth wafer, which has l processing steps, i.e. the length of the corresponding processing route is l, then the kth wafer processing route γ k can be represented as follows:

[0048] γ k = [γ k,1 , γ k,2 , γ k,3 , …, γ k,l ]

[0049] wherein γ k,l represents the lth processing step;

[0050] Each processing step in each processing route corresponds to a processing module, and γ k The lth processing step γ k,l For example, let γ k,l corresponds to the mth processing module PM m , then γ k,l = PM m .

[0051] The processing state includes the combined equipment processing module and the mechanical arm occupation state, wherein the processing module can only process one wafer at a time, and when the wafer is processed and transported to the next processing module by the mechanical arm, a new wafer can be processed. The mechanical arm can only transport one wafer at a time, and after completing the transportation of one wafer, the next wafer can be transported.

[0052] In B, the state variable space includes the number of wafers n, and each wafer corresponds to a processing route length l. When constructing the matrix, each row of the matrix represents the same wafer, and each column represents a processing step in the processing route of the wafer in the corresponding row, that is, for the i-th row and the j-th column γ i,j represents the jth processing step of the i-th (i∈n) wafer, and the other rows and columns are the same as γ n,l represents the lth processing step of the n-th wafer, and further, the to-be-processed matrix p satisfies:

[0053]

[0054] The state variable space also includes the processing time α of each processing module, the mechanical arm moving time μ, and the wafer loading and unloading time β. Each time a wafer is transported, the mechanical arm needs to move to the wafer processing completed processing module, then unload the wafer, transport the wafer to the next processing module, and load it into the next processing module for processing. Then the corresponding task duration П satisfies the following:

[0055] П = 2 · (μ + β) + α

[0056] For the matrix p, each to-be-processed wafer step corresponds to a corresponding task duration П. When constructing the matrix, each row of the matrix represents the same wafer, and each column represents the processing duration of the processing step of the wafer in the corresponding row, that is, for the i-th row and the j-th column П i,j represents the processing duration of the jth processing step of the i-th (i∈n) wafer, and the other rows and columns are the same as П n,l represents the processing duration of the lth processing step of the n-th wafer, and then the task duration matrix r can be represented as:

[0057]

[0058] The action variable space includes the step of selecting candidate wafers to be processed, and the total number of decision steps S satisfies S = l × n. For each step t ∈ S, the operation is a qualified operation in the decision step t.

[0059] In C, a graph neural network (GNN) is used to learn graph features. Given a graph node set V = S and an edge set E representing the connecting arcs of the node set that has undergone decision processing, then for node x satisfying x ∈ V, the y-th multilayer perceptron... Results after iteration It is expressed as follows:

[0060]

[0061] in, Represents the result of the (y-1)th iteration, ∈ (y) Let N(x) represent the learnable parameters in the y-th iteration, and let N(x) represent the set of neighboring nodes of node x, calculating the case of all neighboring nodes. The features of the processing state map are output after the final iteration.

[0062] In D, the constructed DRL scheduling decision model comprises a policy network and a value network. The policy network determines the action to be taken in the current state. During training, the policy network adjusts the probability distribution of these actions by minimizing the total processing time. The value network evaluates the value of a state, i.e., the expected long-term reward that can be obtained in a given state, to aid decision-making in reinforcement learning.

[0063] The Markov decision process of the constructed DRL scheduling decision model is as follows:

[0064] Status: For each step t, including the processed step L f and unprocessed steps L w Then the current processing state diagram G(t) can be represented as G(t) = (S,L) f ,L w When L f empty set hour, This indicates that processing has just begun, when L w empty set hour, This indicates that processing has been completed. For each processing step, we calculate the completion time corresponding to its selection. Assume that in step t, the j-th processing step γ of the i-th wafer is selected. i,j The corresponding processing module completion time PM(γ) i,j Add the corresponding task duration П i,j Achieve PM(γ) i,j) is updated, i.e., PM(γ i,j ) = PM(γ i,j ) + П i,j ;

[0065] Action: For step t, because the wafer processing has a fixed process, the continuous process corresponds to the processing route of the wafer, and only when the current sequence step processing is completed can the processing of the next processing step be carried out, so each time decision needs to select a qualified operation A t , where each wafer can have at most one wafer process step selected at step t, and as more and more steps are completed, A t will become smaller and smaller;

[0066] State transition: for step t, after each selection of a qualified operation A t , update G(t) = (S, L f , L w ), repeat the action selection process until the processing is completed;

[0067] Reward: the reward goal is to minimize the completion time, so when the qualified operation A t is decided at step t, according to the completion time PM(A t-1 ) of the current processing module and the completion time PM(A t ) after selecting A t , the reward R t (A t ) of selecting each qualified operation is calculated, R t (A t ) = PM(A t ) - PM(A t-1 ), the smaller R t (A t ) means that the completion time of the current step selecting the operation is the smallest, i.e., the reward is minimized;

[0068] Strategy: for the current state processing state graph G(t) of step t, the random strategy θ(G(t)) outputs the action probability distribution P(A t ) of the qualified operation A t .

[0069] In E, for real-time combined equipment wafer processing tasks, as shown in Figure 2 , the goal is to minimize the completion time, and each processing extracts the current state graph features through GNN, inputs the graph features into the DRL scheduling decision model, and the scheduling decision model outputs the strategy θ(G(t)) and the action probability distribution P(A t ).

[0070] In F, the combined device can be set according to the above strategy θ(G(t)) and action probability distribution P(A t ) to realize wafer processing scheduling.

[0071] The above-described embodiments are only preferred embodiments of the present application, and are not intended to limit the scope of the present application. Any changes made in the shape and principle of the present application should be covered within the scope of protection of the present application.

Claims

1. A GNN and DRL-based combined device scheduling optimization method, characterized in that, The method comprises the following steps: 1) capturing the dependency relationship and processing state of the combined device wafer processing according to the combined device and wafer processing information; 2) constructing a state variable space and an action variable space according to the dependency relationship and processing state of the combined device wafer processing; 3) constructing a processing state graph according to the state variable space and the action variable space, and extracting the features of the processing state graph using a graph neural network GNN; 4) learning the features of the processing state graph using the constructed deep reinforcement learning DRL scheduling decision model, and obtaining the trained DRL scheduling decision model through a large number of processing instances; The Markov decision process of the constructed DRL scheduling decision model is as follows: State: for each decision step t, including processed steps L f and unprocessed steps L w , the current state processing state diagram G(t) is represented as G(t)=(S, L f , L w ), S represents the total set of decision steps; when L f is empty , it means just starting processing, when L w is empty , it means completed processing; for each processing step, calculate the corresponding completion time after its selection, assuming that the jth processing step γ i,j of the ith wafer is selected at decision step t, then the completion time PM(γ i,j ) of the corresponding processing module is added to the corresponding task duration П i,j to update PM(γ i,j ), that is, PM(γ i,j )=PM(γ i,j )+П i,j ; Action: For decision step t, because wafer processing has fixed process, continuous process corresponds to wafer processing route, only the current sequence step processing is completed, the next processing step processing can be carried out, so each time decision needs to select qualified operation A t , wherein each wafer can only have one wafer process step selected at decision step t, and A t will become smaller and smaller as more and more steps are completed; State transition: for decision step t, select eligible action A each time t Then, update G(t) = (S, L f , L w ) and repeat the action selection process until machining is complete; Reward: The reward objective is to minimize the completion time; therefore, in decision step t, a qualified operation A is selected. t At that time, based on the completion time PM(A) of the current processing module... t-1 And select A t Final completion time PM(A) t ), calculate the reward R for each qualified operation. t (A t ), for R t (A t ) = PM(A t )-PM(A t-1 ), R t (A t The smaller the value, the less the completion time of the current step in selecting this operation, which means minimizing the reward. Policy: For the current state of decision step t, process state diagram G(t), random policy θ(G(t)) outputs action probability distribution P(A t ) of eligible operation A t ; 5) outputting an action decision, i.e., a scheduling decision result, for a real-time combined device wafer processing task through the trained DRL scheduling decision model; 6) scheduling the combined device wafer processing according to the scheduling decision result.

2. The GNN and DRL-based combined device scheduling optimization method of claim 1, wherein, In step 1), the dependency relationship includes a processing route γ corresponding to each wafer type and a processing module M in a combined equipment corresponding to the processing route, assuming that there is an i-th wafer with l processing steps, i.e., the length of the corresponding processing route is l, then the i-th wafer processing route γ i is represented as follows: γ i = [γ i,1 ,γ i,2 ,γ i,3 ,…,γ i,l ] ; wherein γ i,l represents the i-th processing step of the i-th wafer; Each processing step in a processing route corresponds to a processing module, γ i,l Corresponding to the mth processing module M m ; The processing state includes the occupancy of the processing modules and the mechanical arm of the combined device, wherein each processing module can only process one wafer at a time, and when the wafer is processed and transported to the next processing module by the mechanical arm, a new wafer can be processed. The mechanical arm can only transport one wafer at a time, and after transporting one wafer, it can transport the next wafer.

3. The GNN and DRL-based combined device scheduling optimization method of claim 1, wherein, In step 2), the state variable space includes the number of wafers n, each wafer corresponds to a processing route length of l, when constructing the matrix, each row of the matrix represents the same wafer, and each column represents a processing step in the processing route of the corresponding row wafer, that is, for the i-th row and the j-th column γ i,j represents the j-th processing step of the i-th wafer, where i = 1, 2, …, n, j = 1, 2, …, l, and the to-be-processed matrix P satisfies: wherein γ n,l denotes the lth processing step of the nth wafer; The state variable space further includes the processing time α of each processing module, the moving time μ of the mechanical arm, and the loading or unloading time β of the wafer, wherein each time a wafer is transported, the mechanical arm needs to move to the processing module where the wafer processing is completed, then unload the wafer, transport the wafer to the next processing module, and load it into the next processing module for processing. Then the corresponding task duration П satisfies the following: П = 2 · (μ + β) + α; For the matrix p, each wafer processing step corresponds to a corresponding task duration P. When constructing the matrix, each row of the matrix represents the same wafer, and each column represents the task duration corresponding to the wafer processing step of the corresponding row wafer, that is, for the i-th row and the j-th column P i,j represents the task duration of the j-th processing step of the i-th wafer, and the task duration matrix r is represented as: In the formula, П n,l denotes the task duration of the lth processing step of the nth wafer; The action variable space includes the step of selecting candidate wafers to be processed, and the total decision step set S satisfies card(S) = l x n, i.e., the number of elements in the set is l x n. For each operation in the decision step t∈S, it is a qualified operation in the decision step t.

4. The GNN and DRL combined device scheduling optimization method of claim 3, wherein, In step 3), the graph features are learned by using a graph neural network GNN, wherein the node set V=S and the edge set E of the graph are the connection arcs of the node set of the decided processing, and the multi-layer perceptron of the y-th time satisfies x e V when x e V. The result after iteration is represented as follows: wherein, denotes the result of the y-1th iteration, denotes the result of the y-1th iteration of the neighboring node z, ∈ (y) denotes the learnable parameter of the yth iteration, N(x) denotes the neighboring node set of node x, and the case of all neighboring nodes is calculated The features of the processing state graph are output after the final iteration.

5. The GNN and DRL-based combined device scheduling optimization method of claim 1, wherein, In step 4), the constructed DRL scheduling decision model includes a policy network and a value network. The policy network is used to determine the action to be taken in the current state, and in the training process, the policy network adjusts the probability distribution of these actions by minimizing the total processing time. The value network is used to evaluate the value of the state, i.e., the long-term return expected to be obtained in a given state, in order to help decision-making in reinforcement learning.

6. The GNN and DRL-based combined device scheduling optimization method of claim 1, wherein, In step 5), the real-time combination device wafer processing task is processed to minimize the finished time, the current state graph features are extracted through the GNN, the DRL scheduling decision model outputs the random strategy θ(G(t)) and the action probability distribution P(A t ), and the processing scheduling is performed according to the random strategy θ(G(t)) and the action probability distribution P(A t ).

Images