Deep reinforcement learning-based fast response scheduling method and system for photoetching processing area

Abstract

The invention provides a fast response scheduling method and system for a photoetching processing area based on deep reinforcement learning. The method comprises the following steps: collecting and preprocessing data of the photoetching processing area of a wafer manufacturing system; defining a photoetching processing area fast response scheduling Markov sequential decision process, and establishing a photoetching processing area fast response scheduling method model based on deep reinforcement learning; constructing a photoetching processing area self-scheduling agent based on a deep neural network; constructing a photoetching processing area scheduling environment, interacting with the intelligent agent, and training and optimizing a scheduling decision; and deploying the self-scheduling agent model to a deep reinforcement learning photoetching processing area fast response scheduling system to monitor an environment state in real time and make a scheduling decision. The real-time performance and the dynamic response capability are relatively strong, the adaptive optimization of multiple targets in the photoetching area can be realized, the scheduling efficiency and the production efficiency of the photoetching processing area are improved, and an effective solution is provided for the real-time scheduling problem of the photoetching processing area.

Description

Technical Field

[0001] The present invention relates to the technical field of non-equivalent parallel machine scheduling in a flexible job shop, and in particular to a fast response scheduling method and system for a wafer manufacturing lithography processing area based on deep reinforcement learning. Background Art

[0002] Wafer manufacturing systems are the cornerstone of the development of the semiconductor industry. Usually, wafers need to go through repeated oxidation, photolithography, etching, ion implantation, and metal plating processes to produce the designed integrated circuit chips. Due to its processing characteristics, the wafers are processed layer by layer during the processing, and the production of each layer of circuits needs to be carried out in different work areas. There are a lot of heavy processing processes, and the processing flow can reach hundreds or even thousands of processes, which increases the complexity of the wafer processing process.

[0003] The photolithography process is the most important processing step in the wafer manufacturing system. It mainly projects the pattern on the mask onto the photoresist of the wafer through irradiation with light of a specific wavelength. After the pattern detection, the circuit is retained on the wafer through the transfer method to complete the photolithography operation. The processing of the photolithography process is inseparable from the photolithography machine in the photolithography area. Usually, the price of a most advanced photolithography machine can reach hundreds of millions of euros. The high price makes the investment cost of the photolithography area huge. The complexity of the photolithography process also leads to a large proportion of work-in-progress in the photolithography area, which can reach 40% of the entire factory. Therefore, the high cost investment and the high proportion of work-in-progress make the photolithography area the core processing area of ​​the entire wafer manufacturing system.

[0004] In the production process of wafer lithography area, there are dynamic factors such as dynamic arrival of wafer lot, change of wafer priority and insertion of urgent order. Therefore, in order to adapt to the dynamic production environment and improve the efficiency of lithography machine, the scheduling of lithography area needs to have the ability to respond quickly and adapt to the dynamic influencing factors. By searching the existing patent literature, it is found that there are few scheduling methods for wafer lithography area at present. Among them, Chinese patents CN104536412A, CN111199272B and CN117709683A generally have the problems that the calculation time of optimization method increases rapidly with the increase of problem scale, the real-time performance is poor, and the optimization target is relatively single, making it difficult to respond quickly in a short time to obtain a reasonable and effective scheduling plan. In addition, these scheduling methods do not fully consider the equipment-specific constraints, sequence-related preparation time constraints and mask constraints of the lithography area, and are not suitable for the real lithography processing area scheduling scenario. Summary of the invention

[0005] In order to solve the above problems, the technical solution of the present invention provides a fast response scheduling method and system for lithography processing areas based on deep reinforcement learning with strong adaptability and real-time performance, so as to improve the scheduling efficiency of the lithography area.

[0006] The present invention provides a fast response scheduling method for a lithography processing area based on deep reinforcement learning, which is characterized by comprising the following steps:

[0007] Step 1: Collect historical production and operation data of the lithography processing area in the wafer manufacturing system, process missing values ​​and outliers through data cleaning, eliminate the influence of different dimensions through data conversion, and finally divide the data into training data set and verification data set;

[0008] Step 2: By constructing and designing the state space S of the lithography processing area, the action space A of the lithography processing area, and the definition of the adaptive reward function R of the lithography processing area, the fast response scheduling problem of the lithography processing area is transformed into a Markov sequential decision problem, and a deep reinforcement learning fast response scheduling model of the lithography processing area is constructed based on this.

[0009] Step 3: Construct a self-scheduling agent for the lithography processing area based on a deep neural network. The agent uses a deep reinforcement learning PPO algorithm based on policy gradients. The algorithm uses the Actor-Critic algorithm framework to construct a policy network Actor and a value network Critic.

[0010] Step 4: Based on the training data set in step 1, the lithography processing area self-scheduling agent interacts with the lithography processing area environment, generates historical interaction action experience data under different optimization index requirements at each moment and stores it in the experience pool. The lithography processing area agent updates the network parameters according to the experience pool and trains the agent to learn the optimal scheduling strategy with the maximum total reward;

[0011] Step 5: Based on the verification data set in step 1, the accuracy of the trained lithography processing area self-scheduling agent model is verified. If the verification is passed, proceed to step 6, otherwise return to step 4;

[0012] Step 6: Deploy the verified lithography processing area self-scheduling intelligent agent model to the computer to realize real-time detection of state characteristic parameter data in the state space of multiple operation areas of the wafer manufacturing system, wafers in the lithography processing area, and scheduling data triggered when the equipment is idle, so as to realize real-time system detection and quickly decide on scheduling strategies.

[0013] Furthermore, in the fast response scheduling method for the lithography processing area based on deep reinforcement learning provided by the present invention, it can also have the following characteristics: in step 2, the state space S of the lithography processing area = S 1 ∩S 2 ∩S 3 ∩S 4 , where S 1 Indicates the overall optimization index sub-state of the photolithography processing area, S 2 Indicates the wafer lot status in the photolithography processing area, S 3Indicates the sub-state of the lithography equipment in the lithography processing area, S 4 Indicates the status of the mask in the photolithography processing area;

[0014] The photolithography processing area action space A={a 1 ,a 2 ,a 3 ,a 4 ,a 5 ,a 6 ,a 7 ,a 8}, where a 1 , a 2 , ..., a 8 Represents the available actions of the self-scheduling agent in the lithography processing area, which are the heuristic WSPT rule, WLPT rule, FIFO rule, WMDD rule, WCR rule, WEDD rule, ATC rule, and WINQ rule;

[0015] The photolithography area adaptive reward function in represents the instant reward of the optimization index equipment utilization rate of the photolithography processing area at time i, P jm,i is the processing time of wafer j on lithography equipment m at time i, N m is the number of lithography equipment, C i is the processing cycle time of the lithography processing area at time i, M is the set of all equipment in the lithography processing area, M = {1, 2, ..., m}, J is the set of all wafers to be lithography, J = {1, 2, ..., j}, T is the maximum time step of the self-scheduling agent decision; where r 1,i =max(c jm,i -d j ,0) represents the optimization index delay time instant reward of the photolithography processing area at time i, c jm,i is the completion time of wafer j on lithography machine m at time i, d j is the delivery date of wafer j. If the completion time of wafer j is greater than the delivery date, a delay occurs. If the completion time of wafer j is less than the delivery date, the delay time is 0. max(·) is the maximum value function. 2,i =max(t aj +t vj +P jm ) represents the optimization index production cycle instant reward of the photolithography processing area at time i, t aj is the time when wafer j arrives at the photolithography processing area, t vjis the mask adjustment time and sequence switching preparation time required before wafer j starts processing, max(·) is the maximum value function; α, β, γ are the adjustment coefficients of the self-scheduling agent’s adaptive reward function, which can be dynamically adjusted during the self-scheduling agent training process according to the different requirements of the current scheduling for the three optimization indicators, so as to adapt to the demand ratio of different optimization indicators in the real lithography processing area scheduling; norm(·) is the normalization function, which is used to convert instant rewards of different dimensions into the same dimension for weighted calculation.

[0016] Furthermore, in the fast response scheduling method for the lithography processing area based on deep reinforcement learning provided by the present invention, it can also have the following characteristics: in step 3, the lithography processing area self-scheduling intelligent agent Agent adopts the deep reinforcement learning PPO (Proximal Policy Optimization, PPO) algorithm based on policy gradient, and constructs the policy network Actor and the value network Critic based on the deep neural network. The policy network Actor inputs the current environmental state of the lithography processing area and outputs the probability distribution of the action to constitute the current strategy; the value network inputs the current environmental state of the lithography processing area and outputs the state value in the current state, that is, the expected return; the PPO algorithm uses multi-step TD, and after running a trajectory data, it starts to calculate the cumulative returns of each state and the advantages of the action. After the Actor network is updated, the Critic network is updated using the same data sampled from the experience replay pool. The Critic network is updated in the following way: the discounted return and the current state value predicted by the Critic network are calculated, and the network is trained using the loss function.

[0017] Furthermore, in the fast response scheduling method for lithography processing area based on deep reinforcement learning provided by the present invention, it can also have the following characteristics: the calculation method of the loss function is:

[0018]

[0019] in is the expected value at time t, π θ That is, the strategy π with network parameters θ; Represents the probability ratio of the new and old strategies at time t. With the help of importance sampling, the strategy to be optimized π θ and the old strategy Different, so as to achieve data reuse and improve training efficiency. Clip and ε are the clipping function and clipping coefficient respectively. Through this coefficient, the difference between the new and old strategies can be controlled within a controllable range during update. Min(·) is the minimum value function. is the advantage function, which is used to describe the advantage over other actions, and is defined as follows:

[0020]

[0021] in,

[0022] δ t =r t +χV(s t+1 )-V(s t ) (3)

[0023] V(s t ) indicates state s t The state value under r t is the reward at time t, χ is the discount factor, Len represents the trajectory length, and λ is a balance parameter used to adjust the bias and variance in the advantage function estimate.

[0024] Furthermore, in the fast response scheduling method for the lithography processing area based on deep reinforcement learning provided by the present invention, it can also have the following characteristics: in step 4, the lithography processing area self-scheduling agent inputs the overall optimization index sub-state S of the lithography processing area 1 , wafer lot status S in the photolithography processing area 2 , the state of the photolithography machine in the photolithography processing area S 3 , the state of the mask in the photolithography processing area S 4 , the input dimension is the dimension of the state space S; the output action a in the action space A k , the output dimension is the dimension of the action space.

[0025] The present invention also provides a fast response scheduling system for a lithography processing area, which is communicatively connected to a wafer manufacturing system, and is characterized in that it comprises: a model module, a data interaction module, and a fast response scheduling control module for a lithography processing area, wherein the model module has a verified self-scheduling intelligent agent model for a lithography processing area obtained by the fast response scheduling method for a lithography processing area based on deep reinforcement learning described in any one of claims 1 to 5, and the self-scheduling intelligent agent model for a lithography processing area outputs an optimal action according to an input state; the data interaction module obtains lithography processing data, and transmits it to the fast response scheduling control module for the lithography processing area after data processing; the fast response scheduling control module for the lithography processing area comprises a state monitoring unit, a data sorting unit, a state input model unit, a model result receiving unit, a scheduling strategy execution module, and a pre-adjustment data return module; the state monitoring unit monitors the operating state of the lithography processing area in real time; the data sorting unit sorts the operating state information S of the lithography processing area into sub-state information S of the overall optimization index of the lithography processing area 1 , wafer lot status information S in the photolithography processing area 2 , the photolithography machine status information S in the photolithography processing area 3 , the mask status information S in the photolithography processing area 4; The state input model unit will S 1 , S 2 , S 3 , S 4 The data is input into the self-scheduling agent model of the lithography processing area; the model result receiving unit receives the optimal action returned by the self-scheduling agent model of the lithography processing area, and the optimal action is the scheduling strategy; the scheduling strategy execution module executes the corresponding scheduling action according to the received scheduling strategy to realize the scheduling of the wafers and masks waiting in the lithography processing area; the scheduling data return module returns the pre-scheduling result to the data interaction module after the scheduling action is completed.

[0026] Furthermore, the photolithography processing area fast response scheduling system provided by the present invention may also have the following features: wherein the photolithography processing data includes the photolithography processing area wafer, photolithography machine and mask information.

[0027] Furthermore, the fast response scheduling system for the lithography processing area provided by the present invention may also have the following feature: wherein, the data interaction module also outputs the scheduling results and the scheduling process to the data interaction module for data sharing for use by other modules.

[0028] The present invention also provides a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the steps in the above-mentioned fast response scheduling method for lithography processing areas based on deep reinforcement learning are implemented.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1) In view of the problem that the calculation time of existing technical research and methods increases rapidly with the increase of problem scale and the real-time performance is poor, the method of the present invention can quickly solve the scheduling optimization problem in a short time, and can adapt to the environment and make reasonable scheduling when facing different scale scenarios of the same workshop without retraining.

[0031] 2) In order to solve the problem of single optimization target in the existing technical research methods, the method of the present invention designs an adaptive reward function for the lithography processing area, adaptively adjusts the index coefficient according to different scenarios and different index requirements, so as to simultaneously optimize the utilization rate of the lithography equipment, the wafer lot delay time, and the lithography processing cycle time, and realizes adaptive optimization of multiple targets.

[0032] 3) The fast response scheduling method for the lithography processing area based on deep reinforcement learning of the present invention is also designed to add the real-time status of the mask to the scheduling status, so that the scheduling process takes into account the equipment-specific constraints, mask constraints and sequence-related preparation time constraints unique to the lithography processing area. Compared with the existing technology, this method is more in line with the actual lithography processing area scheduling scenario and more scientific and reasonable.

[0033] 4) The fast response scheduling system of the lithography processing area of ​​the present invention can monitor the state changes of the lithography processing area in real time through the fast response scheduling control module of the lithography processing area, output the optimal scheduling strategy, and improve the scheduling efficiency and production performance of the lithography processing area. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flowchart of the implementation of the fast response scheduling method for the lithography processing area based on deep reinforcement learning of the present invention;

[0035] Figure 2 It is a decision and training flow chart of the PPO algorithm for the self-scheduling intelligent agent in the lithography processing area of ​​the present invention;

[0036] Figure 3 It is a framework diagram of the fast response scheduling method of the lithography processing area based on deep reinforcement learning in the present invention;

[0037] Figure 4 This is a structural block diagram of the fast response scheduling system for the lithography processing area based on deep reinforcement learning in the present invention. DETAILED DESCRIPTION

[0038] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments are combined with the accompanying drawings to specifically illustrate the technical solutions of the present invention.

[0039] See also Figure 1 and Figure 3 The embodiment of the present invention provides a fast response scheduling method for a lithography processing area based on deep reinforcement learning with strong adaptability and real-time performance, which improves the scheduling efficiency of the lithography area and has practical engineering significance and application value.

[0040] like Figure 1 As shown, the implementation process of the fast response scheduling method for the lithography processing area based on deep reinforcement learning includes the following steps:

[0041] Step 1: Collect historical production operation data of the lithography processing area in the wafer manufacturing system, perform data cleaning and data conversion, and finally divide the data into training data set and verification data set.

[0042] The historical production and operation data of the lithography processing area involved in the present invention is taken from a continuous period of the lithography process area C of a wafer factory in Shanghai. There are 10 lithography machines in the processing area C, and a total of 20 different types of processed wafers. The data is cleaned to remove missing values ​​and outliers during the operation process, and the influence between different dimensions is eliminated through data conversion. Finally, the data is randomly divided into a training set and a validation set.

[0043] Step 2: By constructing and designing the state space of the lithography processing area, the action space of the lithography processing area and the definition of the adaptive reward function of the lithography processing area, the fast response scheduling problem of the lithography processing area is transformed into a Markov sequential decision problem and modeled.

[0044] The state space of the photolithography processing area S = S 1 ∩S 2 ∩S 3 ∩S 4 , which includes the overall optimization index sub-state S of the lithography processing area 1 , Photolithography processing area wafer lot status S 2 , lithography equipment sub-state S in the lithography processing area 3 , the state of the mask in the photolithography processing area S 4 The state space constructed for the lithography processing area of ​​the wafer fab is shown in Table 1.

[0045] Table 1 Lithography processing area state space

[0046]

[0047] Lithography processing area action space A = {a 1 ,a 2 ,a 3 ,a 4 ,a 5 ,a 6 ,a 7 ,a 8}, where a 1 , a 2 , ..., a 8 Represents the available actions of the self-scheduling intelligent agent in the lithography processing area. The actions of the self-scheduling intelligent agent of the present invention select 8 heuristic rules in the lithography processing area that perform well in minimizing delay time, maximizing equipment utilization and minimizing wafer cycle time indicators, namely WSPT rule, WLPT rule, FIFO rule, WMDD rule, WCR rule, WEDD rule, ATC rule and WINQ rule.

[0048] An adaptive reward function is constructed for the three commonly used optimization indicators in the lithography processing area. The function is as follows:

[0049]

[0050] in represents the instant reward of the optimization index equipment utilization rate of the photolithography processing area at time i, P jm,i is the processing time of wafer j on lithography equipment m at time i, N m is the number of lithography equipment, C iis the processing cycle time of the lithography processing area at time i, M is the set of all equipment in the lithography processing area, M = {1, 2, ..., m}, J is the set of all wafers to be lithography, J = {1, 2, ..., j}, T is the maximum time step of the self-scheduling agent decision; where r 1,i =max(c jm,i -d j ,0) represents the optimization index delay time instant reward of the photolithography processing area at time i, c jm,i is the completion time of wafer j on lithography machine m at time i, d j is the delivery date of wafer j. If the completion time of wafer j is greater than the delivery date, a delay occurs. If the completion time of wafer j is less than the delivery date, the delay time is 0. 2,i =max(t aj +t vj +P jm ) represents the optimization index production cycle instant reward of the photolithography processing area at time i, t aj is the time when wafer j arrives at the photolithography processing area, t vj is the mask adjustment time and sequence switching preparation time required before wafer j starts processing. α, β, γ are the adjustment coefficients of the self-scheduling agent's adaptive reward function. The coefficients can be dynamically adjusted during the self-scheduling agent training process according to the different requirements of the current scheduling for the three optimization indicators to adapt to the demand ratio of different optimization indicators in the real lithography processing area scheduling. norm(·) is a normalization function, which is used to convert instant rewards of different dimensions into the same dimension. After normalization, the three instant rewards are all between [0,1]. The adaptive reward function is the accumulation of the three instant rewards in one operation cycle.

[0051] Step 3: Build a self-scheduling agent for the lithography processing area based on a neural network, and use the policy gradient-based deep reinforcement learning PPO algorithm to train the agent.

[0052] The deep reinforcement learning PPO (Proximal Policy Optimization, PPO) algorithm combines the value function and policy search algorithms. It is fast, stable and easy to adjust. It can make the policy update slightly to ensure that the difference between the new and old policies is within a controllable range. It not only realizes the update of network parameters, but also avoids the problem of unexpected decline in model performance due to excessive update amplitude. The self-scheduling agent in the lithography processing area constructs the policy network Actor and the value network Critic based on the deep neural network. The policy network Actor inputs the current environmental state of the lithography processing area and outputs the probability distribution of the action to form the current policy. The value network inputs the current environmental state of the lithography processing area and outputs the state value in the current state, that is, the expected return. The PPO algorithm uses multi-step TD. After running a trajectory data, it starts to calculate the cumulative return of each state and the advantage of the action. After the Actor network is updated, the Critic network is updated using the same data sampled from the experience replay pool. The update method of the Critic network is: calculate the discounted return and the current state value predicted by the Critic network, and train the network using the loss function. The loss function is calculated as follows:

[0053]

[0054] in is the expected value at time t, π θ That is, the strategy π with network parameters θ; Represents the probability ratio of the new and old strategies at time t. With the help of importance sampling, the strategy to be optimized π θ and the old strategy Different, so as to achieve data reuse and improve training efficiency. Clip and ε are the clipping function and clipping coefficient respectively. Through this coefficient, the difference between the new and old strategies can be controlled within a controllable range during update. Min(·) is the minimum value function. is the advantage function, which is used to describe the advantage over other actions, and is defined as follows:

[0055]

[0056] in,

[0057] δ t =r t +χV(s t+1 )-V(s t ) (3)

[0058] V(s t ) indicates state s t The state value under r tis the reward at time t, χ is the discount factor, Len represents the trajectory length, and λ is a balance parameter used to adjust the bias and variance in the advantage function estimate.

[0059] Step 4: Based on the training data set in step 1, the lithography processing area self-scheduling agent interacts with the lithography processing area environment, generates historical interaction action experience data under different optimization index requirements at each moment and stores it in the experience pool. When the update condition is met, the lithography processing area agent calculates the state s according to the experience trajectory in the experience pool. t The state value V(s t ) and advantage function The loss function is calculated and the network parameters are updated to guide the agent to learn the optimal scheduling strategy. The trained agent saves the optimal model parameters and waits for accuracy verification. The algorithm training and decision flow chart is as follows: Figure 2 shown.

[0060] Step 5: Use the verification set to verify the accuracy of the trained self-scheduling agent. If the verification passes, proceed to step 6, otherwise return to step 4.

[0061] This example uses the above-mentioned invention's lithography processing area self-scheduling PPO agent and deep reinforcement learning DQN (DeepQ-Network, DQN) algorithm, WSPT rule, FIFO rule, WMDD rule and ATC rule to verify accuracy on the validation set. Table 2 compares the performance of the present invention with other methods in terms of delay time, processing cycle and equipment utilization optimization indicators. It can be seen from the table that the lithography processing self-scheduling agent trained by the PPO algorithm provided by the present invention is superior to other methods under three different optimization indicators.

[0062] Table 2 Performance comparison of the present invention and other methods

[0063]

[0064]

[0065] Step 6: Deploy and install the verified lithography area self-scheduling agent model into the deep reinforcement learning lithography area fast response scheduling system, observe the state characteristic parameters in the state space of multiple operation areas of the wafer manufacturing system in real time, and trigger scheduling when the wafer arrives at the lithography area and the equipment is idle, so as to realize real-time system detection and quickly decide on scheduling strategies.

[0066] See also Figure 4 The embodiment of the present invention also provides a photolithography processing area fast response scheduling system, which is a computer system and is connected to the wafer manufacturing system in communication. It includes: a model module, a data interaction module, a photolithography processing area fast response scheduling control module, a computer memory and a processor.

[0067] The model module has a verified lithography area self-scheduling intelligent agent model obtained by the above-mentioned lithography area fast response scheduling method, and the lithography area self-scheduling intelligent agent model outputs the optimal action according to the input state.

[0068] The data interaction module obtains data such as wafer, photolithography machine and mask information in the photolithography processing area, and transmits these data to the photolithography processing area fast response scheduling control module after processing.

[0069] The photolithography processing area fast response scheduling control module includes a state monitoring unit, a data sorting unit, a state input model unit, a model result receiving unit, a scheduling strategy execution module, and a pre-adjustment data return module. The state monitoring unit monitors the operation state of the photolithography processing area in real time. The data sorting unit sorts the operation state information S of the photolithography processing area into the sub-state information S of the overall optimization index of the photolithography processing area. 1 , wafer lot status information S in the photolithography processing area 2 , the photolithography machine status information S in the photolithography processing area 3 , the mask status information S in the photolithography processing area 4 The state input model unit will S 1 , S 2 , S 3 , S 4 The data is input into the self-scheduling agent model of the lithography processing area. The model result receiving unit receives the optimal action returned by the self-scheduling agent model of the lithography processing area, and the optimal action is the scheduling strategy. The scheduling strategy execution module executes the corresponding scheduling action according to the received scheduling strategy to realize the scheduling of the wafers and masks waiting in the lithography processing area. The scheduling data return module returns the pre-scheduling result to the data interaction module after the scheduling action is completed.

[0070] The data interaction module also outputs the scheduling result and the scheduling process to the data interaction module for data sharing for use by other modules.

[0071] Subsequently, the system continues to monitor the production operation status of the lithography processing area to prepare for the next scheduling task.

[0072] An embodiment of the present invention also provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the steps in the aforementioned fast response scheduling method for lithography processing areas based on deep reinforcement learning are performed.

[0073] The above is only a detailed description of the specific embodiments of the present invention, but it does not limit the protection scope of the present invention. Any technician familiar with the profession can use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A fast response scheduling method for lithography processing area based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Collect historical production and operation data of the lithography processing area in the wafer manufacturing system, and divide the data into a training data set and a validation data set after processing; Step 2: By constructing the state space S of the lithography processing area, the action space A of the lithography processing area and the definition of the adaptive reward function R of the lithography processing area, the fast response scheduling problem of the lithography processing area is transformed into a Markov sequential decision problem and modeled; Step 3: Construct a self-scheduling agent for the lithography processing area based on a deep neural network; Step 4: Based on the training data set in step 1, the lithography processing area self-scheduling agent interacts with the lithography processing area environment, generates historical interaction action experience data under different optimization index requirements at each moment and stores it in the experience pool, the lithography processing area self-scheduling agent updates the network parameters according to the experience pool, and trains the agent to learn the optimal scheduling strategy with the maximum total reward; Step 5: Based on the verification data set in step 1, the accuracy of the trained lithography processing area self-scheduling agent model is verified. If the verification is passed, proceed to step 6, otherwise return to step 4; Step 6: Deploy the verified self-scheduling intelligent agent model of the lithography processing area to the computer, and observe in real time the state characteristic parameter data in the state space of multiple operation areas of the wafer manufacturing system, the arrival of wafers in the lithography processing area, and the scheduling data triggered when the equipment is idle, so as to realize real-time system detection and quickly decide on scheduling strategies.

2. The fast response scheduling method for lithography processing area based on deep reinforcement learning as claimed in claim 1, characterized in that: In step 2, the state space S of the lithography processing area = S1∩S2∩S3∩S4, where S1 represents the overall optimization index sub-state of the lithography processing area, S2 represents the wafer lot sub-state in the lithography processing area, S3 represents the lithography equipment sub-state in the lithography processing area, and S4 represents the mask sub-state in the lithography processing area; The photolithography area action space A = {a1, a2, a3, a4, a5, a6, a7, a8}, where a1, a2, ..., a8 represent the available actions of the self-scheduling agent in the photolithography area, which are heuristic WSPT rule, WLPT rule, FIFO rule, WMDD rule, WCR rule, WEDD rule, ATC rule, and WINQ rule respectively; The photolithography area adaptive reward function R is as follows: in, represents the instant reward of the optimization index equipment utilization rate of the photolithography processing area at time i, P jm,i is the processing time of wafer j on lithography equipment m at time i, N m is the number of lithography equipment, C i is the processing cycle time of the lithography processing area at time i, M is the set of all equipment in the lithography processing area, M = {1, 2, ..., m}, J is the set of all wafers to be lithography, J = {1, 2, ..., j}, and T is the maximum time step of the self-scheduling agent decision; r 1,i =max(c jm,i -d j ,0) represents the optimization index delay time instant reward of the photolithography processing area at time i, c jm,i is the completion time of wafer j on lithography machine m at time i, d j is the delivery date of wafer j. If the completion time of wafer j is greater than the delivery date, a delay occurs. If the completion time of wafer j is less than the delivery date, the delay time is 0. max(·) is the maximum value function. r 2,i =max(t aj +t vj +P jm ) represents the optimization index production cycle instant reward of the photolithography processing area at time i, t aj is the time when wafer j arrives at the photolithography processing area, t vj is the mask adjustment time and sequence switching preparation time required before wafer j starts processing, max(·) is the maximum value function; α, β, and γ are function adjustment coefficients, which are dynamically adjusted during the self-scheduling agent training process according to the different requirements of the current scheduling for the three optimization indicators; norm(·) is a normalization function, which is used to convert instant rewards of different dimensions into the same dimension.

3. The fast response scheduling method for lithography processing area based on deep reinforcement learning as claimed in claim 2, characterized in that: In step 3, the lithography processing area self-scheduling agent adopts the PPO algorithm based on policy gradient deep reinforcement learning, and constructs the policy network Actor and the value network Critic based on the deep neural network; The Actor inputs the current state of the lithography processing area environment and outputs the probability distribution of actions to form the current strategy; Critic inputs the current state of the lithography processing area environment and outputs the state value under the current state, that is, the expected return; The PPO algorithm uses multi-step TD. After running a trajectory data, it starts to calculate the cumulative rewards of each state and the advantages of the action. After the Actor network is updated, the Critic network is updated using the same data sampled from the experience replay pool. The critic network is updated by calculating the discounted return and the current state value predicted by the critic network, and training the network using the loss function.

4. The fast response scheduling method for lithography processing area based on deep reinforcement learning as claimed in claim 3, characterized in that: The loss function is calculated as follows: in, is the expected value at time t, π θ That is, the strategy π with network parameters θ; represents the probability ratio of the new and old strategies at time t; clip and ε are the clipping function and the clipping coefficient, respectively, which can control the difference between the new and old strategies within a controllable range during the update; min(·) is the minimum value function; is the advantage function, which is used to describe the advantage over other actions. The definition is as follows: in, d t =r t +xV(s t+1 )-V(s t ) (3) V(s t ) indicates state s t The state value under r t is the reward at time t, χ is the discount factor, Len represents the trajectory length, and λ is a balance parameter used to adjust the bias and variance in the advantage function estimate.

5. The fast response scheduling method for lithography processing area based on deep reinforcement learning as claimed in claim 4, characterized in that: In step 4, the lithography area self-scheduling agent inputs the lithography area overall optimization index sub-state S1, the wafer lot sub-state S2, the lithography machine sub-state S3, and the mask sub-state S4 in the lithography area, and the input dimension is the dimension of the state space S; the output action a in the action space A k , the output dimension is the dimension of the action space.

6. A fast response scheduling system for a lithography processing area, which is connected to a wafer manufacturing system in communication, characterized in that: include: Model module, data interaction module, photolithography processing area fast response scheduling control module, The model module has a verified lithography processing area self-scheduling agent model obtained by the lithography processing area fast response scheduling method based on deep reinforcement learning according to any one of claims 1 to 5, and the lithography processing area self-scheduling agent model outputs the optimal action according to the input state; The data interaction module acquires the photolithography processing data and transmits it to the photolithography processing area fast response scheduling control module after data processing; The photolithography processing area fast response scheduling control module includes a state monitoring unit, a data sorting unit, a state input model unit, a model result receiving unit, a scheduling strategy execution module, and a pre-adjustment data return module; The state monitoring unit monitors the operating state of the photolithography processing area in real time; The data sorting unit sorts the photolithography processing area operation status information S into the photolithography processing area overall optimization index sub-status information S1, the wafer lot sub-status information S2 in the photolithography processing area, the photolithography machine sub-status information S3 in the photolithography processing area, and the mask sub-status information S4 in the photolithography processing area; The state input model unit inputs S1, S2, S3, and S4 into the photolithography processing area self-scheduling agent model; The model result receiving unit receives the optimal action returned by the self-scheduling agent model of the lithography processing area, wherein the optimal action is the scheduling strategy; The scheduling strategy execution module executes corresponding scheduling actions according to the received scheduling strategy to realize the scheduling of the wafers and masks waiting in the lithography processing area; The scheduling data returning module returns the pre-scheduling result to the data interaction module after the scheduling action is completed.

7. The fast response scheduling system for photolithography processing area according to claim 6, characterized in that: in, The photolithography processing data includes the photolithography processing area wafer, photolithography machine and mask information.

8. The fast response scheduling system for photolithography processing area according to claim 6, characterized in that: in, The data interaction module also outputs the scheduling result and the scheduling process to the data of the data interaction module for data sharing.

9. A computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the steps in the fast response scheduling method for a lithography processing area based on deep reinforcement learning as described in any one of claims 1 to 8.

Images