Reactive collaborative scheduling optimization method and system for material distribution in multiple operation areas of wafer manufacturing system

Abstract

The present invention provides a reactive collaborative scheduling optimization method and system for material distribution in multiple work areas of a wafer manufacturing system based on predictive self-collaborative reinforcement learning, comprising the following steps: constructing a total return prediction model for collaborative scheduling of material distribution in multiple work areas based on a deep neural network and training it; defining and designing a collaborative scheduling environment for material distribution in multiple work areas; constructing an intelligent agent for scheduling work areas; interacting with multiple agents and the collaborative scheduling environment for material distribution in multiple work areas to form and collect historical experience data for training multiple agents to learn optimal decision strategies and maximize total returns; integrating the intelligent agent model for scheduling wafer multi-work areas into the collaborative scheduling system for material distribution in multiple work areas of a wafer manufacturing system. The present invention realizes the collaborative scheduling optimization of material distribution in multiple work areas of a wafer manufacturing system through a predictive self-collaborative reinforcement learning method, which can effectively improve the collaborative scheduling performance of intelligent agents, and greatly reduces the production cycle of a wafer manufacturing system and reduces the number of times a wafer lot violates the residence time constraint of an work area through real-time collaborative scheduling of multiple work areas.

Description

Technical Field

[0001] The present invention relates to a reactive collaborative scheduling technology for material distribution in multiple operation areas of a wafer manufacturing system, and belongs to the field of scheduling optimization of wafer manufacturing systems. Background Art

[0002] The wafer manufacturing system consists of multiple operating areas such as diffusion, thin film, lithography, etching, and implantation, and is the foundation and core link of semiconductor manufacturing. Wafer manufacturing is carried out collaboratively by multiple operating areas, and each operating area contains multiple parallel equipment groups. Its production process has complex process flow, large-scale, reentrant process, multiple residence time constraints, batch processing and other complex characteristics, which brings huge challenges to the optimization of coordinated scheduling of material distribution in multiple operating areas. At present, wafer manufacturing faces a production environment with multiple product types, high concurrent orders, and large-scale customization, resulting in a long wafer production cycle. The production cycle is a key performance indicator for evaluating the production efficiency and on-time delivery capability of the wafer manufacturing system. Short production cycle and high on-time delivery rate have become important factors for wafer manufacturing companies to enhance their market competitiveness. Therefore, it is of great theoretical significance and economic value to study the optimization method of coordinated scheduling of material distribution in multiple operating areas of the wafer manufacturing system, reduce the total wafer production cycle and the number of violations of multiple residence time constraints to improve the performance of the wafer manufacturing system.

[0003] For the complex scheduling problem of coordinated scheduling optimization of material distribution in multiple work areas of wafer manufacturing systems, the main methods currently include mathematical programming, intelligent optimization algorithms, scheduling rules and reinforcement learning. Among them, the calculation time of traditional mathematical programming and intelligent optimization algorithms increases rapidly as the scale of the problem increases. Given the large-scale characteristics of coordinated scheduling of material distribution in multiple work areas of wafer manufacturing systems, it is difficult to obtain reasonable and effective scheduling solutions in a timely manner within the scheduling cycle. Scheduling rules quickly select suitable wafers to be processed by assigning priorities to wafers in the waiting queue, but it is difficult to achieve coordinated scheduling optimization of material distribution in multiple work areas by only considering local information, resulting in low performance of scheduling solutions. With the research and application of reinforcement learning in the fields of robot control and production system scheduling optimization, it provides a research basis and ideas for the coordinated scheduling optimization of material distribution in multiple work areas of wafer manufacturing systems. Summary of the invention

[0004] The technical problem to be solved by the present invention is that it is difficult to realize the coordinated scheduling of material distribution in multiple work areas of a wafer manufacturing system in existing research and methods so as to minimize the total production cycle and the number of violations of multiple residence time constraints.

[0005] In order to solve the above technical problems, the technical solution of the present invention is to provide a reactive collaborative scheduling optimization method for material distribution in multiple work areas of a wafer manufacturing system, which is characterized by comprising the following steps:

[0006] Step S1: Collect and process the historical production operation data of multiple operation areas of the wafer manufacturing system to form a training data set and a verification data set. In the training data set and the verification data set, a historical operation chain operation data sample of multiple operation areas is formed through data cleaning and processing, wherein the chain data is respectively linked by the current production operation status, scheduling decision actions, immediate rewards obtained by executing decision actions, and production operation status updated after executing actions in each operation area during the production operation process;

[0007] Step S2: Based on the time-series chain operation data collected from multiple work areas, a total return prediction model for collaborative scheduling of material distribution in multiple work areas is constructed based on the long short-term memory network and trained using the training data set and the verification data set obtained in step 1 to realize the total production cycle generated by executing scheduling decisions at different decision-making stages and states in the collaborative scheduling process of material distribution in multiple work areas and the number of times that the wafer lot violates the residence time constraint in the queue of each work area;

[0008] Step S3: by defining and designing the multi-operation area state space of the wafer manufacturing system, the multi-operation area action space of the wafer manufacturing system, and the multi-operation area reward function of the wafer manufacturing system, a multi-operation area material distribution collaborative scheduling environment, namely, a Markov decision process, is established to transform the multi-operation area scheduling problem into a sequential decision problem and model it;

[0009] Step S4: constructing a self-regulating agent Agent0, a diffusion area scheduling agent Agent1, an oxidation area scheduling agent Agent2, a lithography area scheduling agent Agent3, an etching area scheduling agent Agent4 and an injection area scheduling agent Agent5 based on a deep neural network;

[0010] Step S5: The multi-agent and multi-operation area material distribution collaborative scheduling environment interacts with each other through an adaptive action selection mechanism to form and collect historical experience data, the multi-agent learning self-collaborative strategy is trained through a predictive self-collaborative mechanism, and the Double DQN algorithm is used to maximize the total return to train the agent to learn the optimal decision-making strategy;

[0011] Step S6: deploying the wafer multi-operation area scheduling agent model to the wafer manufacturing system multi-operation area material distribution collaborative scheduling system, monitoring the operating status in real time, generating and executing multi-operation area scheduling decisions, specifically including the following steps:

[0012] By deploying the pre-trained multi-operation area material distribution collaborative scheduling total reward prediction model and the self-regulating agent Agent0, the diffusion operation area scheduling agent Agent1, the oxidation operation area scheduling agent Agent2, the lithography operation area scheduling agent Agent3, the etching operation area scheduling agent Agent4 and the injection operation area scheduling agent Agent5 to the multi-operation area material distribution collaborative scheduling system of the wafer manufacturing system, the state characteristic parameters in the state space of the multi-operation area of ​​the wafer manufacturing system are observed in real time, and the self-regulating and self-collaborative decision-making schemes are adaptively generated with the idleness of the equipment group as the trigger to carry out the multi-operation area material distribution collaborative scheduling of the wafer manufacturing system.

[0013] Preferably, in step S1, the multi-operation area historical operation chain operation data samples include:

[0014] Diffusion operation area:

[0015] Oxidation work area:

[0016] Photolithography work area:

[0017] Etching work area:

[0018] Injection operation area:

[0019] Where: s *，t Indicates the state of the working area* at time t; a *，t represents the decision-making action of the operation area at time t; r *，t It represents the immediate reward of the action executed in the work area* at time t; Indicates the termination status of the operation area; G * represents the total return of the operation area, γ represents the reward discount factor.

[0020] Preferably, in step S3, the wafer manufacturing system multi-operation area state space S=S0∩S1∩S2∩S3∩S4∩S5, including a self-regulating sub-state space S0, a diffusion operation area sub-state space S1, a thin film operation area sub-state space S2, a lithography operation area sub-state space S3, an etching operation area sub-state space S4, and an injection operation area sub-state space S5;

[0021] The multi-operation area action space A of the wafer manufacturing system = A0∩A1∩A2∩A3∩A4∩A5, including a self-regulating sub-action space A0, a diffusion operation area sub-action space A1, a thin film operation area sub-action space A2, a photolithography operation area sub-action space A3, an etching operation area sub-action space A4, and an implantation operation area sub-action space A5;

[0022] The multi-operation area reward function R=R0∩R1∩R2∩R3∩R4∩R5 of the wafer manufacturing system includes a self-regulating sub-reward function R0, a diffusion operation area sub-reward function R1, a thin film operation area sub-reward function R2, a lithography operation area sub-reward function R3, an etching operation area sub-reward function R4, and an injection operation area sub-action space R5.

[0023] Preferably, the self-regulating sub-state space A0={0, 1} includes two actions: adjusting the wafer lot priority in the operation area sub-state space and not adjusting the wafer lot priority in the operation area sub-state space, wherein 0 indicates adjustment and 1 indicates no adjustment;

[0024] The dimensions of the diffusion operation area sub-state space A1={1, 2, ..., m1} and the thin film operation area sub-state space A2={1, 2, ..., m2} are the number of wafer process parameter types m1 and m2 in the waiting queues of the diffusion operation area and the thin film operation area, and the specific values ​​correspond to the corresponding wafer process parameter types respectively;

[0025] The dimensions of the lithography work area sub-state space A3 = {1, 2, ..., m3}, the etching work area sub-state space A4 = {1, 2, ..., m4}, and the injection work area sub-action space A5 = {1, 2, ..., m5} are the number of wafer lots m3, m4 and m5 in the waiting queues of the lithography work area, the etching work area, and the injection work area, and the specific values ​​correspond to the corresponding wafer lots respectively.

[0026] Preferably, the self-regulatory sub-reward function r 0，i =-(r 1，i +r 2，i +r 3，i +r 4，i +r 5，i ) is the immediate reward for self-regulation at time i, r 1，i 、r 2，i 、r 3，i 、r 4，i 、r 5，i They are the instant rewards at time i for the diffusion, thin film, lithography, and implantation operation areas, respectively, and T0 is the total cycle of self-regulation decision-making;

[0027] The diffusion operation area sub-reward function B is a collection of wafer lots with the same process parameters. i is the arrival time of wafer lot i, e ij(k-1)5l is the end time of the k-1th re-entry of wafer lot i layer on equipment group l in the implantation operation area, x ijk1lis the kth re-entry time of wafer lot i, layer j, delivered to the diffusion operation area and started processing on equipment group l, T1 is the total decision cycle of the diffusion operation area;

[0028] The film operating area sub-reward function P c1 is the number of wafer lots in the waiting queue of the thin film operation area that violate the time constraint c1, B is the set of wafer lots with the same process parameters, x ijk2l is the time when the kth re-entry of the wafer lot i, layer j, is delivered to the thin film operation area and starts processing on the equipment group l, e ijk2l is the kth re-entry delivery of wafer lot i, layer j, to the diffusion operation area and the end processing time on equipment group l, T2 is the total decision cycle of the thin film operation area;

[0029] The photolithography area sub-reward function r 3，i =-(P c2 +(x ijk3l -e ijk2l )), P c2 is the number of wafer lots in the waiting queue of the lithography operation area that violate the time constraint c2, e ijk2l is the processing time of the kth re-entry of wafer lot i, layer j, to the thin film operation area, and the end time of equipment group l, x ijk3l is the kth re-entry time of wafer lot i, layer j, delivered to the photolithography work area and started processing on equipment group l, T3 is the total decision cycle of the photolithography work area;

[0030] The etching operation area sub-reward function r 4，i =-(P c2 +(x ijk4l -e ijk3l )), P c3 is the number of wafer lots in the etching operation area waiting queue that violates the time constraint c3, e ijk3l is the processing time of the kth re-entry of wafer lot i, layer j, to the photolithography operation area, and the end time of equipment group l, x ijk4l is the kth re-entry time of wafer lot i, layer j, to the etching operation area and the start processing time on equipment group l, T4 is the total decision cycle of the etching operation area;

[0031] The injected operation area sub-reward function r 5，i =-(x ijk5l -e ijk4l ), e ijk4l is the processing time of the kth re-entry of wafer loti to the etching operation area by equipment group l, xijk5l is the kth time that wafer lot i, layer j, is re-delivered to the implantation operation area and starts processing on equipment group l. T5 is the total decision cycle of the implantation operation area.

[0032] Preferably, in step S4, the input and output dimensions of the self-regulating agent Agent0 are the dimensions of the sub-state space S0 and the sub-action space A0, respectively; the input and output dimensions of the diffusion operation area scheduling agent Agent1 are the dimensions of the sub-state space S1 and the sub-action space A1, respectively; the input and output dimensions of the thin film operation area scheduling agent Agent2 are the dimensions of the sub-state space S2 and the sub-action space A2, respectively; the input and output dimensions of the lithography operation area scheduling agent Agent3 are the dimensions of the sub-state space S3 and the sub-action space A3, respectively; the input and output dimensions of the etching operation area scheduling agent Agent4 are the dimensions of the sub-state space S4 and the sub-action space A4, respectively; the input and output dimensions of the injection operation area scheduling agent Agent5 are the dimensions of the sub-state space S5 and the sub-action space A5, respectively.

[0033] Preferably, in step S5, the adaptive action selection mechanism specifically includes:

[0034] According to the action space and decision-making characteristics of different intelligent agents, an adaptive action selection mechanism is designed. Among them, the action selection mechanism of the self-regulating intelligent agent is an ε-greedy strategy, which selects the action with the largest value function when the random probability is less than ε, and randomly selects the action when the random probability is greater than or equal to ε.

[0035] The action selection mechanism of the scheduling agent in the diffusion operation area and the scheduling agent in the film operation area is an adaptive action selection mechanism. In each decision stage, the agent determines the optional process parameter types in the waiting queue and forms a one-dimensional 0-1 vector of length m1 and a one-dimensional 0-1 vector of length m2, respectively. 0 indicates that the current process parameter type is not selectable, and 1 indicates that the current process parameter type is selectable. Then, the selection probability corresponding to the unselectable process parameter type is set to -∝ through the mask mechanism to avoid being selected by the agent and the optional process parameter type is adaptively selected in combination with the ε-greedy strategy.

[0036] The action selection mechanism of the lithography area scheduling agent, the etching area scheduling agent and the injection area scheduling agent is an adaptive action selection mechanism. In each decision stage, the agent determines the optional wafer lot in the waiting queue and forms a one-dimensional 0-1 vector of length m3, m4, and m5 respectively, where 0 indicates that the current wafer lot is not selectable and 1 indicates that the current wafer lot is selectable. Then, the selection probability corresponding to the unselectable wafer lot is set to -∝ through the mask mechanism to avoid being selected by the agent and the ε-greedy strategy is combined to adaptively select the optional wafer lot.

[0037] Preferably, the predictive self-coordination mechanism in step S5 specifically includes two parts: an optimal self-regulation strategy and an optimal self-coordination scheduling strategy, wherein:

[0038] The optimal self-regulation strategy is that the self-regulating agent takes the state information of each work area, the action information of each work area, the reward information of each work area, the state information of each work area after transfer, and the congestion and hunger information of each buffer as input, obtains the predicted value of the global performance and the number of violations of the residence time constraint through the total reward prediction model of the material distribution collaborative scheduling, and integrates the state information of each work area and the queue information of each buffer to learn the adaptive state feature regulation strategy;

[0039] The optimal self-collaborative scheduling strategy is to obtain the predicted values ​​of the global performance and the number of violations of the residence time constraint through the total reward prediction model of the material distribution collaborative scheduling. The state feature information in the diffusion operation area sub-state space S1, the thin film operation area sub-state space S2, the lithography operation area sub-state space S3, the etching operation area sub-state space S4, and the injection operation area sub-state space S5 are integrated respectively. The optimal self-collaborative scheduling strategy is adaptively learned by end-to-end training of the intelligent agent to minimize the total completion time and the total number of violations of the residence time constraints.

[0040] Preferably, in step S5, the Double DQN algorithm selects actions and estimates total rewards through the policy network based on the DQN algorithm, determines the action value through the evaluation network, and updates the error calculation method to solve the over-estimation problem, wherein the error calculation formula is: Among them, r is the reward function, γ is the reward discount coefficient, To evaluate the network at parameter θ - Q(s, a; θ) is the Q value of the action with the largest Q value in state s′ calculated by the value network, and Q(s, a; θ) is the Q value of action a selected in state s estimated by the value network under parameter θ.

[0041] Another technical solution of the present invention is to provide a wafer manufacturing system multi-operation area material distribution collaborative scheduling optimization system, which is characterized by comprising:

[0042] Wafer manufacturing system multi-operation area production operation scheduling controller and predictive self-collaborative reinforcement learning model, where:

[0043] The predictive self-collaborative reinforcement learning model is the trained predictive self-collaborative reinforcement learning model for multiple operation areas of a wafer manufacturing system in the above-mentioned collaborative scheduling method for material distribution in multiple operation areas of a wafer manufacturing system;

[0044] The wafer manufacturing system multi-operation area production operation scheduling controller outputs the state feature information of the self-regulating sub-state space S0, the diffusion operation area sub-state space S1, the thin film operation area sub-state space S2, the lithography operation area sub-state space S3, the etching operation area sub-state space S4, and the injection operation area sub-state space S5 of the wafer manufacturing system at the current moment according to the actual wafer manufacturing system multi-operation area material distribution collaborative scheduling environment, and transmits it to the predictive self-collaborative reinforcement learning model, and the predictive self-collaborative reinforcement learning model outputs the optimal wafer manufacturing system's self-regulating sub-state space A0, the diffusion operation area sub-action space A1, the thin film operation area sub-action space A2, the lithography operation area sub-action space A3, the etching operation area sub-action space A4, and the injection operation area sub-action space A5 collaborative scheduling action according to the input state information, and returns the optimal scheduling action to the wafer manufacturing system multi-operation area production operation scheduling controller;

[0045] The multi-operation area production operation scheduling controller of the wafer manufacturing system receives the optimal dispatching action and executes the multi-operation area scheduling action; after executing the scheduling action, the multi-operation area production operation scheduling controller of the wafer manufacturing system monitors the operating status of the multi-operation area equipment group. When the equipment group completes the wafer lot processing, the multi-operation area production operation scheduling controller of the wafer manufacturing system observes the multi-operation area material distribution collaborative scheduling environment of the wafer manufacturing system, and transmits the state feature information of the current self-regulation sub-state space S0, diffusion operation area sub-state space S1, thin film operation area sub-state space S2, lithography operation area sub-state space S3, etching operation area sub-state space S4, and injection operation area sub-state space S5 to the multi-operation area predictive self-collaborative reinforcement learning model of the wafer manufacturing system and obtains a new optimal scheduling action.

[0046] The present invention trains reinforcement learning agents to learn the optimal self-cooperative scheduling optimization strategy to perform fast and effective scheduling optimization of multiple operation areas of a wafer manufacturing system. Compared with the existing technical solutions, the present invention has the following technical effects:

[0047] (1) Aiming at the problem of coordinated scheduling optimization of material distribution in multiple work areas of wafer manufacturing system, a method for coordinated scheduling optimization of material distribution in multiple work areas of wafer manufacturing system based on predictive self-cooperative reinforcement learning is proposed. The optimal scheduling decision strategy is learned to quickly and effectively generate a coordinated scheduling scheme for multiple work areas with the least total completion time and the least number of violations of residence time constraints.

[0048] (2) Collect the sequential chain operation data of the wafer manufacturing system, build a total return prediction model for the coordinated scheduling of material distribution in multiple operation areas based on the long short-term memory network, infer the relationship between operation status, decision-making action and optimization index, assist in reinforcement learning intelligent decision-making and improve algorithm performance;

[0049] (3) Design the material distribution collaborative scheduling state space of multiple operation areas of the wafer manufacturing system to accurately characterize the production and operation status of multiple operation areas of the wafer manufacturing system;

[0050] (4) Design an adaptive action selection mechanism for collaborative scheduling of material distribution in multiple work areas of the wafer manufacturing system to further enhance the agent’s ability to explore and optimize the scheduling solution space of multiple work areas, and obtain scheduling solutions with better performance;

[0051] (5) Design a predictive self-cooperative mechanism for collaborative scheduling of material distribution in multiple work areas of the wafer manufacturing system. Fuse the real-time feedback information of the reward prediction model with the congestion and hunger information of the waiting queues in each area to train the self-regulating intelligent agent to learn the optimal self-regulating strategy. Fuse the real-time feedback information of the reward prediction model with the priority control decision to achieve self-cooperative scheduling optimization of the intelligent agents in each area and improve the overall performance of the algorithm.

[0052] (6) A reactive collaborative scheduling system for material distribution in multiple work areas of a wafer manufacturing system based on predictive self-collaborative reinforcement learning is provided. The system uses a controller to observe the status information of multiple work areas of the wafer manufacturing system in real time. When the processing capacity of the equipment group is idle, it observes the environmental status and uses a pre-trained collaborative scheduling model for material distribution in multiple work areas to output the optimal scheduling action, which greatly improves the operating efficiency and production performance of the wafer manufacturing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is an implementation flow chart of a reactive collaborative scheduling optimization method for material distribution in multiple operation areas of a wafer manufacturing system in the present invention;

[0054] Figure 2 It is a flow chart of coordinated scheduling of material distribution in multiple operation areas of the wafer manufacturing system in the present invention;

[0055] Figure 3 A framework diagram of a reactive collaborative scheduling optimization method for material distribution in multiple operation areas of a wafer manufacturing system in the present invention;

[0056] Figure 4 is a diagram illustrating the adaptive action selection mechanism in the present invention;

[0057] Figure 5 This is a diagram illustrating the multi-agent model structure and Double DQN learning algorithm in the present invention;

[0058] Figure 6 It is a structural block diagram of a reactive collaborative scheduling system for material distribution in multiple work areas of a wafer manufacturing system in the present invention. DETAILED DESCRIPTION

[0059] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0060] The present invention provides a reactive collaborative scheduling method and system for material distribution in multiple operation areas of a wafer manufacturing system, and the implementation process is as follows: Figure 1 As shown, the following steps are included:

[0061] Step S1: Collect and process historical production operation data of multiple operating areas of a wafer manufacturing system to form a training data set and a verification data set.

[0062] The historical production operation data of multiple operation areas in the wafer manufacturing system includes the wafer lots currently being processed in the multiple operation areas such as diffusion, oxidation, lithography, etching, and injection in the wafer manufacturing system, the operation status of the equipment group, the wafer lots in the waiting queue, the total production cycle corresponding to the scheduling plan, and the total number of times the wafer lot violates the residence time constraint of the operation area. Through data cleaning and processing, a multi-operation area chain data sample is formed. The chain data is linked by the current production operation status, scheduling decision actions, immediate rewards for executing decision actions, and production operation status updated after executing actions in each operation area during the production operation process. The historical operation chain operation data samples of multiple operation areas are as follows:

[0063] Diffusion operation area:

[0064] Oxidation work area:

[0065] Photolithography work area:

[0066] Etching work area:

[0067] Injection operation area:

[0068] Where: s *，t Indicates the state of the working area* at time t; a *，t represents the decision-making action of the operation area at time t; r *，t It represents the immediate reward of the action executed in the work area* at time t; Indicates the termination status of the operation area; G * represents the total return of the operation area, γ represents the reward discount factor.

[0069] Step S2: Build and train a total return prediction model for collaborative scheduling of material distribution in multiple work areas based on a deep neural network to realize the total production cycle generated by executing scheduling decisions at different decision-making stages and states during the collaborative scheduling of material distribution in multiple work areas and the number of times a wafer lot violates the residence time constraint in each work area queue.

[0070] Based on the time series chain operation data collected from multiple operation areas such as diffusion, oxidation, lithography, etching, and injection, a total return prediction model for material distribution collaborative scheduling in multiple operation areas is constructed based on the long short-term memory network. The long short-term memory network is mainly composed of structures such as forget gate, input gate, and output gate. Among them, the forget gate sigmoid unit determines which information needs to be discarded in the cell state, which is determined by h t-1 and x t Information is used to output a vector between 0 and 1, where the 0-1 value in the vector represents the cell state G t-1 Which information is retained or discarded, 0 means not to retain, 1 means to retain all; the input gate is used to update the unit state, pass the hidden state and input to the sigmoid function to update the values ​​by converting the values ​​to 0 to 1, 0 means not important, 1 means important, pass the hidden state and input to the tanh function and compress them between -1 and 1 to help regulate the network; the output gate determines the next hidden state, passes the previous hidden state and input to the sigmoid function, passes the new unit state to the tanh function, and multiplies the outputs of the two to determine the information that the hidden state should carry. The multi-operation area total return prediction model based on the long short-term memory network is trained by the time-series chain operation data training and test data set described in step S1, so as to realize the accurate prediction of the total production cycle generated by executing the scheduling decision at different decision stages and states in the collaborative scheduling process of material distribution in multiple operation areas and the number of times that the wafer lot violates the residence time constraint in each operation area queue.

[0071] Step S3: Through the definition and design of state space, action space and reward function, a multi-operation area material distribution collaborative scheduling environment, namely the Markov decision process, is established to transform the multi-operation area scheduling problem into a sequential decision problem and model it.

[0072] The multi-operation area state space S of the wafer manufacturing system is S=S0∩S1∩S2∩S3∩S4∩S5, including the self-regulating sub-state space S0, the diffusion operation area sub-state space S1, the thin film operation area sub-state space S2, the lithography operation area sub-state space S3, the etching operation area sub-state space S4, and the injection operation area sub-state space S5. The self-regulating sub-state space S0 includes, but is not limited to, the total production cycle and the predicted value of the number of violations of the retention time constraint output by the total return prediction model, the equipment and the wafer status of the current operation area and the upstream and downstream operation areas, and the priority information of the wafers waiting in the current operation area. The diffusion operation area sub-state space S1 and the thin film operation area sub-state space S2 include but are not limited to the predicted values ​​of the total production cycle and the number of residence time constraint violations output by the total return prediction model, the operation area type, the current processing capacity, the batch processing machine capacity, the process parameter type, the current number of processing layers, the current number of re-entries, the equipment operation status, the start processing time, the end processing time, the remaining processing time, the equipment group load, the number of wafers in the queue, the wafer process parameter type, the wafer processing time, the number of wafer layers processed, the number of wafer re-entries, the remaining processing time, the number of wafers being processed in the previous area, the process parameter type priority and other status characteristic information. The sub-state space S3 of the lithography work area, the sub-state space S4 of the etching work area, and the sub-state space S5 of the injection work area include but are not limited to the predicted values ​​of the total production cycle and the number of residence time constraint violations output by the total return prediction model, the type of work area, the number of equipment groups, the current number of processing layers, the current number of re-entries, the equipment operating status, the start processing time, the end processing time, the remaining processing time, the equipment group load, the wafer lot priority, the number of wafers in the queue, the wafer waiting time, the wafer processing time, the number of wafer processed layers, the number of wafer re-entries, the remaining processing time, the number of wafers being processed in the previous area, the wafer release time in the previous area, the wafer arrival time, the waiting time, the wafer lithography time, the end time of the previous lithography, the previous lithography equipment group and other status characteristic information.

[0073] The multi-operation area action space A=A0∩A1∩A2∩A3∩A4∩A5 of the wafer manufacturing system includes the self-regulating sub-action space A0, the diffusion operation area sub-action space A1, the thin film operation area sub-action space A2, the lithography operation area sub-action space A3, the etching operation area sub-action space A4, and the implantation operation area sub-action space A5. The self-regulating sub-state space A0={0,1} includes two actions: adjusting the wafer lot priority in the operation area sub-state space and not adjusting the wafer lot priority in the operation area sub-state space, where 0 represents adjustment and 1 represents no adjustment. The dimensions of the diffusion operation area sub-state space A1={1,2,...,m1} and the thin film operation area sub-state space A2={1,2,...,m2} are the number of wafer process parameter types m1 and m2 in the waiting queues of the diffusion operation area and the thin film operation area, and the specific values ​​correspond to the corresponding wafer process parameter types. The dimensions of the lithography work area sub-state space A3 = {1, 2, ..., m3}, the etching work area sub-state space A4 = {1, 2, ..., m4}, and the injection work area sub-action space A5 = {1, 2, ..., m5} are the number of wafer lots m3, m4 and m5 in the waiting queues of the lithography work area, the etching work area, and the injection work area, and the specific values ​​correspond to the corresponding wafer lots respectively.

[0074] The wafer manufacturing system multi-operation area reward function R = R0 ∩ R1 ∩ R2 ∩ R3 ∩ R4 ∩ R5, including the self-regulating sub-reward function R0, the diffusion operation area sub-reward function R1, the thin film operation area sub-reward function R2, the lithography operation area sub-reward function R3, the etching operation area sub-reward function R4, and the injection operation area sub-action space R5. Self-regulating sub-reward function where r 0，i =-(r 1，i +r 2，i +r 3，i +r 4，i +r 5，i ) is the immediate reward for self-regulation at time i, r 1，i 、r 2，i 、r 3，i 、r 4，i 、r 5，i are the instant rewards for diffusion, thin film, lithography, and injection work areas at time i, respectively, and T0 is the total cycle of self-regulation decision making. Sub-reward function for diffusion work area in B is a collection of wafer lots with the same process parameters. i is the arrival time of wafer lot i, e ij(k-1)5l is the end time of the k-1th re-entry of wafer loti layer in the equipment group l in the implantation operation area, x ijk1lis the kth re-entry time of wafer lot i, layer j, to the diffusion operation area and the start processing time on equipment group l, T1 is the total decision cycle of the diffusion operation area; the sub-reward function of the thin film operation area in P c1 is the number of wafer lots in the waiting queue of the thin film operation area that violate the time constraint c1, B is the set of wafer lots with the same process parameters, x ijk2l is the time when the kth re-entry of the wafer lot i, layer j, is delivered to the thin film operation area and starts processing on the equipment group l, e ijk2l is the kth re-entry time of wafer lot i, layer j, to the diffusion operation area and the end processing time on equipment group l, T2 is the total decision cycle of the film operation area; sub-reward function of the lithography operation area where r 3，i =-(P c2 +(x ijk3l -e ijk2l )), P c2 is the number of wafer lots in the waiting queue of the lithography operation area that violate the time constraint c2, e ijk2l is the processing time of the kth re-entry of wafer lot i, layer j, to the thin film operation area, and the end time of equipment group l, x ijk3l is the kth re-entry time of wafer lot i, layer j, to the lithography operation area and the start processing time on equipment group l, T3 is the total decision cycle of the lithography operation area; the sub-reward function of the etching operation area where r 4，i =-(P c2 +(x ijk4l -e ijk3l )), P c3 is the number of wafer lots in the etching operation area waiting queue that violates the time constraint c3, e ijk3l is the processing time of the kth re-entry of wafer lot i, layer j, to the photolithography operation area, and the end time of equipment group l, x ijk4l is the kth re-entry time of wafer lot i, layer j, to the etching operation area and the start processing time on equipment group l, T4 is the total decision cycle of the etching operation area; injection operation area sub-reward function where r 5，i =-(x ijk5l -e ijk4l ), e ijk4l is the processing time of the kth re-entry of wafer lot i, layer j, to the etching operation area, and the end time of equipment group l, x ijk5l is the kth time that wafer lot i, layer j, is re-delivered to the implantation operation area and starts processing on equipment group l. T5 is the total decision cycle of the implantation operation area.

[0075] Step S4: Build a self-coordinating and diffusion, oxidation, lithography, etching, injection and other work area scheduling intelligent agent based on the deep neural network.

[0076] Aiming at the demand for collaborative scheduling decision of material distribution in multiple work areas of wafer manufacturing system based on predictive self-cooperative reinforcement learning, self-regulating Agent0 and work area scheduling agents such as diffusion Agent1, oxidation Agent2, lithography Agent3, etching Agent4, and injection Agent5 are constructed based on fully connected neural network. The input and output dimensions of self-regulating agent Agent0 are the dimensions of sub-state space S0 and sub-action space A0 respectively. The input and output dimensions of diffusion work area scheduling agent Agent1 are the dimensions of sub-state space S1 and sub-action space A1 respectively. The input and output dimensions of thin film work area scheduling agent Agent2 are the dimensions of sub-state space S2 and sub-action space A2 respectively. The input and output dimensions of lithography work area scheduling agent Agent3 are the dimensions of sub-state space S3 and sub-action space A3 respectively. The input and output dimensions of etching work area scheduling agent Agent4 are the dimensions of sub-state space S4 and sub-action space A4 respectively. The input and output dimensions of the injection work area scheduling agent Agent5 are the dimensions of the sub-state space S5 and the sub-action space A5 respectively.

[0077] Step S5: Through the adaptive action selection mechanism, multiple agents interact with the collaborative scheduling environment of material distribution in multiple work areas to form and collect historical experience data, and through the predictive self-collaborative mechanism, multiple agents are trained to learn self-collaborative strategies. The Double DQN algorithm is used to maximize the total reward to train the agents to learn the optimal decision-making strategy.

[0078] According to the action space and decision-making characteristics of different agents, an adaptive action selection mechanism is designed to improve the agent's ability to explore the solution space. The action selection mechanism includes the action selection mechanism of self-regulating agent, diffusion area scheduling agent, thin film area scheduling agent, lithography area scheduling agent, etching area scheduling agent, and injection area scheduling agent. The action selection mechanism of the self-regulating agent is an ε-greedy strategy. When the random probability is less than ε, the action with the largest value function is selected, and when the random probability is greater than or equal to ε, the action is randomly selected. The action selection mechanism of the diffusion and thin film area scheduling agent is an adaptive action selection mechanism. At each decision stage, the agent determines the optional process parameter type in the waiting queue and forms a one-dimensional 0-1 vector of length m1 and a one-dimensional 0-1 vector of length m2, respectively, where 0 indicates that the current process parameter type is not selectable and 1 indicates that the current process parameter type is selectable. Subsequently, the selection probability corresponding to the unselectable process parameter type is set to -∝ through the mask mechanism to avoid being selected by the agent and the optional process parameter type is adaptively selected in combination with the ε-greedy strategy. The action selection mechanism of the scheduling agent in the lithography, etching, and implantation work areas is an adaptive action selection mechanism. In each decision stage, the agent determines the optional wafer lot in the waiting queue and forms a one-dimensional 0-1 vector of length m3, m4, and m5, respectively, where 0 indicates that the current wafer lot is not selectable and 1 indicates that the current wafer lot is selectable. Then, the selection probability corresponding to the unselectable wafer lot is set to -∝ through the mask mechanism to avoid being selected by the agent and the optional wafer lot is adaptively selected in combination with the ε-greedy strategy.

[0079] The predictive self-cooperative mechanism includes two parts: the optimal self-regulation strategy and the optimal self-cooperative scheduling strategy. The optimal self-regulation strategy is that the self-regulating agent takes the state information of each work area, the action information of each work area, the reward information of each work area, the state information after the transfer of each work area, and the congestion and hunger information of each buffer as input, obtains the predicted value of the global performance and the number of violations of the residence time constraint through the total reward prediction model of the material distribution collaborative scheduling, and integrates the state information of each work area and the queue information of each buffer to learn the adaptive state feature regulation strategy. The optimal self-cooperative scheduling strategy is that the scheduling agent of the diffusion, oxidation, lithography, etching, injection and other work areas obtains the predicted value of the global performance and the number of violations of the residence time constraint through the total reward prediction model of the material distribution collaborative scheduling, and integrates the state feature information of the diffusion work area sub-state space S1, the thin film work area sub-state space S2, the lithography work area sub-state space S3, the etching work area sub-state space S4, and the injection work area sub-state space S5, respectively, and uses the end-to-end training of the agent to adaptively learn the optimal self-cooperative scheduling strategy to minimize the total completion time and the total number of violations of the residence time constraint.

[0080] The Double DQN algorithm selects actions and estimates the total reward through the policy network based on the DQN algorithm, determines the action value through the evaluation network, and updates the error calculation method to solve the over-estimation problem. The error calculation formula is Among them, r is the reward function, γ is the reward discount coefficient, To evaluate the network at parameter θ - Q(s, a; θ) is the Q value of the action with the largest Q value in state s′ calculated by the value network, and Q(s, a; θ) is the Q value of action a selected in state s estimated by the value network under parameter θ.

[0081] Step S6: Integrate the wafer multi-operation area scheduling intelligent agent model into the wafer manufacturing system multi-operation area material distribution collaborative scheduling system to generate and execute scheduling decisions online.

[0082] By deploying the pre-trained multi-operation area material distribution collaborative scheduling total reward prediction model and self-regulation Agent0 and diffusion Agent1, oxidation Agent2, lithography Agent3, etching Agent4, injection Agent5 and other operation area scheduling agents into the wafer manufacturing multi-operation area material distribution collaborative scheduling system, the controller observes the multi-operation area status information of the wafer manufacturing system in real time, observes the environmental status when the processing capacity of the equipment group is idle, and uses the pre-trained multi-operation area material distribution collaborative scheduling model to output and execute the optimal scheduling action.

[0083] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A reactive collaborative scheduling optimization method for material distribution in multiple work areas of a wafer manufacturing system, characterized in that: The following steps are involved: Step S1: Collect and process the historical production operation data of multiple operation areas of the wafer manufacturing system to form a training data set and a verification data set. In the training data set and the verification data set, a historical operation chain operation data sample of multiple operation areas is formed through data cleaning and processing, wherein the chain data is respectively linked by the current production operation status, scheduling decision actions, immediate rewards obtained by executing decision actions, and production operation status updated after executing actions in each operation area during the production operation process; Step S2: Based on the time-series chain operation data collected from multiple work areas, a total return prediction model for collaborative scheduling of material distribution in multiple work areas is constructed based on the long short-term memory network and trained using the training data set and the verification data set obtained in step 1 to realize the total production cycle generated by executing scheduling decisions at different decision-making stages and states in the collaborative scheduling process of material distribution in multiple work areas and the number of times that the wafer lot violates the residence time constraint in the queue of each work area; Step S3: by defining and designing the multi-operation area state space of the wafer manufacturing system, the multi-operation area action space of the wafer manufacturing system, and the multi-operation area reward function of the wafer manufacturing system, a multi-operation area material distribution collaborative scheduling environment, namely, a Markov decision process, is established to transform the multi-operation area scheduling problem into a sequential decision problem and model it; The multi-operation area reward function R=R0∩R1∩R2∩R3∩R4∩R5 of the wafer manufacturing system includes a self-regulating sub-reward function R0, a diffusion operation area sub-reward function R1, a thin film operation area sub-reward function R2, a lithography operation area sub-reward function R3, an etching operation area sub-reward function R4, and an injection operation area sub-action space R5; The self-regulatory reward function r 0,i =-(r 1,i +r 2,i +r 3,i +r 4,i +r 5,i ) is the immediate reward for self-regulation at time i, r 1,i 、r 2,i 、r 3,i 、r 4,i 、r 5,i They are the instant rewards at time i for the diffusion, thin film, lithography, and implantation operation areas, respectively, and T0 is the total cycle of self-regulation decision-making; The diffusion operation area sub-reward function B is a collection of wafer lots with the same process parameters. i is the arrival time of wafer loti, e ij(k-1)5l is the end time of the k-1th re-entry of wafer loti layer in the equipment group l in the implantation operation area, x ijk1l is the kth re-entry time of wafer loti jth layer delivered to the diffusion operation area and the start processing time on equipment group l, T1 is the total decision cycle of the diffusion operation area; The film operating area sub-reward function P c1 is the number of wafer lots in the waiting queue of the thin film operation area that violate the time constraint c1, B is the set of wafer lots with the same process parameters, x ijk2l is the time when the kth re-entry of wafer loti layer j is delivered to the thin film operation area and starts processing on equipment group l, e ijk2l is the kth re-entry delivery of the jth layer of wafer loti to the diffusion operation area and the end processing time on the lth equipment group, T2 is the total decision cycle of the thin film operation area; The photolithography area sub-reward function r 3,i =-(P c2 +(x ijk3l -e ijk2l )), P c2 is the number of wafer lots in the waiting queue of the lithography operation area that violate the time constraint c2, e ijk2l is the processing time of the kth re-entry of wafer loti to the thin film operation area by equipment group l, x ijk3l The kth re-entry time of wafer loti jth layer delivered to the photolithography work area and the start processing time on the lth equipment group, T3 is the total decision cycle of the photolithography work area; The etching operation area sub-reward function r 4,i =-(P c2 +(x ijk4l -e ijk3l )), P c3 is the number of wafer lots in the etching operation area waiting queue that violates the time constraint c3, e ijk3l is the processing time of the kth re-entry of wafer loti to the photolithography operation area, and the end time of equipment group l, x ijk4l is the kth re-entry time of wafer loti jth layer to the etching operation area and the start processing time on equipment group l, T4 is the total decision cycle of the etching operation area; The injected operation area sub-reward function r 5,i =-(x ijk5l -e ijk4l ), e ijk4l is the processing time of the kth re-entry of wafer loti to the etching operation area by equipment group l, x ijk5l is the kth re-entry time of wafer loti jth layer delivered to the implantation operation area and started processing on equipment group l, T5 is the total decision cycle of the implantation operation area; Step S4: constructing a self-regulating agent Agent0, a diffusion area scheduling agent Agent1, an oxidation area scheduling agent Agent2, a lithography area scheduling agent Agent3, an etching area scheduling agent Agent4 and an injection area scheduling agent Agent5 based on a deep neural network; Step S5: The multi-agent and multi-operation area material distribution collaborative scheduling environment interacts with each other through an adaptive action selection mechanism to form and collect historical experience data, the multi-agent learning self-collaborative strategy is trained through a predictive self-collaborative mechanism, and the Double DQN algorithm is used to maximize the total return to train the agent to learn the optimal decision-making strategy; Step S6: deploying the wafer multi-operation area scheduling agent model to the wafer manufacturing system multi-operation area material distribution collaborative scheduling system, monitoring the operating status in real time, generating and executing multi-operation area scheduling decisions, specifically including the following steps: By deploying the pre-trained multi-operation area material distribution collaborative scheduling total reward prediction model and the self-regulating agent Agent0, the diffusion operation area scheduling agent Agent1, the oxidation operation area scheduling agent Agent2, the lithography operation area scheduling agent Agent3, the etching operation area scheduling agent Agent4 and the injection operation area scheduling agent Agent5 to the multi-operation area material distribution collaborative scheduling system of the wafer manufacturing system, the state characteristic parameters in the state space of the multi-operation area of ​​the wafer manufacturing system are observed in real time, and the self-regulating and self-collaborative decision-making schemes are adaptively generated with the idleness of the equipment group as the trigger to carry out the multi-operation area material distribution collaborative scheduling of the wafer manufacturing system.

2. A reactive collaborative scheduling optimization method for material distribution in multiple work areas of a wafer manufacturing system as claimed in claim 1, characterized in that: In step S1, the multi-operation area historical operation chain operation data samples include: Diffusion operation area: Oxidation work area: Photolithography work area: Etching work area: Injection operation area: Where: s *,t Indicates the state of the working area* at time t; a *,t represents the decision-making action of the operation area at time t; r *,t represents the immediate reward of the action executed in the work area at time t; s *,T* Indicates the termination status of the operation area; G * represents the total return of the operation area, γ represents the reward discount factor.

3. A reactive collaborative scheduling optimization method for material distribution in multiple work areas of a wafer manufacturing system as claimed in claim 2, characterized in that: In step S3, the wafer manufacturing system multi-operation area state space S=S0∩S1∩S2∩S3∩S4∩S5, including the self-regulation sub-state space S0, the diffusion operation area sub-state space S1, the thin film operation area sub-state space S2, the lithography operation area sub-state space S3, the etching operation area sub-state space S4, and the injection operation area sub-state space S5; The multi-operation area action space A=A0∩A1∩A2∩A3∩A4∩A5 of the wafer manufacturing system includes a self-regulating sub-action space A0, a diffusion operation area sub-action space A1, a thin film operation area sub-action space A2, a lithography operation area sub-action space A3, an etching operation area sub-action space A4, and an injection operation area sub-action space A5.

4. A reactive collaborative scheduling optimization method for material distribution in multiple work areas of a wafer manufacturing system as claimed in claim 3, characterized in that: The self-regulating sub-state space A0={0,1} includes two actions: adjusting the wafer lot priority in the operation area sub-state space and not adjusting the wafer lot priority in the operation area sub-state space, where 0 represents adjustment and 1 represents no adjustment; The dimensions of the diffusion operation area sub-state space A1={1,2,…,m1} and the thin film operation area sub-state space A2={1,2,…,m2} are the number of wafer process parameter types m1 and m2 in the waiting queues of the diffusion operation area and the thin film operation area, respectively, and the specific values ​​correspond to the corresponding wafer process parameter types; The dimensions of the lithography work area sub-state space A3 = {1,2,…,m3}, the etching work area sub-state space A4 = {1,2,…,m4}, and the injection work area sub-action space A5 = {1,2,…,m5} are the number of wafer lots m3, m4 and m5 in the waiting queues of the lithography work area, the etching work area, and the injection work area, and the specific values ​​correspond to the corresponding wafer lots respectively.

5. A reactive collaborative scheduling optimization method for material distribution in multiple work areas of a wafer manufacturing system as claimed in claim 4, characterized in that: In step S4, the input and output dimensions of the self-regulating agent Agent0 are the dimensions of the sub-state space S0 and the sub-action space A0 respectively; the input and output dimensions of the diffusion operation area scheduling agent Agent1 are the dimensions of the sub-state space S1 and the sub-action space A1 respectively; the input and output dimensions of the thin film operation area scheduling agent Agent2 are the dimensions of the sub-state space S2 and the sub-action space A2 respectively; the input and output dimensions of the lithography operation area scheduling agent Agent3 are the dimensions of the sub-state space S3 and the sub-action space A3 respectively; the input and output dimensions of the etching operation area scheduling agent Agent4 are the dimensions of the sub-state space S4 and the sub-action space A4 respectively; the input and output dimensions of the injection operation area scheduling agent Agent5 are the dimensions of the sub-state space S5 and the sub-action space A5 respectively.

6. A reactive collaborative scheduling optimization method for material distribution in multiple work areas of a wafer manufacturing system as claimed in claim 5, characterized in that: In step S5, the adaptive action selection mechanism specifically includes: According to the action space and decision-making characteristics of different intelligent agents, an adaptive action selection mechanism is designed. Among them, the action selection mechanism of the self-regulating intelligent agent is an ε-greedy strategy, which selects the action with the largest value function when the random probability is less than ε, and randomly selects the action when the random probability is greater than or equal to ε. The action selection mechanism of the scheduling agent in the diffusion operation area and the scheduling agent in the film operation area is an adaptive action selection mechanism. In each decision stage, the agent determines the optional process parameter types in the waiting queue and forms a one-dimensional 0-1 vector of length m1 and a one-dimensional 0-1 vector of length m2, respectively. 0 indicates that the current process parameter type is not selectable, and 1 indicates that the current process parameter type is selectable. Then, the selection probability corresponding to the unselectable process parameter type is set to -∝ through the mask mechanism to avoid being selected by the agent and the optional process parameter type is adaptively selected in combination with the ε-greedy strategy. The action selection mechanism of the lithography area scheduling agent, the etching area scheduling agent and the injection area scheduling agent is an adaptive action selection mechanism. In each decision stage, the agent determines the optional wafer lot in the waiting queue and forms a one-dimensional 0-1 vector of length m3, m4, and m5 respectively, where 0 indicates that the current wafer lot is not selectable and 1 indicates that the current wafer lot is selectable. Then, the selection probability corresponding to the unselectable wafer lot is set to -∝ through the mask mechanism to avoid being selected by the agent and the ε-greedy strategy is combined to adaptively select the optional wafer lot.

7. The method for reactive collaborative scheduling optimization of material distribution in multiple work areas of a wafer manufacturing system as claimed in claim 1, characterized in that: The predictive self-coordination mechanism in step S5 specifically includes two parts: the optimal self-regulation strategy and the optimal self-coordination scheduling strategy, wherein: The optimal self-regulation strategy is that the self-regulating agent takes the state information of each work area, the action information of each work area, the reward information of each work area, the state information of each work area after transfer, and the congestion and hunger information of each buffer as input, obtains the predicted value of the global performance and the number of violations of the residence time constraint through the total reward prediction model of the material distribution collaborative scheduling, and integrates the state information of each work area and the queue information of each buffer to learn the adaptive state feature regulation strategy; The optimal self-collaborative scheduling strategy is to obtain the predicted values ​​of the global performance and the number of violations of the residence time constraint through the total reward prediction model of the material distribution collaborative scheduling. The state feature information in the diffusion operation area sub-state space S1, the thin film operation area sub-state space S2, the lithography operation area sub-state space S3, the etching operation area sub-state space S4, and the injection operation area sub-state space S5 are integrated respectively. The optimal self-collaborative scheduling strategy is adaptively learned by end-to-end training of the intelligent agent to minimize the total completion time and the total number of violations of the residence time constraints.

8. The reactive collaborative scheduling optimization method for material distribution in multiple work areas of a wafer manufacturing system as claimed in claim 1, characterized in that: In step S5, the Double DQN algorithm selects actions and estimates total rewards through the policy network based on the DQN algorithm, determines the action value through the evaluation network, and updates the error calculation method to solve the over-estimation problem, where the error calculation formula is: Among them, r is the reward function, γ is the reward discount coefficient, To evaluate the network at parameter θ - Q(s,a;θ) is the Q value of the action with the largest Q value in state s′ calculated under the value network, and Q(s,a;θ) is the Q value of action a selected in state s estimated by the value network under parameter θ.

9. A reactive collaborative scheduling optimization system for material distribution in multiple work areas of a wafer manufacturing system, characterized in that: include: Wafer manufacturing system multi-operation area production operation scheduling controller and predictive self-collaborative reinforcement learning model, where: The predictive self-collaborative reinforcement learning model is a wafer manufacturing system multi-operation area predictive self-collaborative reinforcement learning model that has been trained in the wafer manufacturing system multi-operation area material distribution collaborative scheduling method according to any one of claims 1 to 8; The wafer manufacturing system multi-operation area production operation scheduling controller outputs the state feature information of the self-regulating sub-state space S0, the diffusion operation area sub-state space S1, the thin film operation area sub-state space S2, the lithography operation area sub-state space S3, the etching operation area sub-state space S4, and the injection operation area sub-state space S5 of the wafer manufacturing system at the current moment according to the actual wafer manufacturing system multi-operation area material distribution collaborative scheduling environment, and transmits it to the predictive self-collaborative reinforcement learning model. The predictive self-collaborative reinforcement learning model outputs the material distribution collaborative scheduling action of the self-regulating sub-state space A0, the diffusion operation area sub-action space A1, the thin film operation area sub-action space A2, the lithography operation area sub-action space A3, the etching operation area sub-action space A4, and the injection operation area sub-action space A5 of the wafer manufacturing system according to the input state information, and returns the optimal scheduling action to the wafer manufacturing system multi-operation area production operation scheduling controller; The multi-operation area production operation scheduling controller of the wafer manufacturing system receives the optimal scheduling action and executes the multi-operation area scheduling action; after executing the scheduling action, the multi-operation area production operation scheduling controller of the wafer manufacturing system monitors the operating status of the multi-operation area equipment group. When the equipment group completes the wafer lot processing, the multi-operation area production operation scheduling controller of the wafer manufacturing system observes the multi-operation area material distribution collaborative scheduling environment of the wafer manufacturing system, and transmits the state feature information of the current self-regulation sub-state space S0, diffusion operation area sub-state space S1, thin film operation area sub-state space S2, lithography operation area sub-state space S3, etching operation area sub-state space S4, and injection operation area sub-state space S5 to the multi-operation area predictive self-collaborative reinforcement learning model of the wafer manufacturing system and obtains a new optimal scheduling action.

Images