Multi-agent collaborative semiconductor batch processing area scheduling method

Abstract

The invention relates to a multi-agent collaborative semiconductor batch processing area scheduling method, which integrates adversarial learning and a multi-agent collaborative mechanism, constructs a batch grouping agent and a batch assignment agent, and realizes joint optimization of batch combination and task assignment. A gating circulation unit network is introduced to extract dynamic characteristics in the workshop operation process, and the perception and environment adaptability of a scheduling strategy to time sequence information are enhanced. Real and virtual experience samples are generated through adversarial training for pre-training, training data are expanded, and the strategy convergence speed is improved. An internal correlation structure between relation correction network modeling experiences is further introduced, and experience quality and strategy updating efficiency are improved in combination with a relative entropy optimization mechanism. According to the method, the problem that wafer processing system scheduling is often difficult to meet quick response and efficient scheduling through a traditional reinforcement learning method is solved, the scheduling decision can be quickly and accurately generated, the total production cycle in the wafer manufacturing process is effectively shortened, and the method has high engineering application value.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of wafer manufacturing production scheduling, in particular to a multi-agent collaborative semiconductor batch processing area scheduling method enhanced by adversarial learning, which is used for production scheduling of wafer batch processing area workshop and scheduling optimization in semiconductor production process. BACKGROUND

[0002] China is the world's largest semiconductor manufacturing and exporting country, and the semiconductor chip manufacturing industry has a crucial influence on the national economy. In the field of semiconductor manufacturing, wafers are the basic units of chip production, and wafer manufacturing systems are advanced manufacturing systems with high complexity. This system is known for its huge resource demand, re-entrant production process, and high cost, and because of its mixed production of multiple varieties and large quantities in production, it has become one of the most complex and most expensive systems in manufacturing, attracting the attention of domestic and foreign scholars. The wafer manufacturing process covers five stages of wafer preparation, manufacturing, picking testing, chip assembly and packaging, and final testing. In particular, in the wafer manufacturing stage, it includes key steps such as oxidation, etching, lithography, ion implantation, metallization, and deposition. Among them, batch processing areas such as oxidation and deposition areas, due to their long processing cycle and complex constraints, often become the bottleneck link in wafer manufacturing. These areas not only occupy a large part of the time of the entire process, but also put higher requirements on material transportation and subsequent group scheduling, directly affecting the efficiency and performance of the entire wafer manufacturing system.

[0003] The current wafer processing system scheduling faces many challenges, especially the complexity and dynamic changes of the system, which makes traditional scheduling methods such as mathematical programming, heuristic algorithms, and meta-heuristic algorithms, although widely used in various industrial scenarios, often difficult to meet the specific needs of wafer processing systems, and lack of flexibility and rapid adaptation ability. Reinforcement learning provides a possibility to optimize scheduling decisions through continuous learning and adaptation to environmental changes, but due to insufficient sample size in the initial training phase and slow learning process, traditional reinforcement learning methods often fail to meet the demand for rapid response and efficient scheduling.

[0004] The present application aims to solve the key technical problems encountered in semiconductor wafer manufacturing: as production needs diversify and production environments change rapidly, traditional wafer manufacturing scheduling systems struggle to adapt to fluctuations in production line status and dynamic changes in orders in real time, resulting in slow scheduling response and low efficiency. Existing scheduling systems mainly rely on pre-set static planning and lack the ability to flexibly respond to complex production dynamics, especially when facing orders with high variety and short lead time, often failing to effectively optimize resource allocation and production order, thereby affecting the running efficiency and output quality of the entire production line. SUMMARY

[0005] In order to solve the problem that it is difficult to meet the rapid response and efficient scheduling by the traditional reinforcement learning method for wafer processing system scheduling, a semiconductor batch processing area scheduling method based on multi-agent cooperation is provided. A new scheduling method combining multi-agent cooperative reinforcement learning and generative adversarial network is introduced, and the GRU unit is shared by each agent to optimize the scheduling process, which not only significantly speeds up the learning speed and improves the generalization ability of the model, but also overcomes the shortcomings of traditional methods, providing an innovative solution for efficient scheduling in wafer manufacturing systems.

[0006] The technical scheme of the present application is:

[0007] A semiconductor batch processing area scheduling method based on multi-agent cooperation, comprising the following steps:

[0008] S1, obtain the processing task state data of the current batch processing area, machine state data; establish a deep reinforcement learning multiple scheduling module for wafer batch processing workshop scheduling, including batch grouping agent, batch assignment agent, generative adversarial network GAN, relationship correction module, and information interaction center GRU module;

[0009] S2, establish a state parameter model, a scheduling action model and a scheduling reward model for the wafer batch processing workshop scheduling environment; the state parameter model is the input parameter of the agent, including task arrival information, batch state parameter and device processing state parameter; the scheduling action model is the output scheduling action set of the agent, including the batch grouping action set and the batch assignment action set; the scheduling reward model is a step-by-step feedback model equivalent to the scheduling objective function;

[0010] S3, in the initial stage of training, a random strategy is used to collect experience samples to form a real sample pool, and the experience samples in the real sample pool are used to train the generative adversarial network; the generative adversarial network expands the training sample space by generating a virtual sample pool;

[0011] S4, while the GAN generates virtual samples, the relationship correction module is combined to learn the internal relationship between the real samples and the virtual samples, and the quality of the virtual samples is optimized through the relationship correction module; the relationship correction module trains the relationship between the state, action and subsequent state, reward of the real samples;

[0012] S5, optimize the GAN network using relative entropy, and adjust the distribution consistency of the generated virtual samples and the real samples by measuring the relative entropy;

[0013] S6, combine the real sample pool and the virtual sample pool to construct batch training samples, and apply the samples in the independent model of each agent for training to speed up the learning process and optimize the scheduling strategy;

[0014] S7, train multiple modules of wafer batch processing workshop scheduling; through the interaction between the agent and the workshop, the gradient descent method is used to optimize the parameters of the neural network model, and the best scheduling strategy of wafer workshop scheduling is trained;

[0015] S8, the wafer batch processing workshop multi-agent deep reinforcement learning model is deployed into the real wafer workshop production management system through the hybrid driving mechanism of event triggering and time window triggering.

[0016] Further, in step S1, the batch grouping agent and the batch assignment agent output their respective scheduling decisions according to the state information provided by the interaction center module; the generative adversarial network expands the training data set by generating virtual samples to improve the training effect; the relationship correction module optimizes the quality of the GAN generated samples, and improves the representativeness and accuracy of the virtual samples by learning the relationship between the real and virtual samples; the information interaction center GRU module integrates the functions of memory, prediction and agent interaction, records the state changes and scheduling history, and transmits the memory and prediction information to assist decision-making.

[0017] Further, in step S8, the hybrid driving mechanism of event triggering and time window triggering is as follows: when the scheduling is triggered, the batch grouping agent makes batch scheduling decisions according to the actual production situation of the wafer workshop, and the batch assignment agent makes batch assignment scheduling decisions according to the actual production situation of the wafer workshop; before the agent scheduling, the interactive information with memory and prediction is obtained from the information interaction center GRU module, and the real-time state of the workshop; after the agent scheduling, the state of the workshop before scheduling and the corresponding scheduling decision are sent to the information interaction center GRU module; after the scheduling command is issued, the information interaction center GRU module first checks the legality of the scheduling decision, executes if it is legal, updates the state of the workshop, obtains the evaluation value of the scheduling feedback model, records the state, action and reward, and adjusts the model parameters regularly.

[0018] Further, step S1 is as follows:

[0019] The multi-agent collaborative reinforcement learning model of the wafer batch processing area scheduling system is constructed; the multi-agent collaborative deep reinforcement learning model includes two agents, which are responsible for batch grouping decision and batch assignment task respectively; each agent includes a scheduling strategy action output module Actor, namely batch grouping Actor and batch assignment Actor, for generating corresponding scheduling actions; wherein, the batch grouping Actor and the batch assignment Actor realize the mapping from state to action through respective deep neural networks: the batch grouping Actor maps the task state and the batch state into the batch grouping action, and the batch assignment Actor maps the batch state and the device state into the batch assignment action; the dynamic arrival or re-entry of the task triggers the batch grouping Actor to make batch grouping action decision, and the idle of the batch processing device triggers the batch assignment Actor to make batch assignment action decision; the batch grouping Actor and the batch assignment Actor realize the mapping from the batch processing environment state and scheduling decision to scheduling evaluation through a globally shared Critic by a deep neural network, and the two agents share a global GRU network, which constantly fuses the dynamic scheduling information of the wafer batch processing environment, combines the information vector obtained with the observation state when making batch grouping or batch assignment scheduling decision, and guides the agent to make strategy decision as a whole, so as to realize global optimization;

[0020] The mapping of the task, batch state and batch, device state to the batch grouping action and batch assignment action is as follows: when the task dynamically arrives or re-enters, the batch grouping agent first observes the wafer batch processing area task and batch state, and selects the batch grouping action scheduling decision in the executable scheduling decision space according to the state information, and feeds back the batch grouping reward to the batch grouping agent according to the contribution of the batch grouping action scheduling decision to the target; when the device is idle, the batch assignment scheduling is triggered, the batch assignment agent observes the wafer batch processing area batch and device state, and selects the corresponding batch assignment scheduling decision in the batch assignment scheduling action space according to the state information, and feeds back the batch assignment reward to the batch assignment agent according to the contribution of the batch assignment scheduling decision to the target; the batch grouping and batch assignment agents are updated with time sequence, and the batch processing process is promoted, the batch grouping and batch assignment scheduling agents are updated through reward feedback, and the scheduling strategy optimization is realized.

[0021] Further, in this process, the dynamic scheduling information of the wafer batch processing environment is fused, and the specific operation process is as follows:

[0022] The GRU module, i.e., the information interaction center GRU module, obtains wafer batch scheduling state information and scheduling strategy decisions as inputs, imports them into the network for encoding and memorizing, realizes fusion of wafer batch dynamic scheduling information, removes redundant information in wafer batch scheduling through reset and update gates in the network, combines and updates key information with original information, and outputs a message vector to provide key wafer batch environment dynamic scheduling information for the group batch and batch assignment agent; after the task state, batch state and device state vector and the scheduling decision are combined, they are input into the GRU network, information transmission is performed through hidden states h and c, and the input and output of the GRU unit can be represented as shown in formula (1):

[0023] m dc-1 = GRU (h dc-2 , c dc-2 , [s dc-1 , a dc-1 ]; ψ) (1)

[0024] Taking the dc-th scheduling decision as an example, the message vector m dc-1 needs to be obtained as a reference and guidance, the information vector m dc after the dc-th scheduling is an interaction information vector, which is a record and encoding of the wafer batch environment state and the strategy decision set [s dc-1 , a dc-1 ]; specifically, s dc-1 is the wafer batch environment state when the (dc-1)-th scheduling is performed, a dc-1 is the group batch or batch assignment action decision when the (dc-1)-th scheduling is performed, ψ is a parameter of the GRU network, h dc-2 and c dc-2 are the GRU hidden states when the (dc-2)-th scheduling is performed; when each interaction information is input into the GRU network, the key information in the wafer batch environment is extracted through reset and update of the hidden state of the recurrent network, and redundant information is removed and forgotten.

[0025] The global GRU network is used to optimize the wafer batch scheduling process, i.e., global optimization: the group batch agent and the batch assignment agent continuously input dynamic scheduling information into the global GRU network, and obtain the message vector m dc-1 processed by the GRU before each scheduling decision, as a guide for the scheduling decision; specifically, the GRU network records and processes the environment state and decision information when the (dc-1)-th scheduling is performed, and the generated message vector m dc-1 is transmitted to the agent that needs to perform the (dc)-th scheduling, so as to realize collaborative scheduling between the group batch agent and the batch assignment agent through message interaction.

[0026] The state observed by the batch grouping agent includes a lot state f1 of a wafer to be grouped and a batch state f2, and the state observed by the batch assignment agent includes the batch state f2 and a state f3 of a wafer batch processing device;

[0027] The state vector of the local observation of the batch grouping agent is designed as follows:

[0028] s B = [f1, f2] (2)

[0029] The state vector of the local observation of the batch assignment agent is designed as follows:

[0030] s S = [f2, f3] (3)

[0031] At the dc-th scheduling, the agent Ai considers the local state s dc and dynamic information m dc-1 to make scheduling, and the scheduling strategy of the agent Ai can be represented as π i (s dc , m dc-1 ; θ i , ψ), and the batch grouping scheduling strategy function π0 and the batch assignment scheduling strategy function π1 are approximated by two deep neural networks with parameters θ i ; the global state of the wafer batch processing environment is s dc , the scheduling decision made by the agent Ai is a dc , and the Critic can be represented as follows after being approximated by a deep neural network with parameters :

[0032]

[0033] Further, in step S2, the state parameter model, the scheduling action parameter model and the scheduling feedback model of the wafer batch processing area scheduling environment are constructed as follows:

[0034] The state parameter model: the wafer batch processing scheduling includes three objects of tasks, batches and devices, and the environment state is described by a vector F = [f1, f2, f3], wherein f1 = [f 1,1 ,..., f 1,n ] is the state of a task to be grouped, and the element f 1,j = [f 1,j,1 ,..., f 1,j,5 ] is a feature vector of the task j j ; the subscripts 1-5 represent the task weight, the task arrival time, the number of layers to be processed, the expected flow time and the wafer type of the task j respectively; f2 = [f 2,1 ,..., f 2,b] represents the feature vector of batch B 2,k 2,k,1 ,...,f 2,k,2 ] represents the feature vector of batch B k , and subscripts 1 and 2 represent batch capacity proportion and batch type, respectively; f3 = [f 3,1 ,...,f 3,m ] is the device state, wherein the element f 3,m =[f 3,m,1 ,...,f 3,m,3 ] is the feature vector of device M i , and subscripts 1-3 represent device remaining processing time, device current processing type, and device current processing proportion, respectively;

[0035] The scheduling action model is a set of scheduling decisions executable by the wafer scheduling workshop in different states; the group batch scheduling decision space is based on the characteristics that the wafer manufacturing batch processing device is incompatible with the process type, a number of q group batch buffers are set for each process type, and the group batch scheduling decision adds the current to-be-grouped task to a group batch buffer or temporarily groups the batch, and the group batch scheduling decision space is defined as:

[0036] Scheduling decision 1: select the kth group batch buffer, k represents the number of the buffer:

[0037] a = k (0 ≤ k ≤ q)

[0038] Scheduling decision 2: wait:

[0039] a = q (this case indicates no grouping, and waiting is taken)

[0040] When making a group batch scheduling decision, if a group batch buffer is selected, i.e., scheduling decision 1, the current to-be-grouped task is added to the group batch buffer; if waiting is selected, i.e., scheduling decision 2, the task is temporarily grouped; if the batch to which the task is added by the group batch agent exceeds the maximum capacity of the batch, the grouping fails, and the result is equivalent to scheduling decision 2;

[0041] The batch assignment scheduling decision selects all group batch buffers of process types in the group batch buffer and matches to idle devices for processing; the batch assignment decision space is a multiple of the process type of the group batch decision scheduling space; selecting a group batch buffer, the batch is matched to a device for processing, and the group batch buffer is emptied; selecting waiting indicates that no batch is selected for processing;

[0042] The scheduling feedback model is a scheduling reward function, after the group batch agent makes a decision, the reward function is related to the current flow time, the subsequent expected flow time of the group batch task, and the capacity proportion of the decision batch slot, so the group batch reward r B ​A weighted sum of the current flow time for batch success, the subsequent expected flow time, and the capacity proportion of the decision batch tank:

[0043] r B = - [W1(t-a ti ) + W2F i + W3V j ] · sw B (t) - W4(1-sw B (t)) (5)

[0044]

[0045] where W1, W2, W3, W4 are weight factors, t is the current time, a ti is the initial arrival time of the task to be batched, F i is the subsequent expected flow time, V j is the capacity proportion of the decision batch tank, sw B (t) indicates whether the task is successfully batched at time t.

[0046] Based on the objective of minimizing the wafer production cycle, the reward function of the batch assignment agent is as follows:

[0047]

[0048] where t is the current time, is the earliest start processing time of the batch processing equipment, is the processing time of the batch processing equipment group that has been performed;

[0049] Since the batch and batch assignment agents jointly optimize the scheduling objective of minimizing the wafer production cycle, the reward values of batch and batch assignment are globally discounted to realize the mutual influence and correlation between batch and batch assignment. Taking the dc scheduling as an example, the globally discounted reward is expressed as:

[0050]

[0051] where γ is the discount factor, r t is the reward value at time t, dc' indicates the total number of scheduling, through task-driven, continuously scheduling, obtaining a large amount of workshop scheduling interaction data <s, a, r, s'> and storing, the batch agent and the device assignment agent continuously update the parameters through the feedback reward function, realize the reward maximization, that is, the wafer manufacturing production cycle scheduling objective minimization.

[0052] Further, the training in steps S3 and S7 is specifically:

[0053] The multi-agent model of wafer batch processing workshop scheduling is trained; after the network parameters of the batch grouping and batch assignment agent are initialized, the corresponding state observation input is input to make a decision, and the network parameters are updated through the feedback reward function for continuous optimization; the model parameter updating method is improved on the PPO algorithm, the GRU network is globally updated with the Actor and Critic to realize the synchronous optimization of the GRU network and the agent; the output of the GRU network is part of the input of the Actor network and the Critic network, and the gradient backpropagation of the Actor network and the Critic network to the prefix GRU network realizes global parameter optimization.

[0054] Further, steps S4, S5 and S6 are specifically:

[0055] The generative adversarial network and the relationship correction module are designed and trained.

[0056] The real experience sample set in the deep reinforcement learning process is a pair of (state s, action a) and (subsequent state s', reward r); the state s at the previous moment corresponds to the corresponding action a, (s, a) is called a state-action pair, the next moment moves to the state s', and the immediate reward r is obtained, (s', r) is called a subsequent state-reward pair; therefore, the real experience sample set Dx=[s, a, s', r] can be divided into two parts:

[0057] Dx=[(s, a), (s′, r)]=[(x1,x2)] (9)

[0058] Wherein, x1 represents a state-action pair, and x2 represents a subsequent state-reward pair; since the subsequent state s' and the reward r are based on the previous state s and the corresponding action a, there is a certain relationship between x1 and x2, and this relationship is represented by mutual information M:

[0059]

[0060] Wherein, X1 represents a set of state-action pairs, X2 represents a set of subsequent state-reward pairs, p represents a probability, E(x2) represents an entropy of x2, and is used to measure the uncertainty of x2; E(x2|x1) represents the uncertainty of x2 given x1; M(x2;x1) represents the amount of reduction in the uncertainty of x2 caused by x1; since x1 and x2 are related to each other, the mutual information M cannot be 0; therefore, a deep neural network, i.e. a relationship correction network, is constructed, which inputs x1 and outputs x2, and is used to train the internal relationship between x1 and x2 in the experience sample set; consistent with the experience sample set, the experience sample G(z)=[s z ,az,s′,r z ] generated by the GAN can also be divided into two corresponding parts:

[0061] G(z) = [(s z ,a z ),(s′,r z )] = [G1(z),G2(z)] (11)

[0062] Wherein, G1(z) represents the generated state-action pair, G2(z) represents the generated subsequent state-reward pair; in order to improve the quality of the generated sample, the structural relationship between the generated G2(z) and G1(z) should be consistent with the structural relationship in the real sample [x1,x2] on the basis of the generated sample G(z); therefore, the generated sample G1(z) is combined with the mutual information M, G1(z) is input into the relationship correction network, and the output thereof is taken as the corrected subsequent state-reward pair G2(z)'; the relative entropy, i.e. KL divergence, is used to represent the similarity between G2(z) and G2(z)', and the formula is as follows:

[0063]

[0064] Wherein, P represents the generated subsequent state-reward pair G2(z), Q represents the corrected subsequent state-reward pair G2(z)', q(i) and p(i) represent the probability of distribution Q and P on event i respectively;

[0065] When the similarity between the generated subsequent state-reward pair G2(z) and the corrected subsequent state-reward pair G2(z)' is very high, the relative entropy of the two tends to 0, the quality of the sample generated by GAN is high, and the value function W(D,G) can be represented as:

[0066]

[0067] Wherein, D(x) is the probability estimation of the discriminator that the real sample x is "real", G(z) represents that the generator maps the noise z~Pz into the pseudo sample, V represents the adversarial objective function, D represents the discriminator, G represents the generator, and K is a weight parameter; at the same time, the GAN model is optimized by the L2 regularization method.

[0068] Further, the generative adversarial network is a Corr-GAN, and the network structure of the Corr-GAN is specifically: a GAN for generating experience samples; a real sample pool D1 and a virtual sample pool D2, the D1 is used for collecting real experience samples obtained in the interaction process of the agent and the scheduling environment, the D2 is used for collecting samples generated by the GAN, and the two together provide training samples for the agent; a relationship correction module representing a constructed deep neural network, used for training the relationship between the state-action pair [s, a] in the D1 and the subsequent state-reward pair [s', r] to guide the generator model G to improve performance; in addition, the relationship correction module also analyzes the internal relationship between [s, a] and [s', r] in the D1, inputs the state-action pair [s, a] generated by the generator model G into the relationship correction module, and outputs the subsequent state-reward pair [s'', r'] conforming to the internal relationship, and then compares the subsequent state-reward pair [s', r] generated by the generator model G, to further prompt the generator model G to generate more actual samples;

[0069] Before training the GAN and the relationship correction network, experience data needs to be collected from the actual scheduling environment; in the training stage, the GAN generates virtual data; the relationship correction network learns the relationship between the real data and the virtual data; in the data enhancement stage, the system respectively calls the pre-trained generative adversarial network GAN and the relationship correction module to cooperatively generate higher-quality virtual samples; finally, in the agent pre-training stage, the multi-agent deep reinforcement learning model is trained by using the virtual data and the historical real data.

[0070] Further, the deployment process in step S8 is specifically:

[0071] The wafers lots with dynamically arriving tasks are put into the corresponding type of batch slots through the batch grouping agent, the number of batches for grouping is fixed, and is valued according to the actual scheduling process; the process type of the wafers lots in the batch slot is determined by the process type of the first wafer lot entering the slot, and when the equipment is idle, the corresponding batch slot is dispatched by the batch assignment agent to enter the batch processing equipment for processing, the wafers are processed layer by layer, and after processing a layer, the wafers are re-entered for processing the next layer of the wafers, until all the wafers lots are processed to the last layer.

[0072] The beneficial effects of the present application are:

[0073] The application provides a semiconductor batch processing area scheduling method based on adversarial learning enhanced multi-agent cooperation, which can effectively deal with the shortcomings of traditional scheduling methods in dynamic and complex production environments. Traditional mathematical programming and heuristic algorithms often lack flexibility and adaptability when dealing with multi-variety and large-batch production in wafer manufacturing, resulting in inefficient production scheduling. By introducing generative adversarial networks (GAN) and multi-agent deep reinforcement learning, the system can generate diverse training samples and optimize scheduling decisions. In addition, the application combines a hybrid mechanism of event triggering and time window triggering. In actual production, the system works cooperatively with group batch agents and batch assignment agents, makes decisions in real time according to the production situation of the workshop, and shares state and prediction information through an information interaction center to optimize scheduling strategies and improve the flexibility and response speed of the system, thereby effectively reducing production delays, reducing resource waste and production costs. BRIEF DESCRIPTION OF DRAWINGS

[0074] Figure 1 A flowchart of the semiconductor batch processing area scheduling method based on adversarial learning enhanced multi-agent cooperation of the application is shown.

[0075] Figure 2 An interaction diagram of the reinforcement learning agent and the workshop of the application is shown.

[0076] Figure 3 A schematic diagram of the overall structure of the reinforcement learning scheduling model of the application is shown.

[0077] Figure 4 A schematic diagram of the operation logic of the GRU module of the application is shown.

[0078] Figure 5 A schematic diagram of the structure of the GAN network and the relationship correction module of the application is shown.

[0079] Figure 6 A schematic diagram of the training process of the GAN network and the relationship correction module of the application is shown. DETAILED DESCRIPTION

[0080] The application will be described in detail below in conjunction with the drawings and specific embodiments. The present embodiment is based on the technical solution of the application and gives a detailed implementation and specific operation process, but the protection scope of the application is not limited to the following examples.

[0081] The present application aims to minimize the wafer production cycle. The method combines adversarial learning and multi-agent collaboration mechanism to construct group batch agents and batch assignment agents, realizing the joint optimization of batch combination and task allocation. The gated recurrent unit (GRU) network is introduced to extract the dynamic characteristics in the workshop operation process, enhancing the perception of scheduling strategy to time sequence information and the environmental adaptability. Through adversarial training, real and virtual experience samples are generated for pre-training, expanding the training data and improving the strategy convergence speed. Further, the relationship correction network is introduced to model the internal association structure between experiences, combining the relative entropy optimization mechanism to improve the experience quality and strategy updating efficiency. The present application can quickly and accurately generate scheduling decisions, effectively shorten the total production cycle in the wafer manufacturing process, and has high engineering application value.

[0082] The present application proposes a multi-agent collaborative semiconductor batch processing area scheduling method enhanced by adversarial learning, aiming to optimize the production scheduling cycle of wafer batch processing workshop. The method framework includes multiple scheduling modules: group batch agent, batch assignment agent, and generative adversarial network (GAN), relationship correction module. The generative adversarial network module expands the training dataset by generating virtual samples, improving the training effect; the relationship correction module optimizes the quality of GAN generated samples, enhancing the representativeness of samples. At the same time, the GRU (gated recurrent unit) model is used to process the state sequence of the workshop, capturing the timing characteristics in the production process. The method works collaboratively through multi-agent, combined with deep reinforcement learning, effectively improving the flexibility and efficiency of production scheduling, and can adapt to the state changes of production line and dynamic demand of orders in real time.

[0083] The operation steps are referred to Figure 1 The adversarial learning enhanced multi-agent collaborative semiconductor batch processing area scheduling method of the present application comprises the following steps:

[0084] An adversarial learning enhanced multi-agent collaborative semiconductor batch processing area scheduling method, characterized in that it comprises the following steps:

[0085] S1, acquire the processing task state data and machine state data of the current arrived batch processing area; establish a deep reinforcement learning multiple scheduling module for wafer batch processing workshop scheduling, including group batch agent, batch assignment agent, generative adversarial network (GAN), relationship correction module, and information interaction center GRU module;

[0086] The group batch agent and the batch assignment agent output scheduling decisions respectively according to state information provided by the information interaction center module; the generative adversarial network expands the training data set by generating virtual samples, and improves the training effect; the relationship correction module optimizes the quality of the GAN generated samples, and improves the representativeness and accuracy of the virtual samples by learning the relationship between the real and virtual samples; the information interaction center GRU module integrates the functions of memory, prediction and agent interaction, records state changes and scheduling history, and transmits memory and prediction information to assist decision-making.

[0087] S2, a state parameter model, a scheduling action model and a scheduling reward model of a wafer batch processing workshop scheduling environment are established; the state parameter model is an input parameter of the agent, including task arrival information, batch state parameters and device processing state parameters; the scheduling action model is an output scheduling action set of the agent, including a group batch action set and a batch assignment action set; the scheduling reward model is a step-by-step feedback model equivalent to a scheduling objective function, used for mutual rewards between the group batch agent and the batch assignment agent, to achieve a global optimization scheduling as much as possible, and to minimize the processing cycle of the wafer lot;

[0088] S3, in the initial stage of training, a random strategy is used to collect experience samples to form a real sample pool, and the experience samples in the real sample pool are used to train the generative adversarial network; the generative adversarial network expands the training sample space by generating a virtual sample pool;

[0089] S4, while the GAN generates virtual samples, the relationship correction module is combined to learn the internal relationship between the real samples and the virtual samples, and the quality of the virtual samples is optimized through the relationship correction module. The relationship correction module further improves the quality of the virtual samples generated by the GAN by training the relationship between the state, action and subsequent state, reward of the real samples, and enhances the representativeness and effectiveness of the samples;

[0090] S5, the GAN network is optimized by using relative entropy, the distribution consistency of the generated virtual samples and the real samples is adjusted by measuring the relative entropy, the effectiveness of the virtual samples is enhanced, and the quality of the sample pool is improved;

[0091] S6, combining the real sample pool and the virtual sample pool, batch training samples are constructed, and the samples are applied to the independent model of each agent for training to speed up the learning process and optimize the scheduling strategy;

[0092] S7, multiple modules of wafer batch processing workshop scheduling are trained. Through the interaction between the agent and the workshop, the neural network model is optimized by using the gradient descent method, and the best scheduling strategy of wafer workshop scheduling is trained;

[0093] S8, deploy the wafer batch processing workshop multi-agent deep reinforcement learning model to the real wafer workshop production management system, through a hybrid driving mechanism of event triggering and time window triggering, when the scheduling is triggered, the batch grouping agent makes batch grouping scheduling decision according to the actual production situation of the wafer workshop, and the batch assignment agent makes batch assignment scheduling decision according to the actual production situation of the wafer workshop; before the scheduling of the agent, the interactive information with memory and prediction and the real-time state of the workshop are obtained from the information interaction center GRU module; after the scheduling of the agent, the state of the workshop before the scheduling and the corresponding scheduling decision are sent to the information interaction center GRU module; after the scheduling command is issued, the information interaction center GRU module first checks the legality of the scheduling decision, if it is legal, it is executed, and the state of the workshop is updated, the evaluation value of the scheduling feedback model is obtained, the state, action and reward are recorded, and the model parameters are adjusted regularly.

[0094] The specific implementation is as follows:

[0095] Step 1 (corresponding to S1). Obtain the processing task state data of the current arrival batch processing area and the machine state data.

[0096] Step 2 (corresponding to S1). Construct a multi-agent collaborative reinforcement learning model of the wafer batch processing area scheduling system, as shown in Figure 3 .

[0097] The multi-agent collaborative deep reinforcement learning model includes two agents, which are responsible for batch grouping decision and batch assignment task respectively. Each agent includes a scheduling strategy action output module (Actor), namely batch grouping Actor and batch assignment Actor, for generating corresponding scheduling actions. Among them, the batch grouping Actor and the batch assignment Actor realize the mapping from state to action through their respective deep neural networks: the batch grouping Actor maps the task state and batch state to the batch grouping action, and the batch assignment Actor maps the batch state and device state to the batch assignment action, as shown in Figure 2 . Task dynamic arrival or re-entry triggers the batch grouping Actor to make batch grouping action decision, and batch processing device idle triggers the batch assignment Actor to make batch assignment action decision. The batch grouping Actor and the batch assignment Actor realize the mapping from the batch processing environment state and scheduling decision to the scheduling evaluation through a deep neural network by a globally shared Critic, and the two agents share a global GRU network, which constantly fuses the dynamic scheduling information of the wafer batch processing environment, combines the information vector obtained with the observation state when making batch grouping or batch assignment scheduling decision, and guides the agent to make strategy decision as a whole, realizing global optimization.

[0098] The mapping of the task, batch state and batch, device state to group batch action and batch assignment action is as follows: when the task dynamically arrives or re-enters, the group batch agent first observes the wafer batch processing area task and batch state, and selects a group batch action scheduling decision in the executable scheduling decision space according to the state information, and feeds back a group batch reward to the group batch agent according to the contribution of the target to the group batch action scheduling decision; when the device is idle, the batch assignment scheduling is triggered, the batch assignment agent observes the wafer batch processing area batch and device state, and selects the corresponding batch assignment scheduling decision in the batch assignment scheduling action space according to the state information, and feeds back the batch assignment reward to the batch assignment agent according to the contribution of the target to the batch assignment scheduling decision; the group batch and batch assignment agents are updated with time and advance the batch processing process, and the group batch and batch assignment scheduling agents are constantly updated through reward feedback to realize scheduling strategy optimization.

[0099] In this process, the dynamic scheduling information of the fusion wafer batch processing environment is involved, and the specific operation process is as follows:

[0100] The GRU module (i.e. the information interaction center GRU module) obtains the wafer batch processing scheduling state information and scheduling strategy decision as input, encodes and remembers in the network, realizes the fusion of the wafer batch processing dynamic scheduling information, removes the redundant information in the wafer batch processing scheduling through the reset gate and update gate in the network, combines and updates the key information with the original information, and outputs the message vector to provide the key wafer batch processing environment dynamic scheduling information for the group batch and batch assignment agents. As shown in Figure 3 After the task state, batch state and device state vector (input vector) and scheduling decision are combined, they are input to the GRU network, information transmission is carried out through the hidden state h and c, and the input and output of the GRU unit can be represented as shown in formula (1):

[0101] m dc-1 = GRU (h dc-2 , c dc-2 , [s dc-1 , a dc-1 ] ; ψ) (1)

[0102] Taking the dc-th scheduling decision as an example, the message vector m dc-1 is needed as a reference and guide, and the information vector m dc after the dc-th scheduling is the interaction information vector, which is a record and encoding of the wafer batch processing environment state and strategy decision set [s dc-1 , a dc-1 ]. Specifically, s dc-1 is the wafer batch processing environment state when the dc-1th scheduling is performed, and a dc-1is the group batch or batch assignment action decision at the first dc-1 scheduling, ψ is the parameter of the GRU network, h dc-2 and c dc-2 is the GRU hidden state at the second dc-2 scheduling. At each interaction information input of the GRU network, the key information in the wafer batch processing environment is extracted, and the redundant information is eliminated and forgotten through the reset and update of the hidden state of the recurrent network. The working mechanism of the GRU is shown in Figure 4 .

[0103] The process involves a group batch and batch assignment agent collaborative interaction mechanism, and the process of completing a message interaction is shown in Figure 3 .

[0104] Based on the global GRU (Gated Recurrent Unit) network to optimize the wafer batch processing scheduling process (i.e. to achieve global optimization): the group batch agent and the batch assignment agent continuously input the dynamic scheduling information to the global GRU network, and obtain the message vector m dc-1 processed by the GRU before each scheduling decision, as a guide for scheduling decision. Specifically, the GRU network records and processes the environmental state and decision information at the first dc-1 scheduling, and the generated message vector m dc-1 is passed to the agent that needs to be scheduled at the dc time, so as to realize the collaborative scheduling between the group batch agent and the batch assignment agent through message interaction.

[0105] The state observed by the group batch agent includes the state f1 of the wafer lot to be grouped and the batch state f2, and the state observed by the batch assignment agent includes the batch state f2 and the state f3 of the wafer batch processing device.

[0106] The state vector observed by the group batch agent is designed as follows:

[0107] s B = [f1, f2] (2)

[0108] The state vector observed by the batch assignment agent is designed as follows:

[0109] s S = [f2, f3] (3)

[0110] At the dc scheduling, the agent Ai considers the local state s dc and the dynamic information m dc-1 to make scheduling, and the scheduling strategy of the agent Ai can be represented as π i (s dc ,m dc-1 ; θ i , ψ), and is realized through the parameter θ iTwo deep neural networks approximate the batch scheduling policy function π0 and the batch assignment scheduling policy function π1, respectively. The global state of the wafer batch processing environment is s. dc The intelligent agent Ai executes the scheduling decision as a dc Critic uses parameters The deep neural network can be approximated as:

[0111]

[0112] Step 3 (corresponding to S2). Construct the state parameter model, scheduling action parameter model, and scheduling feedback model of the wafer batch processing area scheduling environment.

[0113] State parameter model: Wafer batch processing scheduling mainly includes three objects: task, batch, and device. The environment state is described by the vector F = [f1, f2, f3], where f1 = [f 1,1 ,...,f 1,n [] represents the state of tasks to be batched, where element f is... 1,j =[f 1,j,1 ,...,f 1,j,5 [For task J] j The feature vector, with subscripts 1-5 representing the task weight, task arrival time, number of layers to be processed, expected flow time, and wafer type of task j, respectively; f2 = [f 2,1 ,...,f 2,b ] represents the batch state, where element f is... 2,k =[f 2,k,1 ,...,f 2,k,2 ] indicates batch B k The feature vector, with subscripts 1 and 2 representing the batch size percentage and batch type, respectively; f3 = [f 3,1 ,...,f 3,m [] represents the device state, where element f is... 3,m =[f 3,m,1 ,...,f 3,m,3 [For device M] i The feature vector, with subscripts 1-3 representing the remaining processing time of the equipment, the current processing type of the equipment, and the current processing percentage of the equipment, respectively;

[0114] The scheduling action model is the set of scheduling decisions that can be executed in different states of the wafer scheduling workshop. The batch scheduling decision space is based on the characteristic of incompatible process types in wafer manufacturing batch processing equipment. Each process type has a set of q batch buffers. The batch scheduling decision adds the currently pending batching task to a batch buffer or postpones batching. The batch scheduling decision space is defined as follows:

[0115] Scheduling decision 1: select the kth group buffer (k represents the number of the buffer):

[0116] a = k (0≤k≤q)

[0117] Scheduling decision 2: wait:

[0118] a = q (this case represents no grouping, and waiting is taken)

[0119] When making a grouping scheduling decision, if a group buffer is selected (i.e. scheduling decision 1), the current task to be grouped is added to the group buffer; if waiting is selected (i.e. scheduling decision 2), the task is temporarily suspended for grouping. If the batch number added by the grouping agent exceeds the maximum batch capacity, the grouping fails, and the result is equivalent to scheduling decision 2.

[0120] Batch assignment scheduling decision selects all group buffers of all process types in the group buffer, and matches to idle equipment for processing. The batch assignment decision space is a multiple of the process type of the grouping decision scheduling space (the multiple is determined according to the number of process types). Selecting a group buffer, the batch is matched to the equipment and processed, and the group buffer is emptied; selecting waiting, it means that no batch is selected for processing.

[0121] The scheduling feedback model is a scheduling reward function. After the grouping agent makes a decision, the reward function is related to the current flow time, the subsequent expected flow time and the capacity proportion of the decision batch slot of the grouping task, so the grouping reward r B is the weighted sum of the current flow time, the subsequent expected flow time and the capacity proportion of the decision batch slot for successful grouping:

[0122] r B = -[W1(t-a ti )+W2F i +W3V j ]·sw B (t)-W4(1-sw B (t)) (5)

[0123]

[0124] where W1, W2, W3, W4 are weight factors, t is the current time, a ti is the initial arrival time of the task to be grouped, F i is the subsequent expected flow time, V j is the capacity proportion of the decision batch slot, sw B (t) represents whether the task is successfully grouped at time t.

[0125] Based on the objective of minimizing the wafer production cycle, the reward function of the batch assignment agent is as follows:

[0126]

[0127] where t is the current time, is the earliest time that the batch processing equipment starts processing, is the processing time that the batch processing equipment group has already performed.

[0128] Since the group batch and batch assignment agents jointly optimize the minimum wafer production cycle scheduling target, the rewards of the group batch and batch assignment are globally discounted to realize the mutual influence and correlation between the group batch and batch assignment. Taking the dc times scheduling as an example, the global discounted reward is expressed as:

[0129]

[0130] where γ is the discount factor, r t is the reward value at time t, and dc' represents the total number of schedules. Through task driving, scheduling is continuously performed to obtain a large amount of workshop scheduling interaction data <s, a, r, s'> and store them. The group batch agent and the equipment assignment agent continuously update the parameters through the feedback reward function to realize the maximization of the reward, that is, the minimization of the wafer manufacturing production cycle scheduling target.

[0131] Step 4 (corresponding to S3 and S7). The multi-agent model of the wafer batch processing workshop scheduling is trained. After the network parameters of the group batch and batch assignment agents are initialized, the corresponding state observation inputs are input to make decisions, and the network parameters are updated through the feedback reward function for continuous optimization. The model parameter updating method is improved on the basis of the traditional PPO algorithm, the GRU network is globally updated with the Actor and Critic to realize the synchronous optimization of the GRU network and the agent. The output of the GRU network is part of the input of the Actor network and the Critic network, and the gradient backpropagation to the prefix GRU network is realized when the Actor network and the Critic network are updated to realize global parameter optimization.

[0132] Step 5 (corresponding to S4, S5, S6). Generation of the adversarial network and design and training of the relationship correction module.

[0133] The real experience sample set in the deep reinforcement learning process is a pair of (state s, action a), (subsequent state s', reward r). The state s of the previous moment corresponds to the corresponding action a, (s, a) is called a state-action pair, the next moment moves to the state s', and the immediate reward r is obtained, (s', r) is called a subsequent state-reward pair. Therefore, the real experience sample set Dx = [s, a, s', r] can be divided into two parts:

[0134] Dx = [(s, a), (s', r)] = [(x1, x2)] (9)

[0135] where x1 represents the state-action pair, and x2 represents the subsequent state-reward pair. Since the subsequent state s' and the reward r are based on the previous state s and the corresponding action a, there is a certain relationship between x1 and x2, which is represented by mutual information M:

[0136]

[0137] where X1 represents the set of state-action pairs, X2 represents the set of subsequent state-reward pairs, p represents the probability, E(x2) represents the entropy of x2, which measures the uncertainty of x2; E(x2|x1) represents the uncertainty of x2 given x1; M(x2;x1) represents the amount of reduction in the uncertainty of x2 caused by x1. Since x1 and x2 are related to each other, the mutual information M cannot be 0. Therefore, a deep neural network (i.e., a relationship correction network) is constructed, which takes x1 as input and outputs x2, to train the intrinsic relationship between x1 and x2 in the experience sample set. Consistent with the experience sample set, the experience sample G(z) = [s z ,az,s′,r z ] generated by GAN can also be divided into two corresponding parts:

[0138] G(z) = [(s z ,a z ),(s′,r z )] = [G1(z), G2(z)] (11)

[0139] where G1(z) represents the generated state-action pair, and G2(z) represents the generated subsequent state-reward pair. In order to improve the quality of the generated samples, the structural relationship between G2(z) and G1(z) based on the generated sample G(z) should be consistent with the structural relationship in the real sample [x1,x2]. Therefore, we combine the generated sample G1(z) with the mutual information M, input G1(z) into the relationship correction network, and take its output as the corrected subsequent state-reward pair G2(z)', the purpose is to make the generated subsequent state-reward pair G2(z) have high similarity with the corrected subsequent state-reward pair G2(z)'. Relative entropy (KL divergence) is used to represent the similarity between G2(z) and G2(z)', the formula is as follows:

[0140]

[0141] Wherein, P represents the generated subsequent state reward pair G2(z), Q represents the modified subsequent state reward pair G2(z)', q(i), p(i) respectively represent the probability of distribution Q, P on event i.

[0142] When the generated subsequent state reward pair G2(z) is highly similar to the modified subsequent state reward pair G2(z)', the relative entropy of the two tends to 0, the sample quality generated by the GAN is high, and the value function W(D, G) can be represented as:

[0143]

[0144] Wherein, D(x) is the probability estimation of the discriminator that the real sample x is "real", G(z) represents that the generator maps the noise z ~ Pz to the pseudo sample, V represents the adversarial target function, D represents the discriminator, G represents the generator, and K is the weight parameter. At the same time, the GAN model can be optimized by L2 regularization method, and the sample generation quality is further improved. When the GAN performance is continuously optimized, G1(z) and G2(z) will be continuously updated according to the real experience sample set, and D KL will tend to 0. At the same time, W(D, G) will reach the global minimum value.

[0145] When W(D, G) reaches the global minimum value, the generated experience sample of the GAN has high similarity with the real experience sample, and the generated state-action pair G1(z) and the subsequently generated state-reward pair G2(z) have high correlation inside the generated experience sample G(z).

[0146] The network structure of Corr-GAN is as shown in Figure 5 The network structure of Corr-GAN is as shown in

[0147] The training process of the algorithm is as shown in Figure 6The shown. Before training the GAN and the relationship correction network, experience data needs to be collected from the actual scheduling environment. In the training phase, the GAN generates virtual data; the relationship correction network learns the relationship between the real data and the virtual data; in the data enhancement phase, the system respectively calls the pre-trained generative adversarial network (GAN) and the relationship correction module to cooperatively generate higher-quality virtual samples, thereby enhancing data diversity and reality; finally, in the agent pre-training phase, the multi-agent deep reinforcement learning model is trained using virtual data and historical real data.

[0148] Step 6 (corresponding to S8). Deploy the multi-agent deep reinforcement learning model to the wafer batch processing workshop, and test and verify the proposed algorithm and other better reinforcement learning scheduling algorithms through standard examples and enterprise instances.

[0149] The task dynamically arrived wafer lot is put into the corresponding type of batch slot through the batch grouping agent, the batch quantity is fixed, and the actual scheduling process is valued. The process type of the wafer lot in the batch slot is determined by the process type of the first wafer lot entering the slot, and when the device is idle, the corresponding batch slot is scheduled through the batch assignment agent and enters the batch processing device for processing. The wafers are processed layer by layer, and after processing a layer, the wafers are re-entered for processing the next layer of the wafers until all the wafers are processed to the last layer.

[0150] The above-described embodiments only express one embodiment of the present application, which is described in detail and specifically, but it should not be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A multi-agent collaborative semiconductor batch processing area scheduling method, characterized by, Comprise the following steps: S1, obtain the current arrival of the batch processing area processing task state data, machine state data; Establish wafer batch processing workshop scheduling deep reinforcement learning multiple scheduling modules, including batch intelligent agent, batch assignment intelligent agent, generative adversarial network GAN, relationship correction module, and information interaction center GRU module; S2, establish the state parameter model of wafer batch processing workshop scheduling environment, scheduling action model and scheduling reward model; The state parameter model is the input parameter of the agent, including task arrival information, batch state parameter and device processing state parameter; The scheduling action model is the output scheduling action set of the agent, including the batch action set and the batch assignment action set; The scheduling reward model is equivalent to the scheduling target function Feedback model in steps; S3, in the early stage of training, a random strategy is used to collect experience samples, form a real sample pool, and use the experience samples in the real sample pool to train the generative adversarial network; The generative adversarial network expands the training sample space by generating a virtual sample pool; S4, while the GAN generates virtual samples, the relationship correction module is combined to learn the internal relationship between the real samples and the virtual samples, and the quality of the virtual samples is optimized through the relationship correction module; The relationship correction module trains the relationship between the state, action and subsequent state, reward of the real sample; S5, use relative entropy to optimize the GAN network, and adjust the distribution consistency of the generated virtual samples and real samples through relative entropy; S6, combine the real sample pool and the virtual sample pool to construct batch training samples, and apply the samples in the independent model of each agent for training to speed up the learning process and optimize the scheduling strategy; S7, train multiple modules of wafer batch processing workshop scheduling; Through the interaction between the agent and the workshop, the neural network model is optimized by gradient descent method, and the best scheduling strategy of wafer workshop scheduling is trained; S8, the hybrid driving mechanism of event triggering and time window triggering is used to deploy the wafer batch processing workshop multi-agent deep reinforcement learning model to the real wafer workshop production management system.

2. The multi-agent coordinated semiconductor batch processing zone scheduling method of claim 1, wherein, In step S1, the batch intelligent agent and the batch assignment intelligent agent output their respective scheduling decisions according to the state information provided by the interaction center module; The generative adversarial network expands the training data set by generating virtual samples to improve the training effect; The relationship correction module optimizes the quality of the GAN generated samples, learns the relationship between real and virtual samples, and improves the representativeness and accuracy of virtual samples; The information interaction center GRU module integrates memory, prediction and agent interaction functions, records state changes and scheduling history, and transmits memory and prediction information to assist decision-making.

3. The multi-agent coordinated semiconductor batch processing zone scheduling method of claim 1, wherein, In step S8, the hybrid driving mechanism of event triggering and time window triggering is as follows: in the dispatch triggering, the batch grouping agent makes a batch grouping scheduling decision according to the actual production situation of the wafer workshop, and the batch assignment agent makes a batch assignment scheduling decision according to the actual production situation of the wafer workshop; before the scheduling of the agent, the interactive information with memory and prediction and the real-time state of the workshop are obtained from the information interaction center GRU module; after the scheduling of the agent, the state of the workshop before the scheduling and the corresponding scheduling decision are sent to the information interaction center GRU module; after the scheduling command is issued, the information interaction center GRU module first checks the legality of the scheduling decision, executes if it is legal, updates the state of the workshop, obtains the evaluation value of the scheduling feedback model, records the state, action and reward, and adjusts the model parameters regularly.

4. The multi-agent coordinated semiconductor batch processing zone scheduling method of claim 1, wherein, Step S1 is specifically as follows: A multi-agent collaborative reinforcement learning model of the wafer batch processing area scheduling system is constructed; the multi-agent collaborative deep reinforcement learning model includes two agents responsible for batch grouping decision and batch assignment task; each agent includes a scheduling strategy action output module Actor, i.e., batch grouping Actor and batch assignment Actor, for generating corresponding scheduling actions; wherein the batch grouping Actor and the batch assignment Actor realize the mapping of state to action through respective deep neural networks: the batch grouping Actor maps the task state and the batch state to the batch grouping action, and the batch assignment Actor maps the batch state and the device state to the batch assignment action; the task dynamic arrival or re-entry triggers the batch grouping Actor to make a batch grouping action decision, and the batch processing device idle triggers the batch assignment Actor to make a batch assignment action decision; the batch grouping Actor and the batch assignment Actor realize the mapping from the batch processing environment state and the scheduling decision to the scheduling evaluation through a globally shared Critic by a deep neural network, and the two agents share a global GRU network, which constantly fuses the dynamic scheduling information of the wafer batch processing environment, combines the obtained information vector and the observation state when making a batch grouping or batch assignment scheduling decision, and guides the agent to make a strategy decision as a whole to realize global optimization; The mapping of the task, batch state and batch, device state to the group batch action and batch assignment action is as follows: when the task dynamically arrives or re-enters, the group batch agent first observes the wafer batch processing area task and batch state, and selects a group batch action scheduling decision in the executable scheduling decision space according to the state information, and feeds back a group batch reward to the group batch agent according to the contribution of the target to the group batch action scheduling decision; when the device is idle, the batch assignment scheduling is triggered, the batch assignment agent observes the wafer batch processing area batch and device state, and selects a corresponding batch assignment scheduling decision in the batch assignment scheduling action space according to the state information, and feeds back a batch assignment reward to the batch assignment agent according to the contribution of the target to the batch assignment scheduling decision; the group batch and batch assignment agents are updated with time and advance the batch processing process, and the group batch and batch assignment scheduling agents are constantly updated through reward feedback to realize scheduling strategy optimization.

5. The multi-agent coordinated semiconductor batch processing zone scheduling method of claim 4, wherein, In this process, the dynamic scheduling information of the fusion wafer batch processing environment is involved, and the specific operation process is as follows: the GRU module, that is, the information interaction center GRU module, obtains the wafer batch processing scheduling state information and scheduling strategy decision as input, and imports it into the network for encoding and memory, realizes the fusion of the wafer batch processing dynamic scheduling information, removes the redundant information in the wafer batch processing scheduling through the reset gate and update gate in the network, combines and updates the key information with the original information, and the output message vector provides the key wafer batch processing environment dynamic scheduling information for the group batch and batch assignment agents; the task state, batch state and device state vector and the scheduling decision are combined and input into the GRU network, and the information is transmitted through the hidden state h and c, and the input and output of the GRU unit can be represented as formula (1): m dc-1 = GRU(h dc-2 , c dc-2 , [s dc-1 , a dc-1 ] ; ψ) (1) Taking the DC scheduling decision as an example, it is necessary to obtain the message vector m. dc-1 As a reference and guide, the information vector m after the dcth scheduling is... dc The interactive information vector is a set of wafer batch processing environment states and strategy decisions [s]. dc-1 ,a dc-1 The records and encoding of ]; specifically, s dc-1 This refers to the wafer batch processing environment state during the (dc-1)th scheduling operation, a dc-1 It refers to the batch or batch assignment action decision during the (dc-1)th scheduling, where ψ is a parameter of the GRU network, and h dc-2 and c dc-2 It is the hidden state of GRU during the dc-2th scheduling; during each interaction of information input in the GRU network, key information in the wafer batch processing environment is extracted by resetting and updating the hidden state of the recurrent network, and redundant information is eliminated and forgotten. The global GRU network optimizes the wafer batch processing scheduling process, that is, global optimization is realized: the batch grouping agent and the batch assignment agent continuously input dynamic scheduling information to the global GRU network, and obtain the message vector m processed by the GRU before each scheduling decision, as a guide for the scheduling decision dc-1 ; specifically, the GRU network records and processes the environment state and decision information at the dc-1th scheduling, generates a message vector m dc-1 , which is passed to the agent that needs to be scheduled at the dcth time, so as to realize collaborative scheduling between the batch grouping agent and the batch assignment agent through message interaction; The state observed by the group batch agent includes the lot state f1 and the batch state f2, and the state observed by the batch assignment agent includes the batch state f2 and the wafer batch processing device state f3; The state vector observed by the group batch agent is designed as follows: s B = [f1, f2] (2) The state vector observed by the batch assignment agent is designed as follows: s S = [f2, f3] (3) At the time of dc sub-scheduling, the agent Ai considers the local state s of the wafer batch processing environment when making decisions dc and dynamic information m dc-1 Perform scheduling, the scheduling strategy of agent Ai can be represented as π i (s dc ,m dc-1 ; θ i , ψ), and the group batch scheduling strategy function π0 and the batch assignment scheduling strategy function π1 are approximated by two deep neural networks with parameters θ i ; The global state of the wafer batch processing environment is s dc , and the scheduling decision made by the agent Ai is a dc After the Critic is approximated by a deep neural network with parameters , it can be represented as:

6. The multi-agent coordinated semiconductor batch processing zone scheduling method of claim 1, wherein, In step S2, the state parameter model, the scheduling action parameter model and the scheduling feedback model of the wafer batch processing area scheduling environment are constructed as follows: State parameter model: Wafer batch processing scheduling includes three objects: task, batch, and device. The environment state is described by the vector F = [f1, f2, f3], where f1 = [f 1,1 ,...,f 1,n [] represents the state of tasks to be batched, where element f is... 1,j =[f 1,j,1 ,...,f 1,j,5 [For task J] j The feature vector, with subscripts 1-5 representing the task weight, task arrival time, number of layers to be processed, expected flow time, and wafer type of task j, respectively; f2 = [f 2,1 ,...,f 2,b ] represents the batch state, where element f is... 2,k =[f 2,k,1 ,...,f 2,k,2 ] indicates batch B k The feature vector, with subscripts 1 and 2 representing the batch size percentage and batch type, respectively; f3 = [f 3,1 ,...,f 3,m [] represents the device state, where element f is... 3,m =[f 3,m,1 ,...,f 3,m,3 [For device M] i The feature vector, with subscripts 1-3 representing the remaining processing time of the equipment, the current processing type of the equipment, and the current processing percentage of the equipment, respectively; The scheduling action model is a set of scheduling decisions executable by the wafer scheduling workshop in different states; the group batch scheduling decision space is based on the characteristics that the wafer manufacturing batch processing device is incompatible with the process type, a number of q group batch buffers are set for each process type, the group batch scheduling decision adds the current to-be-grouped task to a group batch buffer or temporarily groups, and the group batch scheduling decision space is defined as: Scheduling decision 1: select the kth group batch buffer, k represents the number of the buffer: a=k (0≤k≤q) Scheduling decision 2: wait: a=q (this case represents not grouping, and waiting) When making a group batch scheduling decision, if a group batch buffer is selected, i.e., scheduling decision 1, the current task to be grouped is added to the group batch buffer; if waiting is selected, i.e., scheduling decision 2, the task is temporarily delayed for grouping; if the batch number added by the group batch agent exceeds the maximum batch capacity, the grouping fails, and the result is equivalent to scheduling decision 2; The batch assignment scheduling decision selects all group batch buffers of all process types in the group batch buffer and matches to idle equipment for processing; the batch assignment decision space is a multiple of the process type of the group batch decision scheduling space; selecting a group batch buffer, the batch is matched to the equipment and processed, and the group batch buffer is emptied; selecting waiting means not selecting any batch for processing; The scheduling feedback model is a scheduling reward function. After the group batch agent makes a decision, the reward function is related to the current flow time, the subsequent expected flow time and the capacity proportion of the decision batch slot of the group batch task. Therefore, the group batch reward r B is the weighted sum of the current flow time, the subsequent expected flow time and the decision batch slot capacity proportion of the successful group batch: r B = -[W1(t-a ti )+W2F i +W3V j ]·sw B (t)-W4(1-sw B (t)) (5) where W1, W2, W3, W4 are weight factors, t is the current time, a ti is the time of the initial arrival of the tasks to be batched, F i is the subsequent expected flow time, V j is the capacity proportion of the decision batch slot, sw B (t) indicates whether the task is successfully batched at time t; Based on the objective of minimizing the wafer production cycle, the reward function of the batch assignment agent is as follows: where t is the current time, is the time at which the batch processing device earliest started processing, is the time at which the batch processing device group has already processed; Since the group batch and batch assignment agents jointly optimize the scheduling objective of minimizing the wafer production cycle, the reward values of the group batch and batch assignment are globally discounted to realize the mutual influence and correlation between the group batch and batch assignment. Taking the dc-time scheduling as an example, the globally discounted reward is represented as: where γ is a discount factor, r t is the reward value at time t, dc' represents the total number of scheduling, through task driving, constantly scheduling, obtaining a large amount of workshop scheduling interaction data <s, a, r, s'> and storing, the group batch agent and the equipment assignment agent constantly update the parameters through the feedback reward function, and realize the reward maximization, that is, the wafer manufacturing production cycle scheduling target minimization.

7. The multi-agent coordinated semiconductor batch processing zone scheduling method of claim 1, wherein, The training in steps S3 and S7 is specifically: The multi-agent model of the wafer batch processing workshop scheduling is trained; after the network parameters of the group batch and batch assignment agents are initialized, the corresponding state observation inputs are input to make decisions, and the network parameters are updated through the feedback reward function for continuous optimization; the model parameter updating method is improved based on the PPO algorithm, the GRU network is globally updated with the Actor and Critic to realize the synchronous optimization of the GRU network and the agent; the output of the GRU network is part of the input of the Actor network and the Critic network, and the gradient backpropagation to the prefix GRU network is realized when the Actor network and the Critic network are updated to realize global parameter optimization.

8. The multi-agent coordinated semiconductor batch processing zone scheduling method of claim 1, wherein, Steps S4, S5, and S6 are specifically: Design and training of the generative adversarial network and the relationship correction module; In the deep reinforcement learning process, the real experience sample set is a pair of (state s, action a), (subsequent state s', reward r), the state s at the previous moment corresponds to the corresponding action a, (s, a) is called a state-action pair, the next moment moves to the state s', and the immediate reward r is obtained, (s', r) is called a subsequent state-reward pair; therefore, the real experience sample set Dx=[s,a,s',r] can be divided into two parts: Dx=[(s,a),(s′,r)]=[(x1,x2)] (9) Where x1 represents a state-action pair, and x2 represents a subsequent state-reward pair; since the subsequent state s' and the reward r are based on the previous state s and the corresponding action a, there is a certain relationship between x1 and x2, and this relationship is represented by mutual information M: Where X1 represents the set of state-action pairs, X2 represents the set of subsequent state-reward pairs, p represents probability, E(x2) represents the entropy of x2, used to measure the uncertainty of x2; E(x2|x1) represents the uncertainty of x2 given x1; M(x2;x1) represents the reduction in uncertainty of x2 caused by x1; since x1 and x2 are correlated, the mutual information M cannot be 0; therefore, a deep neural network, namely the relation correction network, is constructed, with x1 as input and x2 as output, to train the intrinsic relationship between x1 and x2 in the empirical sample set; consistent with the empirical sample set, the empirical sample G(z) generated by GAN = [s z ,az,s′,r z It can also be divided into two corresponding parts: G(z) = [(s z ,a z ),(s′,r z )] = [G1(z), G2(z)] (11) Wherein, G1(z) represents the generated state-action pair, G2(z) represents the generated subsequent state-reward pair; in order to improve the quality of the generated sample, the structural relationship between the generated G2(z) and G1(z) should be consistent with the structural relationship in the real sample [x1, x2] on the basis of the generated sample G(z); therefore, the generated sample G1(z) is combined with the mutual information M, G1(z) is input into the relationship correction network, and the output thereof is taken as the corrected subsequent state-reward pair G2(z)'; the relative entropy, i.e. the KL divergence, is used to represent the similarity between G2(z) and G2(z)', and the formula is as follows: Wherein, P represents the generated subsequent state-reward pair G2(z), Q represents the corrected subsequent state-reward pair G2(z)', q(i) and p(i) represent the probabilities of the distribution Q and P on the event i respectively; When the similarity between the generated subsequent state-reward pair G2(z) and the corrected subsequent state-reward pair G2(z)' is very high, the relative entropy of the two tends to 0, the quality of the sample generated by the GAN is higher, and the value function W(D, G) can be represented as: Wherein, D(x) is the probability estimation of the discriminator on the real sample x being "real", G(z) represents that the generator maps the noise z~Pz into the pseudo sample, V represents the adversarial objective function, D represents the discriminator, G represents the generator, and K is a weight parameter; meanwhile, the GAN model is optimized by the L2 regularization method.

9. The multi-agent collaborated semiconductor batch processing zone scheduling method of claim 1, wherein, The generative adversarial network is a Corr-GAN, and the network structure of the Corr-GAN is as follows: GAN, which is used to generate experience samples; a real sample pool D1 and a virtual sample pool D2, D1 is used to collect real experience samples obtained in the interaction process of the agent and the scheduling environment, and D2 is used to collect samples generated by the GAN, and the two together provide training samples for the agent; A relationship correction module, which represents a constructed deep neural network, is used to train the relationship between the state-action pair [s, a] and the subsequent state-reward pair [s', r] in D1 to guide the generator model G to improve the performance; in addition, the relationship correction module also analyzes the internal relationship between [s, a] and [s', r] in D1, inputs the state-action pair [s, a] generated by the generator model G into the relationship correction module, and outputs the subsequent state-reward pair [s'', r'] conforming to the internal relationship, and then compares the subsequent state-reward pair [s'', r'] with the subsequent state-reward pair [s', r] generated by the generator model G, to further prompt the generator model G to generate more actual samples; Before training the GAN and the relationship correction network, experience data needs to be collected from the actual scheduling environment; in the training stage, the GAN is trained to generate virtual data; The relationship correction network learns the relationship between the real data and the virtual data; In the data enhancement stage, the system respectively calls the pre-trained generative adversarial network GAN and the relationship correction module to cooperatively generate virtual samples with higher quality; finally, in the agent pre-training stage, the multi-agent deep reinforcement learning model is trained by using the virtual data and the historical real data.

10. The multi-agent coordinated semiconductor batch processing zone scheduling method of claim 1, wherein, The deployment process in step S8 is as follows: The wafer lots dynamically arrived for tasks are put into corresponding batch slots by a batch grouping agent. The number of batches for grouping is fixed, and the values are assigned according to the actual scheduling process. The process type of the wafer lots in the batch slot is determined by the process type of the first wafer lot entering the slot. When the equipment is idle, the corresponding batch slot is scheduled by a batch assignment agent, and the wafer is processed in the batch processing equipment. After processing one layer, the wafer is re-entered for processing the next layer, until all the wafer lots are processed to the last layer.

Images