Real-time scheduling method at wafer fab level in semiconductor manufacturing based on deep reinforcement learning

Abstract

The present invention provides a wafer fab-level real-time scheduling method in semiconductor manufacturing based on deep reinforcement learning, which relates to the field of semiconductor technology, including building a state space vector based on deep reinforcement learning by collecting equipment status, batch information and process flow information, and inputting a pre-trained deep reinforcement learning network model, which includes a value evaluation network and a policy network. The model generates a matching probability distribution between batches and equipment, and uses a hierarchical reinforcement learning method to decompose it into two subtasks of batch selection and equipment allocation to generate multiple candidate matching schemes. Then, a multi-objective reward and punishment function is applied to evaluate the candidate schemes, and the optimal scheduling scheme is selected in combination with the Monte Carlo tree search algorithm, and processing instructions are issued. Real-time processing data is collected during the process to update the state space vector and realize real-time scheduling.

Description

Technical Field

[0001] The present invention relates to semiconductor technology, and in particular to a wafer fab-level real-time scheduling method in semiconductor manufacturing based on deep reinforcement learning. Background Art

[0002] The wafer manufacturing process is highly complex, involving hundreds of processes and a large number of precision equipment. How to efficiently schedule these resources to maximize production capacity and minimize production cycle and cost has always been a research hotspot in the industry. Traditional wafer fab scheduling methods mainly rely on rule-making and empirical judgment, such as first-come-first-served and shortest processing time. In addition, some advanced scheduling methods, such as mathematical programming and heuristic algorithms, have also been applied to wafer fab scheduling problems.

[0003] However, traditional wafer fab scheduling methods have some limitations. First, rule-based scheduling methods are difficult to adapt to complex production environments. When production conditions change, such as equipment failures, emergency order insertions, etc., these rules often cannot make optimal scheduling decisions. Secondly, although mathematical programming methods can find the optimal solution in theory, their computational complexity is high and it is difficult to apply to real-time scheduling scenarios, especially in large-scale wafer fabs. Finally, traditional scheduling methods usually lack the ability to predict future production status and cannot effectively deal with uncertainties in the production process.

[0004] These defects lead to low utilization of wafer fab resources, long production cycles, and high production costs, which restrict the further development of the semiconductor manufacturing industry. Therefore, it is crucial to develop a wafer fab scheduling method that can adapt to complex production environments, respond to production changes in real time, and have predictive capabilities. Summary of the invention

[0005] The embodiments of the present invention provide a wafer fab-level real-time scheduling method in semiconductor manufacturing based on deep reinforcement learning, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] Provides a real-time scheduling method for wafer fab-level in semiconductor manufacturing based on deep reinforcement learning, including:

[0008] Collecting wafer fab equipment status information, wafer batch information, and processing process information through a production management system, extracting equipment idle time, equipment maintenance plan time, and equipment failure rate based on the equipment status information, extracting batch waiting time, number of remaining batch processes, and batch priority based on the wafer batch information, extracting process processing time, process quality yield, and process changeover time based on the processing process information, and constructing the extracted information into a state space vector;

[0009] The state space vector is input into a pre-trained deep reinforcement learning network model, wherein the deep reinforcement learning network model includes a value evaluation network and a policy network, wherein the value evaluation network is used to evaluate the value score of the current equipment and batch combination, and the policy network is used to generate a matching probability distribution of the batch and the equipment according to the value score; based on the matching probability distribution, a hierarchical reinforcement learning method is used to decompose the batch and equipment matching process into two subtasks, batch selection and equipment allocation, the priority sequence of the batches to be processed is determined by the batch selection subtask, and then the equipment allocation subtask matches the appropriate equipment for each batch to be processed, and generates multiple candidate matching schemes; each candidate matching scheme is evaluated by applying a preset multi-objective reward and punishment function, and the evaluation result of the multi-objective reward and punishment function is used as the selection weight of the Monte Carlo tree search algorithm, and the optimal scheduling scheme for the batch and the equipment is selected through multiple rounds of search iterations;

[0010] According to the optimal scheduling plan, processing instructions are issued to the corresponding equipment through the equipment control system, real-time processing data is collected during the process, the state space vector is updated based on the real-time processing data, and the updated state space vector is re-input into the deep reinforcement learning network model for real-time generation of a scheduling plan.

[0011] In an optional embodiment,

[0012] The step of inputting the state space vector into a pre-trained deep reinforcement learning network model, wherein the deep reinforcement learning network model includes a value evaluation network and a policy network, wherein the value evaluation network is used to evaluate the value score of the current device and batch combination, and the policy network is used to generate a matching probability distribution of the batch and the device according to the value score comprises:

[0013] Inputting a state space vector including device state information, wafer batch information, and processing process information into a deep reinforcement learning network model, wherein the device state information is encoded using real value, the wafer batch information is encoded using one-hot encoding, and the processing process information is encoded using position encoding, normalizing different features through a normalization layer, and mapping them to an embedding space of uniform dimension through a fully connected layer to generate a state representation vector;

[0014] The state representation vector is input into a value assessment network. The value assessment network adopts a multi-head self-attention mechanism to map the state representation vector into a query matrix, a key matrix and a value matrix respectively. An attention weight matrix is ​​generated according to a dot product operation of the query matrix and the key matrix. The attention weight matrix is ​​multiplied by the value matrix of the corresponding subspace to obtain a subspace attention representation. The attention representations of all subspaces are concatenated and fused. The fused features are input into a feedforward neural network containing a residual connection, and mapped into a value score after normalization.

[0015] The value score and the state representation vector are concatenated and input into the strategy network, the strategy network includes a double-layer gated recurrent unit network, the update gate of the double-layer gated recurrent unit network determines the retention ratio of historical information according to the correlation between the historical hidden state and the current input, and the reset gate calculates the forgetting ratio of historical information according to the value score and the state representation vector at the current moment, and the hidden state of the double-layer gated recurrent unit network is mapped through a fully connected layer with jump connections to generate a matching vector for the batch and the device, and the matching vector is converted into a matching probability distribution through a normalization function;

[0016] After obtaining the matching probability distribution, the value assessment network and the policy network are collaboratively trained using the mean square error and policy gradient method.

[0017] In an optional embodiment,

[0018] The steps of co-training the value assessment network and the policy network using the mean square error and policy gradient method include:

[0019] The value score of the value evaluation network is used as the benchmark function, and the target value is calculated by using temporal difference learning. The target value is composed of the immediate reward, the future multi-step reward and the predicted state value. The value network loss function is constructed based on the mean square error between the target value and the value score of the value evaluation network.

[0020] The policy gradient is calculated based on the matching probability distribution of the batch and the device output by the policy network. The calculation of the policy gradient uses the state value estimate output by the value evaluation network as a benchmark, takes the difference between the actual cumulative reward and the state value estimate as the advantage function, and calculates the policy gradient in combination with the log-likelihood of the probability output by the policy network;

[0021] The parameters of the value evaluation network and the policy network are updated by an alternating training method, wherein during the period when the policy network parameters are fixed, multiple state transition trajectories are generated based on the current policy sampling, and sequence samples including state transition, action selection, immediate reward and next state are collected, and the sequence samples are input into the value evaluation network to calculate the state value estimate, and the value evaluation network parameters are optimized by back propagation based on the value network loss function; then the value evaluation network parameters are fixed, and the state value estimate output by the optimized value evaluation network is used as the training benchmark of the policy network, and the policy network parameters are optimized based on the policy gradient method, and the optimized policy network is resampled and iteratively optimized;

[0022] Construct a fixed-capacity experience replay buffer, store the state-action-reward sequence obtained from each round of sampling in the buffer in a first-in-first-out manner, construct training batches based on historical samples in the buffer, set different learning rates for the value assessment network and the policy network to update parameters, introduce a regularization term based on the entropy of the policy distribution into the loss function of the policy network, and balance exploration and utilization by adjusting the entropy weight coefficient;

[0023] During the training process, the average error of the state value estimation and the average return of the policy network are monitored, and the training is stopped when the change rate of the average error and the average return are both less than a preset threshold.

[0024] In an optional embodiment,

[0025] Based on the matching probability distribution, a hierarchical reinforcement learning method is used to decompose the batch-to-equipment matching process into two subtasks: batch selection and equipment allocation. The batch selection subtask determines the priority sequence of the batches to be processed, and the equipment allocation subtask matches the appropriate equipment for each batch to be processed. The steps of generating multiple candidate matching solutions include:

[0026] Constructing a batch-equipment environment state space, wherein the environment state space includes a processing state of a set of batches to be processed, an equipment availability state, and a matching probability distribution between the batches and the equipment;

[0027] Decomposing the batch-to-device matching problem into a batch selection strategy and a device allocation strategy, constructing a batch selection strategy network and a device allocation strategy network, wherein the batch selection strategy network and the device allocation strategy network achieve collaborative optimization by sharing the environment state space;

[0028] Constructing a batch priority evaluation function based on the batch selection strategy network, inputting the processing state, equipment availability state and matching probability distribution in the environmental state space into the batch priority evaluation function, and using a Monte Carlo sampling method to generate multiple batch priority sequences;

[0029] For each batch priority sequence, the device allocation process is executed through the device allocation strategy network, the device allocation process takes the environment state space and the information of the batch to be allocated as input, and generates multiple device allocation plans by using different random seeds to initialize device allocation strategy parameters, setting different exploration parameters, and retaining multiple device selection plans at decision points;

[0030] A state transfer function is constructed to verify the feasibility of the generated equipment allocation plan, wherein the feasibility verification includes process precedent constraint verification, equipment process constraint verification, time window constraint verification and processing capacity constraint verification. The environmental state space and the equipment allocation plan are input into the state transfer function, the environmental state changes are deduced in time sequence, constraint violations are recorded, and a plan feasibility index is output. Based on the feasibility index, the equipment allocation plan that has passed the feasibility verification is used as a candidate matching plan.

[0031] In an optional embodiment,

[0032] The preset multi-objective reward and penalty function is applied to each candidate matching solution for evaluation, and the evaluation result of the multi-objective reward and penalty function is used as the selection weight of the Monte Carlo tree search algorithm. The steps of selecting the optimal scheduling solution for batches and equipment through multiple rounds of search iterations include:

[0033] Constructing a multi-objective reward and punishment function, the multi-objective reward and punishment function includes a production efficiency objective function, a quality consistency objective function, an energy consumption objective function and an equipment load balancing objective function, wherein the production efficiency objective function is composed of a batch completion time and an equipment waiting time, the quality consistency objective function is composed of a deviation between a batch quality parameter and a target reference value, the energy consumption objective function is composed of an equipment energy consumption coefficient, equipment power and operating time, and the equipment load balancing objective function is composed of a range of equipment load rates;

[0034] Assigning weight coefficients to each objective function of the multi-objective reward and punishment function to construct a comprehensive reward and punishment function;

[0035] Calculate the instant evaluation value of the comprehensive reward and punishment function for each candidate matching scheme, calculate the selection weight based on the instant evaluation value and the temperature parameter, and dynamically adjust the temperature parameter with the search iteration round; construct a Monte Carlo search tree based on the selection weight, take each candidate matching scheme as a search tree node, record the cumulative evaluation value of the comprehensive reward and punishment function, the number of node visits and the corresponding selection weight at each node, and use the upper confidence bound criterion to select and expand the node;

[0036] The optimal scheduling scheme is determined based on the cumulative evaluation value, wherein the search process ends when the iteration termination condition is met, and the iteration termination condition includes reaching the maximum number of iterations, the cumulative evaluation value of the optimal scheduling scheme has not improved for multiple consecutive rounds, and the calculation time reaches a threshold. The optimal scheduling scheme includes a batch priority sorting sequence, a matching relationship between batches and equipment, a batch schedule, and a processing sequence of equipment.

[0037] In an optional embodiment,

[0038] The instant evaluation value of the comprehensive reward and punishment function is calculated for each candidate matching scheme, and the selection weight is calculated based on the instant evaluation value and the temperature parameter, and the temperature parameter is dynamically adjusted with the search iteration round; a Monte Carlo search tree is constructed based on the selection weight, each candidate matching scheme is used as a search tree node, and the cumulative evaluation value of the comprehensive reward and punishment function, the number of node visits and the corresponding selection weight are recorded at each node. The steps of using the upper confidence bound criterion to select and expand nodes include:

[0039] A selection weight is calculated based on the instant evaluation value through Boltzmann distribution, wherein a temperature parameter is introduced in the calculation process of the selection weight, the temperature parameter is dynamically adjusted according to the iteration rounds using an exponential decay strategy, the initial value of the temperature parameter is set by a temperature initial value parameter, and the decay rate of the temperature parameter is controlled by a decay coefficient;

[0040] An upper confidence bound criterion is used for node selection and expansion, wherein the calculation of the upper confidence bound criterion includes a node cumulative evaluation value item, a node selection weight item, and an access frequency item, wherein the node cumulative evaluation value item represents the historical performance of the node, the node selection weight item introduces heuristic information based on the comprehensive reward and punishment function, and the access frequency item is calculated based on the ratio of the number of parent node visits to the number of current node visits;

[0041] A node with the largest upper confidence bound criterion value is selected from all child nodes of the current node as an extended node, the cumulative evaluation value mean and the number of node visits of the extended node are updated, and the exploration parameters of the node are adjusted based on the updated number of node visits; and the optimal scheduling plan for batches and equipment corresponding to the node with the largest cumulative evaluation value mean is obtained through multiple rounds of iterative search.

[0042] In an optional embodiment,

[0043] The steps of issuing processing instructions to corresponding equipment through the equipment control system according to the optimal scheduling plan, collecting real-time processing data during the process, updating the state space vector based on the real-time processing data, and re-inputting the updated state space vector into the deep reinforcement learning network model for real-time generation of the scheduling plan include:

[0044] Sending a processing instruction to the corresponding equipment through the equipment control system, wherein the processing instruction includes a batch identification, equipment identification, process parameter configuration, expected start time and expected completion time;

[0045] Collect real-time processing data during the process, including actual start time, actual completion time, equipment idle waiting time, process yield of the current batch, equipment key parameter deviation, equipment state conversion record, maintenance event information and abnormal alarm information;

[0046] Constructing a state influence evaluation matrix based on the real-time processing data, wherein the state influence evaluation matrix includes an equipment state correlation submatrix, a batch state correlation submatrix, and a process parameter correlation submatrix, and is used to quantify the mutual influence relationship between the state parameters of equipment idle time, batch waiting time, and process processing time;

[0047] For the state parameters in the state impact evaluation matrix, a sliding time window mechanism is used to collect state transition data, and the state impact evaluation matrix is ​​dynamically updated through a temporal causal analysis method;

[0048] The state space vector is updated using the updated state impact evaluation matrix, where the update of the equipment state component takes into account the spillover effect of adjacent equipment, the update of the batch state component is based on the upstream batch completion rate, and the update of the process parameter component incorporates the impact of equipment state fluctuations;

[0049] The updated state space vector is input into the deep reinforcement learning network model to generate a new scheduling scheme, which is used for issuing the next round of processing instructions.

[0050] According to a second aspect of the embodiments of the present invention,

[0051] Provides a fab-level real-time scheduling system for semiconductor manufacturing based on deep reinforcement learning, including:

[0052] The first unit is used to collect wafer fab equipment status information, wafer batch information and processing process information through a production management system, extract equipment idle time, equipment maintenance plan time and equipment failure rate according to the equipment status information, extract batch waiting time, number of remaining batch processes and batch priority according to the wafer batch information, extract process processing time, process quality yield and process changeover time according to the processing process information, and construct the extracted information into a state space vector;

[0053] The second unit is used to input the state space vector into a pre-trained deep reinforcement learning network model, wherein the deep reinforcement learning network model includes a value evaluation network and a policy network, wherein the value evaluation network is used to evaluate the value score of the current equipment and batch combination, and the policy network is used to generate a matching probability distribution between the batch and the equipment according to the value score; based on the matching probability distribution, a hierarchical reinforcement learning method is used to decompose the batch and equipment matching process into two subtasks: batch selection and equipment allocation; the batch selection subtask determines the priority sequence of the batches to be processed, and then the equipment allocation subtask matches the appropriate equipment for each batch to be processed, and generates multiple candidate matching schemes; each candidate matching scheme is evaluated by applying a preset multi-objective reward and punishment function, and the evaluation result of the multi-objective reward and punishment function is used as the selection weight of the Monte Carlo tree search algorithm, and the optimal scheduling scheme for the batch and the equipment is selected through multiple rounds of search iterations;

[0054] The third unit is used to issue processing instructions to the corresponding equipment through the equipment control system according to the optimal scheduling plan, collect real-time processing data during the process processing, update the state space vector based on the real-time processing data, and re-input the updated state space vector into the deep reinforcement learning network model for real-time generation of a scheduling plan.

[0055] According to a third aspect of the embodiments of the present invention,

[0056] An electronic device is provided, comprising:

[0057] processor;

[0058] a memory for storing processor-executable instructions;

[0059] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0060] A fourth aspect of the embodiments of the present invention is:

[0061] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0062] The present invention collects equipment status, batch information and process flow information in real time, and combines it with a deep reinforcement learning model to dynamically optimize the matching of batches and equipment, reduce batch waiting time and equipment idle time, and thus improve the overall production efficiency of the wafer fab.

[0063] The present invention takes into account factors such as process quality and yield, and evaluates them in a multi-objective reward and punishment function, which can effectively avoid allocating batches to equipment with high failure rates or low yields, thereby improving the quality and yield of the final product.

[0064] The present invention uses real-time processing data to update the state space vector and re-inputs the deep reinforcement learning model for scheduling, which can quickly respond to changes in the production process, such as equipment failures, emergency orders, etc., thereby improving the flexibility and robustness of the scheduling plan. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a flowchart of a wafer fab-level real-time scheduling method in semiconductor manufacturing based on deep reinforcement learning according to an embodiment of the present invention;

[0066] Figure 2 Schematic diagram of the structure of a wafer fab-level real-time scheduling system in semiconductor manufacturing based on deep reinforcement learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0069] Figure 1 Schematic diagram of the process of a wafer fab-level real-time scheduling method in semiconductor manufacturing based on deep reinforcement learning according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0070] S1. Collect the wafer plant equipment status information, wafer batch information and processing process information through the production management system, extract the equipment idle time, equipment maintenance plan time and equipment failure rate according to the equipment status information, extract the batch waiting time, the number of remaining batch processes and batch priority according to the wafer batch information, extract the process processing time, process quality yield and process changeover time according to the processing process information, and construct the extracted information into a state space vector;

[0071] S2. Input the state space vector into a pre-trained deep reinforcement learning network model, wherein the deep reinforcement learning network model includes a value evaluation network and a policy network, wherein the value evaluation network is used to evaluate the value score of the current equipment and batch combination, and the policy network is used to generate a matching probability distribution between the batch and the equipment according to the value score; based on the matching probability distribution, a hierarchical reinforcement learning method is used to decompose the batch and equipment matching process into two subtasks: batch selection and equipment allocation. The priority sequence of the batches to be processed is determined by the batch selection subtask, and then the equipment allocation subtask matches the appropriate equipment for each batch to be processed, generating multiple candidate matching schemes; each candidate matching scheme is evaluated by applying a preset multi-objective reward and punishment function, and the evaluation result of the multi-objective reward and punishment function is used as the selection weight of the Monte Carlo tree search algorithm, and the optimal scheduling scheme for the batch and the equipment is selected through multiple rounds of search iterations;

[0072] S3. According to the optimal scheduling plan, processing instructions are issued to the corresponding equipment through the equipment control system, real-time processing data is collected during the process, the state space vector is updated based on the real-time processing data, and the updated state space vector is re-input into the deep reinforcement learning network model for real-time generation of a scheduling plan.

[0073] In an optional embodiment,

[0074] The step of inputting the state space vector into a pre-trained deep reinforcement learning network model, wherein the deep reinforcement learning network model includes a value evaluation network and a policy network, wherein the value evaluation network is used to evaluate the value score of the current device and batch combination, and the policy network is used to generate a matching probability distribution of the batch and the device according to the value score comprises:

[0075] Inputting a state space vector including device state information, wafer batch information, and processing process information into a deep reinforcement learning network model, wherein the device state information is encoded using real value, the wafer batch information is encoded using one-hot encoding, and the processing process information is encoded using position encoding, normalizing different features through a normalization layer, and mapping them to an embedding space of uniform dimension through a fully connected layer to generate a state representation vector;

[0076] The state representation vector is input into a value assessment network. The value assessment network adopts a multi-head self-attention mechanism to map the state representation vector into a query matrix, a key matrix and a value matrix respectively. An attention weight matrix is ​​generated according to a dot product operation of the query matrix and the key matrix. The attention weight matrix is ​​multiplied by the value matrix of the corresponding subspace to obtain a subspace attention representation. The attention representations of all subspaces are concatenated and fused. The fused features are input into a feedforward neural network containing a residual connection, and mapped into a value score after normalization.

[0077] The value score and the state representation vector are concatenated and input into the strategy network, the strategy network includes a double-layer gated recurrent unit network, the update gate of the double-layer gated recurrent unit network determines the retention ratio of historical information according to the correlation between the historical hidden state and the current input, and the reset gate calculates the forgetting ratio of historical information according to the value score and the state representation vector at the current moment, and the hidden state of the double-layer gated recurrent unit network is mapped through a fully connected layer with jump connections to generate a matching vector for the batch and the device, and the matching vector is converted into a matching probability distribution through a normalization function;

[0078] After obtaining the matching probability distribution, the value assessment network and the policy network are collaboratively trained using the mean square error and policy gradient method.

[0079] For example, in order to improve the efficiency and accuracy of equipment and batch matching in semiconductor production, a method for intelligent equipment and batch matching based on deep reinforcement learning is proposed. This method uses a pre-trained deep reinforcement learning network model to generate a matching probability distribution between batches and equipment based on equipment status, batch information, and process flow information, thereby realizing intelligent equipment and batch matching.

[0080] First, collect and preprocess relevant data in the production process, including equipment status information, wafer batch information, and processing process information. Equipment status information, such as equipment operating time, temperature, pressure, etc., is encoded using real values. Wafer batch information, such as batch size, wafer type, target parameters, etc., is encoded using one-hot encoding. Processing process information, such as process steps and process parameters, is encoded using position encoding. In order to eliminate the dimensional differences between different features, all features are normalized, such as Min-Max normalization or Z-score normalization. After that, the processed feature vector is input into the fully connected layer, mapped to an embedding space of uniform dimension, and a state representation vector is generated. For example, the 10-dimensional equipment status information, the 5-dimensional batch information, and the 20-dimensional process information are processed by normalization and the fully connected layer to generate a 128-dimensional state representation vector.

[0081] Next, the generated state representation vector is input into the value assessment network. The value assessment network adopts a multi-head self-attention mechanism. Specifically, the state representation vector is linearly transformed into a query matrix, a key matrix, and a value matrix respectively. For example, the 128-dimensional state representation vector is transformed into 8 64-dimensional query matrices, key matrices, and value matrices respectively, representing 8 attention heads. Then, the dot product of each query matrix and the corresponding key matrix is ​​calculated to obtain the attention weight matrix. The attention weight matrix is ​​multiplied by the corresponding value matrix to obtain the subspace attention representation of each attention head. Finally, the 8 64-dimensional subspace attention representations are concatenated to obtain a 512-dimensional feature vector. The feature vector is input into a feedforward neural network with residual connections, and after normalization, it is mapped to a value score. For example, the value score can represent the expected return or production efficiency of the current equipment and batch combination.

[0082] Then, the value score is concatenated with the state representation vector and input into the policy network. The policy network uses a two-layer gated recurrent unit network (GRU). The update gate of the two-layer GRU determines the retention ratio of historical information based on the correlation between the historical hidden state and the current input. The reset gate calculates the forgetting ratio of historical information based on the value score and the state representation vector at the current moment. For example, if the current value score is high, the reset gate will tend to forget more historical information in order to better utilize the current information. The hidden state of the two-layer GRU is mapped through a fully connected layer with jump connections to generate a matching vector for the batch and the device. For example, the matching vector can be a vector with a dimension equal to the number of devices, and each element represents the degree of matching between the current batch and the corresponding device. Finally, the matching vector is converted into a matching probability distribution through a normalization function such as Softmax.

[0083] Finally, the value evaluation network and the policy network are jointly trained using mean square error and policy gradient methods. For example, a reinforcement learning algorithm can be used to update the network parameters based on the reward signal of the environment to maximize the long-term cumulative reward.

[0084] The present invention automatically learns the matching strategy of equipment and batches through a deep reinforcement learning model, avoiding the tedious process of manual matching and thus improving the matching efficiency; it can generate the optimal matching solution according to the equipment status, batch information and process flow information, thereby optimizing resource allocation and improving production efficiency; through intelligent matching, the appropriate batch can be allocated to the most suitable equipment for processing, thereby improving product quality and yield.

[0085] In an optional embodiment,

[0086] The steps of co-training the value assessment network and the policy network using the mean square error and policy gradient method include:

[0087] The value score of the value evaluation network is used as the benchmark function, and the target value is calculated by using temporal difference learning. The target value is composed of the immediate reward, the future multi-step reward and the predicted state value. The value network loss function is constructed based on the mean square error between the target value and the value score of the value evaluation network.

[0088] The policy gradient is calculated based on the matching probability distribution of the batch and the device output by the policy network. The calculation of the policy gradient uses the state value estimate output by the value evaluation network as a benchmark, takes the difference between the actual cumulative reward and the state value estimate as the advantage function, and calculates the policy gradient in combination with the log-likelihood of the probability output by the policy network;

[0089] The parameters of the value evaluation network and the policy network are updated by an alternating training method, wherein during the period when the policy network parameters are fixed, multiple state transition trajectories are generated based on the current policy sampling, and sequence samples including state transition, action selection, immediate reward and next state are collected, and the sequence samples are input into the value evaluation network to calculate the state value estimate, and the value evaluation network parameters are optimized by back propagation based on the value network loss function; then the value evaluation network parameters are fixed, and the state value estimate output by the optimized value evaluation network is used as the training benchmark of the policy network, and the policy network parameters are optimized based on the policy gradient method, and the optimized policy network is resampled and iteratively optimized;

[0090] Construct a fixed-capacity experience replay buffer, store the state-action-reward sequence obtained from each round of sampling in the buffer in a first-in-first-out manner, construct training batches based on historical samples in the buffer, set different learning rates for the value assessment network and the policy network to update parameters, introduce a regularization term based on the entropy of the policy distribution into the loss function of the policy network, and balance exploration and utilization by adjusting the entropy weight coefficient;

[0091] During the training process, the average error of the state value estimation and the average return of the policy network are monitored, and the training is stopped when the change rate of the average error and the average return are both less than a preset threshold.

[0092] For example, in order to achieve the coordinated training of the value evaluation network and the policy network to improve the decision-making ability of the intelligent agent in a complex environment, a reinforcement learning method based on mean square error and policy gradient is proposed. The core idea of ​​this method is to use the value evaluation network to evaluate the value of the state and guide the policy network to select actions. The parameters of the two networks are optimized by alternating training, and finally the intelligent agent learns the optimal strategy.

[0093] First, construct the value assessment network and the policy network. The input of the value assessment network is the state of the environment, and the output is the value score corresponding to the state. The input of the policy network is also the state of the environment, and the output is the probability distribution of taking different actions under the state. For example, the value assessment network can be a multi-layer perceptron, the input is the feature vector of the state, and the output is a scalar value representing the value of the state. The policy network can also be a multi-layer perceptron, the output is a vector, each element represents the probability of taking the corresponding action.

[0094] Next, initialize the experience replay buffer. This is a fixed-capacity queue that stores historical experiences of the agent's interactions with the environment, including states, actions, rewards, and next states. For example, the size of the buffer can be set to 10,000 to store the most recent 10,000 experiences.

[0095] Then, the training cycle begins. In each round of training, multiple state transition trajectories are generated based on the interaction between the current policy network and the environment. For example, if the agent plays 10 games in a game, a state transition trajectory will be generated for each game.

[0096] The collected state transition trajectories are stored in the experience replay buffer. If the buffer is full, the old experience is replaced in a first-in-first-out manner.

[0097] A batch of samples are randomly drawn from the experience replay buffer to train the value evaluation network. For example, 64 samples are randomly drawn from the buffer each time.

[0098] Calculate the target value. The target value consists of the immediate reward, the future multi-step reward, and the predicted state value. For example, one-step temporal difference learning can be used, and the target value is equal to the immediate reward plus the discount factor multiplied by the predicted value of the next state. Assuming the immediate reward of the current state is 1, the predicted value of the next state is 0.9, and the discount factor is 0.95, the target value is 1+0.95*0.9=1.855.

[0099] Calculate the value network loss function. The value network loss function is the mean square error between the target value and the value score output by the value evaluation network. For example, if the value score output by the value evaluation network is 0.8, the loss function is (1.855-0.8) 2 =1.113025.

[0100] Update the parameters of the value evaluation network through the back-propagation algorithm. For example, the Adam optimizer can be used with a learning rate of 0.001.

[0101] Next, the updated value evaluation network is used to calculate the state value estimate, which serves as the training benchmark for the policy network.

[0102] Calculate the policy gradient. The policy gradient is calculated using the state value estimate output by the value evaluation network as the benchmark, the difference between the actual cumulative reward and the state value estimate as the advantage function, and the policy gradient is calculated by combining the log-likelihood of the probability output by the policy network. For example, assuming the actual cumulative reward is 2 and the state value estimate is 1.5, the advantage function is 2-1.5=0.5.

[0103] Introduce a regularization term based on the entropy of the policy distribution into the loss function of the policy network. For example, the entropy weight coefficient can be set to 0.01.

[0104] Update the parameters of the policy network using the policy gradient method. For example, you can use the Adam optimizer with a learning rate of 0.0001.

[0105] Monitor the average error of the state value estimation and the average return of the policy network during training. For example, calculate the average error and average return every 100 training rounds.

[0106] The training is stopped when the rate of change of the average error and the average return are both less than a preset threshold. For example, the preset threshold can be set to 0.001.

[0107] By alternately training the value evaluation network and the policy network, the present invention can effectively utilize value information to guide policy learning, thereby accelerating the learning speed and enabling the intelligent agent to converge to the optimal policy more quickly; using the value evaluation network as a benchmark can effectively reduce the variance of the policy gradient, improve the stability of policy learning, and avoid the phenomenon of policy oscillation; the use of the experience replay buffer can break the correlation between data, improve the utilization efficiency of training data, enhance the generalization ability of the policy, and enable the intelligent agent to make reasonable decisions even in unseen states.

[0108] In an optional embodiment,

[0109] Based on the matching probability distribution, a hierarchical reinforcement learning method is used to decompose the batch-to-equipment matching process into two subtasks: batch selection and equipment allocation. The batch selection subtask determines the priority sequence of the batches to be processed, and the equipment allocation subtask matches the appropriate equipment for each batch to be processed. The steps of generating multiple candidate matching solutions include:

[0110] Constructing a batch-equipment environment state space, wherein the environment state space includes a processing state of a set of batches to be processed, an equipment availability state, and a matching probability distribution between the batches and the equipment;

[0111] Decomposing the batch-to-device matching problem into a batch selection strategy and a device allocation strategy, constructing a batch selection strategy network and a device allocation strategy network, wherein the batch selection strategy network and the device allocation strategy network achieve collaborative optimization by sharing the environment state space;

[0112] Constructing a batch priority evaluation function based on the batch selection strategy network, inputting the processing state, equipment availability state and matching probability distribution in the environmental state space into the batch priority evaluation function, and using a Monte Carlo sampling method to generate multiple batch priority sequences;

[0113] For each batch priority sequence, the device allocation process is executed through the device allocation strategy network, the device allocation process takes the environment state space and the information of the batch to be allocated as input, and generates multiple device allocation plans by using different random seeds to initialize device allocation strategy parameters, setting different exploration parameters, and retaining multiple device selection plans at decision points;

[0114] A state transfer function is constructed to verify the feasibility of the generated equipment allocation plan, wherein the feasibility verification includes process precedent constraint verification, equipment process constraint verification, time window constraint verification and processing capacity constraint verification. The environmental state space and the equipment allocation plan are input into the state transfer function, the environmental state changes are deduced in time sequence, constraint violations are recorded, and a plan feasibility index is output. Based on the feasibility index, the equipment allocation plan that has passed the feasibility verification is used as a candidate matching plan.

[0115] For example, the intelligent matching method of batches and equipment decomposes the complex matching process into two subtasks: batch selection and equipment allocation through hierarchical reinforcement learning technology. This method can effectively handle large-scale batch and equipment matching problems and generate multiple feasible candidate solutions to meet different production needs.

[0116] First, the environment state space of batches and equipment is constructed. This state space contains three key information: the processing status of the batch to be processed, such as batch ID, arrival time, process requirements, etc.; the available status of the equipment, such as equipment ID, current status (idle, busy), executable processes, etc.; and the matching probability distribution between batches and equipment, which can be obtained based on historical data or expert experience, reflecting the success rate or efficiency of processing a specific batch on a specific equipment. For example, assuming there are three batches A, B, and C and two equipment D1 and D2, the state space can be represented as a table, where rows represent batches, columns represent equipment, and the values ​​in the table represent matching probabilities. For example, the matching probability of batch A on equipment D1 is 0.8, and the matching probability on equipment D2 is 0.5.

[0117] Next, the batch-to-device matching problem is decomposed into a batch selection strategy and a device allocation strategy, and the corresponding policy networks are constructed respectively. The two policy networks share the previously constructed environment state space to achieve collaborative optimization. The policy network can use a deep neural network, such as a multi-layer perceptron or a convolutional neural network.

[0118] Then, a batch priority evaluation function is constructed based on the batch selection strategy network. The processing state, equipment availability state and matching probability distribution in the environment state space are input into the function, and the Monte Carlo sampling method is used to generate multiple batch priority sequences. For example, assuming there are three batches A, B, and C, multiple batch priority sequences can be generated through Monte Carlo sampling, such as [A, B, C], [B, C, A], [C, A, B], etc.

[0119] For each generated batch priority sequence, the device allocation process is performed through the device allocation strategy network. This process takes the environment state space and the information of the batch to be allocated as input, and generates multiple device allocation schemes by initializing the device allocation strategy parameters with different random seeds, setting different exploration parameters, and retaining multiple device selection schemes at decision points. For example, for the batch priority sequence [A, B, C], the device allocation strategy network will first select a suitable device for batch A, assuming that device D1 is selected. Then, considering that device D1 has been assigned to batch A, select a suitable device for batch B, assuming that device D2 is selected. Finally, select a suitable device for batch C. By adopting different random seeds and exploration parameters, multiple device allocation schemes can be generated, such as [A-D1, B-D2, C-D1], [A-D2, B-D1, C-D2], etc.

[0120] Finally, a state transfer function is constructed to verify the feasibility of the generated equipment allocation plan. This verification includes process pre-order constraint verification, such as batch B must be processed after batch A is completed; equipment process constraint verification, such as equipment D1 can only process specific types of batches; time window constraint verification, such as batch A must be processed within a specific time period; and processing capacity constraint verification, such as equipment D1 can only process one batch within the same time period. The environmental state space and equipment allocation plan are input into the state transfer function, the environmental state changes are deduced in time sequence, the constraint violations are recorded, and the feasibility index of the plan is output. For example, for the plan [A-D1, B-D2, C-D1], if the processing time of batch C conflicts with the processing time of batch A on equipment D1, the plan is considered infeasible. According to the feasibility index, the verified equipment allocation plan is output as a candidate matching plan.

[0121] Through hierarchical reinforcement learning and Monte Carlo sampling, the present invention can quickly generate multiple candidate matching solutions, and screen out the optimal solution through feasibility verification, thereby improving matching efficiency; it can comprehensively consider batch processing status, equipment availability status and matching probability distribution, thereby optimizing equipment allocation and improving resource utilization; by adopting different random seeds, exploration parameters and retaining multiple equipment selection solutions, it can generate diversified matching solutions to adapt to different production needs and scenarios.

[0122] In an optional embodiment,

[0123] The preset multi-objective reward and penalty function is applied to each candidate matching solution for evaluation, and the evaluation result of the multi-objective reward and penalty function is used as the selection weight of the Monte Carlo tree search algorithm. The steps of selecting the optimal scheduling solution for batches and equipment through multiple rounds of search iterations include:

[0124] Constructing a multi-objective reward and punishment function, the multi-objective reward and punishment function includes a production efficiency objective function, a quality consistency objective function, an energy consumption objective function and an equipment load balancing objective function, wherein the production efficiency objective function is composed of a batch completion time and an equipment waiting time, the quality consistency objective function is composed of a deviation between a batch quality parameter and a target reference value, the energy consumption objective function is composed of an equipment energy consumption coefficient, equipment power and operating time, and the equipment load balancing objective function is composed of a range of equipment load rates;

[0125] Assigning weight coefficients to each objective function of the multi-objective reward and punishment function to construct a comprehensive reward and punishment function;

[0126] Calculate the instant evaluation value of the comprehensive reward and punishment function for each candidate matching scheme, calculate the selection weight based on the instant evaluation value and the temperature parameter, and dynamically adjust the temperature parameter with the search iteration round; construct a Monte Carlo search tree based on the selection weight, take each candidate matching scheme as a search tree node, record the cumulative evaluation value of the comprehensive reward and punishment function, the number of node visits and the corresponding selection weight at each node, and use the upper confidence bound criterion to select and expand the node;

[0127] The optimal scheduling scheme is determined based on the cumulative evaluation value, wherein the search process ends when the iteration termination condition is met, and the iteration termination condition includes reaching the maximum number of iterations, the cumulative evaluation value of the optimal scheduling scheme has not improved for multiple consecutive rounds, and the calculation time reaches a threshold. The optimal scheduling scheme includes a batch priority sorting sequence, a matching relationship between batches and equipment, a batch schedule, and a processing sequence of equipment.

[0128] Exemplarily, first, a multi-objective reward and punishment function is constructed. This function consists of four sub-objective functions: production efficiency objective function, quality consistency objective function, energy consumption objective function and equipment load balancing objective function. The production efficiency objective function takes into account the batch completion time and equipment waiting time. For example, the shorter the completion time, the higher the score; the shorter the waiting time, the higher the score. The quality consistency objective function takes into account the deviation of the batch quality parameter from the target reference value. For example, the smaller the deviation, the higher the score. The energy consumption objective function takes into account the equipment energy consumption coefficient, equipment power and operating time. For example, the lower the energy consumption, the higher the score. The equipment load balancing objective function takes into account the extreme difference of the equipment load rate. For example, the smaller the load rate extreme difference, the higher the score.

[0129] Next, assign weight coefficients to each objective function to construct a comprehensive reward and punishment function. For example, the weight of the production efficiency objective function is 0.4, the weight of the quality consistency objective function is 0.3, the weight of the energy consumption objective function is 0.2, and the weight of the equipment load balancing objective function is 0.1. Assuming that the production efficiency score of a scheduling plan is 80, the quality consistency score is 90, the energy consumption score is 70, and the equipment load balancing score is 60, the comprehensive score of the plan is 0.4*80+0.3*90+0.2*70+0.1*60=79.

[0130] Then, the instant evaluation value of the comprehensive reward and punishment function is calculated for each candidate matching scheme. Assuming there are three candidate schemes, their comprehensive scores are calculated respectively. The selection weight is calculated based on the instant evaluation value and the temperature parameter. The initial value of the temperature parameter is set to 1, and it is dynamically adjusted with the search iteration rounds. For example, the temperature parameter is reduced by 0.1 every 100 iterations. A Monte Carlo search tree is constructed based on the selection weight. Each candidate matching scheme is used as a search tree node, and the cumulative evaluation value of the comprehensive reward and punishment function, the number of node visits, and the corresponding selection weight are recorded at each node. The upper confidence bound criterion is used for node selection and expansion. The larger the selection weight, the higher the probability of being selected.

[0131] Finally, the optimal scheduling scheme is determined based on the cumulative evaluation value. The search process ends when the iteration termination condition is met. The iteration termination conditions include reaching the maximum number of iterations (for example, 1000 times), the cumulative evaluation value of the optimal scheduling scheme has not improved for multiple consecutive rounds (for example, 50 rounds), and the calculation time reaches a threshold (for example, 30 minutes). Assuming that after multiple rounds of iterations, the cumulative evaluation value of scheme 2 is the highest, then scheme 2 is the optimal scheduling scheme. The optimal scheduling scheme includes the batch priority sorting sequence (for example, batch A, batch B, batch C), the matching relationship between batches and equipment (for example, batch A is assigned to equipment 1, batch B is assigned to equipment 2, and batch C is assigned to equipment 3), the time arrangement of batches (for example, batch A starts processing at 8:00, batch B starts processing at 9:00, and batch C starts processing at 10:00) and the processing order of equipment (for example, equipment 1 processes batch A first, and then batch D).

[0132] The present invention improves production efficiency by optimizing the matching and time arrangement of batches and equipment, reducing equipment waiting time and batch completion time; selects a better scheduling plan by considering the deviation between batch quality parameters and target reference values, thereby improving product quality consistency; and selects a scheduling plan with lower energy consumption by considering equipment energy consumption coefficient, equipment power and operating time, thereby reducing energy consumption.

[0133] In an optional embodiment,

[0134] The instant evaluation value of the comprehensive reward and punishment function is calculated for each candidate matching scheme, and the selection weight is calculated based on the instant evaluation value and the temperature parameter, and the temperature parameter is dynamically adjusted with the search iteration round; a Monte Carlo search tree is constructed based on the selection weight, each candidate matching scheme is used as a search tree node, and the cumulative evaluation value of the comprehensive reward and punishment function, the number of node visits and the corresponding selection weight are recorded at each node. The steps of using the upper confidence bound criterion to select and expand nodes include:

[0135] A selection weight is calculated based on the instant evaluation value through Boltzmann distribution, wherein a temperature parameter is introduced in the calculation process of the selection weight, the temperature parameter is dynamically adjusted according to the iteration rounds using an exponential decay strategy, the initial value of the temperature parameter is set by a temperature initial value parameter, and the decay rate of the temperature parameter is controlled by a decay coefficient;

[0136] An upper confidence bound criterion is used for node selection and expansion, wherein the calculation of the upper confidence bound criterion includes a node cumulative evaluation value item, a node selection weight item, and an access frequency item, wherein the node cumulative evaluation value item represents the historical performance of the node, the node selection weight item introduces heuristic information based on the comprehensive reward and punishment function, and the access frequency item is calculated based on the ratio of the number of parent node visits to the number of current node visits;

[0137] A node with the largest upper confidence bound criterion value is selected from all child nodes of the current node as an extended node, the cumulative evaluation value mean and the number of node visits of the extended node are updated, and the exploration parameters of the node are adjusted based on the updated number of node visits; and the optimal scheduling plan for batches and equipment corresponding to the node with the largest cumulative evaluation value mean is obtained through multiple rounds of iterative search.

[0138] Exemplarily, the information of the batches and equipment to be scheduled is obtained, including the processing time of each batch, resource requirements, and the available time and resource capacity of each equipment. For example, suppose there are three batches, which require processing time of 2 hours, 3 hours, and 1 hour respectively, and resource requirements of 10 units, 15 units, and 5 units respectively. At the same time, there are two equipments, both of which have an available time of 5 hours and a resource capacity of 20 units.

[0139] Calculate the instant evaluation value of the comprehensive reward and penalty function for each candidate matching solution. The comprehensive reward and penalty function can take into account factors such as batch completion time, resource utilization, and device load balance. For example, a solution assigns batch 1 to device 1, batch 2 to device 2, and batch 3 to device 1. Then, an instant evaluation value of, for example, 80 points, can be calculated based on the solution's completion time, resource utilization, and device load balance.

[0140] The selection weight is calculated based on the instant evaluation value and the temperature parameter. The temperature parameter is dynamically adjusted with the search iteration rounds, using an exponential decay strategy. The initial value of the temperature parameter is set by the temperature initial value parameter, for example, set to 10. The decay coefficient controls the decay rate of the temperature parameter, for example, set to 0.9. Assuming that the current iteration round is 1, the temperature parameter is 10. Based on the instant evaluation value of 80 points and the temperature parameter of 10, the selection weight of the scheme is calculated by the Boltzmann distribution, for example, 0.6.

[0141] Construct a Monte Carlo search tree. Take each candidate matching solution as a search tree node. Record the cumulative evaluation value of the comprehensive reward and penalty function, the number of node visits, and the corresponding selection weight at each node. For example, the root node represents the initial state, and its child nodes represent solutions for allocating the first batch to different devices.

[0142] The upper confidence bound criterion is used for node selection and expansion. The calculation of the upper confidence bound criterion includes the node cumulative evaluation value item, the node selection weight item and the access frequency item. The node cumulative evaluation value item represents the historical performance of the node, the node selection weight item introduces the heuristic information based on the comprehensive reward and punishment function, and the access frequency item is calculated based on the ratio of the number of parent node visits to the number of current node visits. For example, if a node has a cumulative evaluation value of 85 points, a selection weight of 0.7, a parent node visit number of 10, and a current node visit number of 2, the upper confidence bound value of the node can be calculated.

[0143] Select the node with the largest upper confidence limit value from all child nodes of the current node as the expansion node. Update the cumulative evaluation value mean and node visit count of the expansion node, and adjust the node exploration parameters based on the updated node visit count. For example, select the node with the largest upper confidence limit value, assign the corresponding batch to the corresponding device, calculate the new instant evaluation value, and update the cumulative evaluation value mean and node visit count.

[0144] Repeat multiple rounds of iterative search until the preset number of iterations is reached or a solution that meets the termination condition is found. For example, 100 rounds of iterative search are performed.

[0145] Finally, the optimal scheduling solution for the batch and equipment corresponding to the node with the largest cumulative evaluation value mean is obtained. For example, after 100 rounds of iterative search, a solution with the highest cumulative evaluation value mean is found, and this solution is the optimal scheduling solution.

[0146] The present invention can quickly explore the search space through the Monte Carlo search tree and the upper confidence bound criterion, avoid traversing all possible scheduling schemes, and thus improve the scheduling efficiency; the comprehensive reward and punishment function takes into account factors such as resource utilization, and can find a better scheduling scheme to improve resource utilization; the dynamic adjustment mechanism of the temperature parameter can adjust the balance between exploration and utilization according to the search process, so that the algorithm can better adapt to different scheduling scenarios.

[0147] In an optional embodiment,

[0148] The steps of issuing processing instructions to corresponding equipment through the equipment control system according to the optimal scheduling plan, collecting real-time processing data during the process, updating the state space vector based on the real-time processing data, and re-inputting the updated state space vector into the deep reinforcement learning network model for real-time generation of the scheduling plan include:

[0149] Sending a processing instruction to the corresponding equipment through the equipment control system, wherein the processing instruction includes a batch identification, equipment identification, process parameter configuration, expected start time and expected completion time;

[0150] Collect real-time processing data during the process, including actual start time, actual completion time, equipment idle waiting time, process yield of the current batch, equipment key parameter deviation, equipment state conversion record, maintenance event information and abnormal alarm information;

[0151] Constructing a state influence evaluation matrix based on the real-time processing data, wherein the state influence evaluation matrix includes an equipment state correlation submatrix, a batch state correlation submatrix, and a process parameter correlation submatrix, and is used to quantify the mutual influence relationship between the state parameters of equipment idle time, batch waiting time, and process processing time;

[0152] For the state parameters in the state impact evaluation matrix, a sliding time window mechanism is used to collect state transition data, and the state impact evaluation matrix is ​​dynamically updated through a temporal causal analysis method;

[0153] The state space vector is updated using the updated state impact evaluation matrix, where the update of the equipment state component takes into account the spillover effect of adjacent equipment, the update of the batch state component is based on the upstream batch completion rate, and the update of the process parameter component incorporates the impact of equipment state fluctuations;

[0154] The updated state space vector is input into the deep reinforcement learning network model to generate a new scheduling scheme, which is used for issuing the next round of processing instructions.

[0155] Exemplarily, first, a deep reinforcement learning network model is constructed. The input of the model is a state space vector, and the output is a scheduling plan. The state space vector contains information such as equipment status, batch status, and process parameters, and the scheduling plan contains processing instructions for each batch, such as batch identification, equipment identification, process parameter configuration, expected start time, and expected completion time. The initial scheduling plan is generated by empirical rules or other scheduling algorithms as the initial training data of the deep reinforcement learning model.

[0156] Then, the processing instructions are issued to the corresponding equipment through the equipment control system. For example, the instructions for processing batch A are issued to equipment No. 1, and the process parameters are set as temperature 80°C, pressure 1MPa, expected start time 8:00, and expected completion time 10:00.

[0157] During the process, the processing data is collected in real time. For example, the actual start time of batch A on equipment No. 1 is recorded as 8:05, the actual completion time is 10:10, the equipment idle waiting time is 5 minutes, the process yield of batch A is 98%, the deviation of key equipment parameters is 2°C temperature fluctuation, 0.1MPa pressure fluctuation, the equipment state transition record is "standby-run-standby", the maintenance event information is empty, and the abnormal alarm information is empty.

[0158] Next, a state impact evaluation matrix is ​​constructed based on real-time processing data. This matrix contains the equipment state correlation submatrix, batch state correlation submatrix, and process parameter correlation submatrix. For example, the equipment state correlation submatrix describes the impact of the idle time of equipment No. 1 on equipment No. 2, the batch state correlation submatrix describes the impact of the waiting time of batch A on batch B, and the process parameter correlation submatrix describes the impact of temperature fluctuations on pressure fluctuations. These impacts are obtained by analyzing historical data and expert experience.

[0159] For the state parameters in the state impact evaluation matrix, a sliding time window mechanism is used to collect state transition data, such as collecting the state transition data of device No. 1 in the past 24 hours. Then, the state impact evaluation matrix is ​​dynamically updated through the time series causal analysis method. For example, if it is found that the idle time of device No. 1 has increased its impact on device No. 2 in the past 24 hours, the value in the device state association submatrix is ​​adjusted accordingly.

[0160] The state space vector is updated using the updated state impact evaluation matrix. The update of the equipment state component takes into account the spillover effect of adjacent equipment. For example, if the idle time of equipment No. 1 increases, the waiting time of equipment No. 2 may also increase. The update of the batch state component is based on the completion rate of the upstream batch. For example, if the completion rate of batch A is lower than expected, the start time of batch B may need to be delayed. The update of the process parameter component incorporates the impact of equipment state fluctuations. For example, if the temperature fluctuation of equipment No. 1 is large, the temperature setting value of batch B needs to be adjusted.

[0161] The updated state space vector is input into the deep reinforcement learning network model to generate a new scheduling plan. For example, the model may decide to transfer the processing task of batch B from equipment No. 1 to equipment No. 3 to avoid the impact of the fluctuation of equipment No. 1 on batch B.

[0162] The new scheduling scheme is used to issue the next round of processing instructions, forming a closed-loop control. By continuously iterating this process, the deep reinforcement learning model continues to learn and optimize, ultimately achieving efficient dynamic scheduling.

[0163] The present invention uses real-time data feedback and dynamic adjustment. This method can effectively reduce equipment idle time and batch waiting time, thereby improving production efficiency. Dynamic adjustment is performed according to equipment status fluctuations and process parameter deviations to ensure the stability of product quality. This method can adapt to changes in the production environment, such as equipment failures, material shortages, etc., and make corresponding adjustments to ensure production continuity. For example, when a certain device fails, the system can automatically transfer tasks to other available devices to avoid production interruptions.

[0164] Figure 2Schematic diagram of the structure of a wafer fab-level real-time scheduling system in semiconductor manufacturing based on deep reinforcement learning according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0165] The first unit is used to collect wafer fab equipment status information, wafer batch information and processing process information through a production management system, extract equipment idle time, equipment maintenance plan time and equipment failure rate according to the equipment status information, extract batch waiting time, number of remaining batch processes and batch priority according to the wafer batch information, extract process processing time, process quality yield and process changeover time according to the processing process information, and construct the extracted information into a state space vector;

[0166] The second unit is used to input the state space vector into a pre-trained deep reinforcement learning network model, wherein the deep reinforcement learning network model includes a value evaluation network and a policy network, wherein the value evaluation network is used to evaluate the value score of the current equipment and batch combination, and the policy network is used to generate a matching probability distribution between the batch and the equipment according to the value score; based on the matching probability distribution, a hierarchical reinforcement learning method is used to decompose the batch and equipment matching process into two subtasks: batch selection and equipment allocation; the batch selection subtask determines the priority sequence of the batches to be processed, and then the equipment allocation subtask matches the appropriate equipment for each batch to be processed, and generates multiple candidate matching schemes; each candidate matching scheme is evaluated by applying a preset multi-objective reward and punishment function, and the evaluation result of the multi-objective reward and punishment function is used as the selection weight of the Monte Carlo tree search algorithm, and the optimal scheduling scheme for the batch and the equipment is selected through multiple rounds of search iterations;

[0167] The third unit is used to issue processing instructions to the corresponding equipment through the equipment control system according to the optimal scheduling plan, collect real-time processing data during the process processing, update the state space vector based on the real-time processing data, and re-input the updated state space vector into the deep reinforcement learning network model for real-time generation of a scheduling plan.

[0168] According to a third aspect of the embodiments of the present invention,

[0169] An electronic device is provided, comprising:

[0170] processor;

[0171] a memory for storing processor-executable instructions;

[0172] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0173] A fourth aspect of the embodiments of the present invention is:

[0174] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0175] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A wafer fab-level real-time scheduling method in semiconductor manufacturing based on deep reinforcement learning, characterized in that: include: Collecting wafer fab equipment status information, wafer batch information, and processing process information through a production management system, extracting equipment idle time, equipment maintenance plan time, and equipment failure rate based on the equipment status information, extracting batch waiting time, number of remaining batch processes, and batch priority based on the wafer batch information, extracting process processing time, process quality yield, and process changeover time based on the processing process information, and constructing the extracted information into a state space vector; The state space vector is input into a pre-trained deep reinforcement learning network model, wherein the deep reinforcement learning network model includes a value evaluation network and a policy network, wherein the value evaluation network is used to evaluate the value score of the current equipment and batch combination, and the policy network is used to generate a matching probability distribution between the batch and the equipment according to the value score; based on the matching probability distribution, a hierarchical reinforcement learning method is used to decompose the batch and equipment matching process into two subtasks, batch selection and equipment allocation, the batch selection subtask determines the priority sequence of the batches to be processed, and then the equipment allocation subtask matches the appropriate equipment for each batch to be processed, and generates multiple candidate matching schemes; Each candidate matching solution is evaluated by applying a preset multi-objective reward and penalty function, and the evaluation result of the multi-objective reward and penalty function is used as the selection weight of the Monte Carlo tree search algorithm. The optimal scheduling solution for batches and equipment is selected through multiple rounds of search iterations. According to the optimal scheduling plan, processing instructions are issued to the corresponding equipment through the equipment control system, real-time processing data is collected during the process, the state space vector is updated based on the real-time processing data, and the updated state space vector is re-input into the deep reinforcement learning network model for real-time generation of a scheduling plan.

2. The method according to claim 1, characterized in that The step of inputting the state space vector into a pre-trained deep reinforcement learning network model, wherein the deep reinforcement learning network model includes a value evaluation network and a policy network, wherein the value evaluation network is used to evaluate the value score of the current device and batch combination, and the policy network is used to generate a matching probability distribution of the batch and the device according to the value score comprises: Inputting a state space vector including device state information, wafer batch information, and processing process information into a deep reinforcement learning network model, wherein the device state information is encoded using real value, the wafer batch information is encoded using one-hot encoding, and the processing process information is encoded using position encoding, normalizing different features through a normalization layer, and mapping them to an embedding space of uniform dimension through a fully connected layer to generate a state representation vector; The state representation vector is input into a value assessment network. The value assessment network adopts a multi-head self-attention mechanism to map the state representation vector into a query matrix, a key matrix and a value matrix respectively. An attention weight matrix is ​​generated according to a dot product operation of the query matrix and the key matrix. The attention weight matrix is ​​multiplied by the value matrix of the corresponding subspace to obtain a subspace attention representation. The attention representations of all subspaces are concatenated and fused. The fused features are input into a feedforward neural network containing a residual connection, and mapped into a value score after normalization. The value score and the state representation vector are concatenated and input into the strategy network, the strategy network includes a double-layer gated recurrent unit network, the update gate of the double-layer gated recurrent unit network determines the retention ratio of historical information according to the correlation between the historical hidden state and the current input, and the reset gate calculates the forgetting ratio of historical information according to the value score and the state representation vector at the current moment, and the hidden state of the double-layer gated recurrent unit network is mapped through a fully connected layer with jump connections to generate a matching vector for the batch and the device, and the matching vector is converted into a matching probability distribution through a normalization function; After obtaining the matching probability distribution, the value assessment network and the policy network are collaboratively trained using the mean square error and policy gradient method.

3. The method according to claim 2, characterized in that The steps of co-training the value assessment network and the policy network using the mean square error and policy gradient method include: The value score of the value evaluation network is used as the benchmark function, and the target value is calculated by using temporal difference learning. The target value is composed of the immediate reward, the future multi-step reward and the predicted state value. The value network loss function is constructed based on the mean square error between the target value and the value score of the value evaluation network. The policy gradient is calculated based on the matching probability distribution of the batch and the device output by the policy network. The calculation of the policy gradient uses the state value estimate output by the value evaluation network as a benchmark, takes the difference between the actual cumulative reward and the state value estimate as the advantage function, and calculates the policy gradient in combination with the log-likelihood of the probability output by the policy network; The parameters of the value evaluation network and the policy network are updated by an alternating training method, wherein during the period when the policy network parameters are fixed, multiple state transition trajectories are generated based on the current policy sampling, and sequence samples including state transition, action selection, immediate reward and next state are collected, and the sequence samples are input into the value evaluation network to calculate the state value estimate, and the value evaluation network parameters are optimized by back propagation based on the value network loss function; then the value evaluation network parameters are fixed, and the state value estimate output by the optimized value evaluation network is used as the training benchmark of the policy network, and the policy network parameters are optimized based on the policy gradient method, and the optimized policy network is resampled and iteratively optimized; Construct a fixed-capacity experience replay buffer, store the state-action-reward sequence obtained from each round of sampling in the buffer in a first-in-first-out manner, construct training batches based on historical samples in the buffer, set different learning rates for the value assessment network and the policy network to update parameters, introduce a regularization term based on the entropy of the policy distribution into the loss function of the policy network, and balance exploration and utilization by adjusting the entropy weight coefficient; During the training process, the average error of the state value estimation and the average return of the policy network are monitored, and the training is stopped when the change rate of the average error and the average return are both less than a preset threshold.

4. The method according to claim 1, characterized in that Based on the matching probability distribution, a hierarchical reinforcement learning method is used to decompose the batch-to-equipment matching process into two subtasks: batch selection and equipment allocation. The batch selection subtask determines the priority sequence of the batches to be processed, and the equipment allocation subtask matches the appropriate equipment for each batch to be processed. The steps of generating multiple candidate matching solutions include: Constructing a batch-equipment environment state space, wherein the environment state space includes a processing state of a set of batches to be processed, an equipment availability state, and a matching probability distribution between the batches and the equipment; Decomposing the batch-to-device matching problem into a batch selection strategy and a device allocation strategy, constructing a batch selection strategy network and a device allocation strategy network, wherein the batch selection strategy network and the device allocation strategy network achieve collaborative optimization by sharing the environment state space; Constructing a batch priority evaluation function based on the batch selection strategy network, inputting the processing state, equipment availability state and matching probability distribution in the environmental state space into the batch priority evaluation function, and using a Monte Carlo sampling method to generate multiple batch priority sequences; For each batch priority sequence, the device allocation process is executed through the device allocation strategy network, the device allocation process takes the environment state space and the information of the batch to be allocated as input, and generates multiple device allocation plans by using different random seeds to initialize device allocation strategy parameters, setting different exploration parameters, and retaining multiple device selection plans at decision points; A state transfer function is constructed to verify the feasibility of the generated equipment allocation plan, wherein the feasibility verification includes process precedent constraint verification, equipment process constraint verification, time window constraint verification and processing capacity constraint verification. The environmental state space and the equipment allocation plan are input into the state transfer function, the environmental state changes are deduced in time sequence, constraint violations are recorded, and a plan feasibility index is output. Based on the feasibility index, the equipment allocation plan that has passed the feasibility verification is used as a candidate matching plan.

5. The method according to claim 1, characterized in that The preset multi-objective reward and penalty function is applied to each candidate matching solution for evaluation, and the evaluation result of the multi-objective reward and penalty function is used as the selection weight of the Monte Carlo tree search algorithm. The steps of selecting the optimal scheduling solution for batches and equipment through multiple rounds of search iterations include: Constructing a multi-objective reward and punishment function, the multi-objective reward and punishment function includes a production efficiency objective function, a quality consistency objective function, an energy consumption objective function and an equipment load balancing objective function, wherein the production efficiency objective function is composed of a batch completion time and an equipment waiting time, the quality consistency objective function is composed of a deviation between a batch quality parameter and a target reference value, the energy consumption objective function is composed of an equipment energy consumption coefficient, equipment power and operating time, and the equipment load balancing objective function is composed of a range of equipment load rates; Assigning weight coefficients to each objective function of the multi-objective reward and punishment function to construct a comprehensive reward and punishment function; Calculate the instant evaluation value of the comprehensive reward and punishment function for each candidate matching scheme, calculate the selection weight based on the instant evaluation value and the temperature parameter, and dynamically adjust the temperature parameter with the search iteration round; construct a Monte Carlo search tree based on the selection weight, take each candidate matching scheme as a search tree node, record the cumulative evaluation value of the comprehensive reward and punishment function, the number of node visits and the corresponding selection weight at each node, and use the upper confidence bound criterion to select and expand the node; The optimal scheduling scheme is determined based on the cumulative evaluation value, wherein the search process ends when the iteration termination condition is met, and the iteration termination condition includes reaching the maximum number of iterations, the cumulative evaluation value of the optimal scheduling scheme has not improved for multiple consecutive rounds, and the calculation time reaches a threshold. The optimal scheduling scheme includes a batch priority sorting sequence, a matching relationship between batches and equipment, a batch schedule, and a processing sequence of equipment.

6. The method according to claim 5, characterized in that Calculating the instant evaluation value of the comprehensive reward and penalty function for each candidate matching solution, calculating the selection weight based on the instant evaluation value and a temperature parameter, and the temperature parameter is dynamically adjusted with the search iteration rounds; The steps of constructing a Monte Carlo search tree based on the selection weight, taking each candidate matching scheme as a search tree node, recording the cumulative evaluation value of the comprehensive reward and punishment function, the number of node visits and the corresponding selection weight at each node, and using the upper confidence bound criterion to select and expand nodes include: A selection weight is calculated based on the instant evaluation value through Boltzmann distribution, wherein a temperature parameter is introduced in the calculation process of the selection weight, the temperature parameter is dynamically adjusted according to the iteration rounds using an exponential decay strategy, the initial value of the temperature parameter is set by a temperature initial value parameter, and the decay rate of the temperature parameter is controlled by a decay coefficient; An upper confidence bound criterion is used for node selection and expansion, wherein the calculation of the upper confidence bound criterion includes a node cumulative evaluation value item, a node selection weight item, and an access frequency item, wherein the node cumulative evaluation value item represents the historical performance of the node, the node selection weight item introduces heuristic information based on the comprehensive reward and punishment function, and the access frequency item is calculated based on the ratio of the number of parent node visits to the number of current node visits; A node with the largest upper confidence bound criterion value is selected from all child nodes of the current node as an extended node, the cumulative evaluation value mean and the number of node visits of the extended node are updated, and the exploration parameters of the node are adjusted based on the updated number of node visits; and the optimal scheduling plan for batches and equipment corresponding to the node with the largest cumulative evaluation value mean is obtained through multiple rounds of iterative search.

7. The method according to claim 1, characterized in that The steps of issuing processing instructions to corresponding equipment through the equipment control system according to the optimal scheduling plan, collecting real-time processing data during the process, updating the state space vector based on the real-time processing data, and re-inputting the updated state space vector into the deep reinforcement learning network model for real-time generation of the scheduling plan include: Sending a processing instruction to the corresponding equipment through the equipment control system, wherein the processing instruction includes a batch identification, equipment identification, process parameter configuration, expected start time and expected completion time; Collect real-time processing data during the process, including actual start time, actual completion time, equipment idle waiting time, process yield of the current batch, equipment key parameter deviation, equipment state conversion record, maintenance event information and abnormal alarm information; Constructing a state influence evaluation matrix based on the real-time processing data, wherein the state influence evaluation matrix includes an equipment state correlation submatrix, a batch state correlation submatrix, and a process parameter correlation submatrix, and is used to quantify the mutual influence relationship between the state parameters of equipment idle time, batch waiting time, and process processing time; For the state parameters in the state impact evaluation matrix, a sliding time window mechanism is used to collect state transition data, and the state impact evaluation matrix is ​​dynamically updated through a temporal causal analysis method; The state space vector is updated using the updated state impact evaluation matrix, where the update of the equipment state component takes into account the spillover effect of adjacent equipment, the update of the batch state component is based on the upstream batch completion rate, and the update of the process parameter component incorporates the impact of equipment state fluctuations; The updated state space vector is input into the deep reinforcement learning network model to generate a new scheduling scheme, which is used for issuing the next round of processing instructions.

8. A wafer fab-level real-time scheduling system in semiconductor manufacturing based on deep reinforcement learning, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to collect wafer fab equipment status information, wafer batch information and processing process information through a production management system, extract equipment idle time, equipment maintenance plan time and equipment failure rate according to the equipment status information, extract batch waiting time, number of remaining batch processes and batch priority according to the wafer batch information, extract process processing time, process quality yield and process changeover time according to the processing process information, and construct the extracted information into a state space vector; The second unit is used to input the state space vector into a pre-trained deep reinforcement learning network model, wherein the deep reinforcement learning network model includes a value evaluation network and a policy network, wherein the value evaluation network is used to evaluate the value score of the current equipment and batch combination, and the policy network is used to generate a matching probability distribution between the batch and the equipment according to the value score; based on the matching probability distribution, a hierarchical reinforcement learning method is used to decompose the batch and equipment matching process into two subtasks, batch selection and equipment allocation, the batch selection subtask determines the priority sequence of the batches to be processed, and then the equipment allocation subtask matches the appropriate equipment for each batch to be processed, and generates multiple candidate matching schemes; Each candidate matching solution is evaluated by applying a preset multi-objective reward and penalty function, and the evaluation result of the multi-objective reward and penalty function is used as the selection weight of the Monte Carlo tree search algorithm. The optimal scheduling solution for batches and equipment is selected through multiple rounds of search iterations. The third unit is used to issue processing instructions to the corresponding equipment through the equipment control system according to the optimal scheduling plan, collect real-time processing data during the process processing, update the state space vector based on the real-time processing data, and re-input the updated state space vector into the deep reinforcement learning network model for real-time generation of a scheduling plan.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images