Q-time control method for semiconductor manufacturing based on deep reinforcement learning

Abstract

The present invention provides a semiconductor manufacturing Q-time control method based on deep reinforcement learning, which relates to the field of semiconductor technology, including obtaining process parameters, equipment status, waiting time between processes and batch information, inputting them into a deep reinforcement learning model based on a dual deep neural network, calculating the Q-time violation risk, and selecting scheduling actions; the model is trained using a comprehensive reward function including Q-time violation penalties, production efficiency rewards, equipment utilization rewards and batch balance rewards; finally, a multi-objective optimization decision module generates scheduling instructions based on real-time monitoring data and selected scheduling actions, comprehensively considering production capacity maximization and Q-time violation minimization, and dynamically adjusting the process schedule.

Description

Technical Field

[0001] The present invention relates to semiconductor technology, and in particular to a semiconductor manufacturing Q-time control method based on deep reinforcement learning. Background Art

[0002] Traditional scheduling methods, such as rule-based methods and mathematical programming-based methods, often have difficulty effectively handling the complexity and uncertainty in the semiconductor manufacturing process, such as equipment failures, process fluctuations, and order changes. Existing semiconductor manufacturing scheduling methods have the following defects and deficiencies:

[0003] Difficult to adapt to dynamic changes: Traditional scheduling methods are usually based on pre-set rules or plans, which make it difficult to quickly respond and adjust to real-time changes in the production process (such as equipment failures, emergency orders, etc.), resulting in reduced production efficiency and delivery delays.

[0004] Lack of effective control of Q-time: Q-time refers to the maximum allowed residence time of a workpiece in the production process. The existing scheduling method lacks effective control of Q-time, which easily leads to workpiece overtime, thus affecting product quality and yield.

[0005] Difficulty in achieving multi-objective optimization: Multiple objectives need to be considered in the semiconductor manufacturing process, such as maximizing production capacity, minimizing Q-time violations, maximizing equipment utilization, etc. Existing scheduling methods often find it difficult to optimize multiple objectives at the same time, resulting in a loss of focus and difficulty in obtaining a globally optimal scheduling solution. Summary of the invention

[0006] The embodiments of the present invention provide a semiconductor manufacturing Q-time control method based on deep reinforcement learning, which can solve the problems in the prior art.

[0007] According to a first aspect of the embodiments of the present invention,

[0008] Provides a semiconductor manufacturing Q-time control method based on deep reinforcement learning, including:

[0009] Obtaining process parameters, equipment status information, inter-process waiting time and batch information in a semiconductor manufacturing process, standardizing the process parameters, extracting timing features of the equipment status information, and inputting the standardized process parameters, timing features of the equipment status information, inter-process waiting time and batch information into a deep reinforcement learning model;

[0010] The deep reinforcement learning model constructs a state space based on the input information, and inputs the state space into a dual deep neural network, wherein the first neural network of the dual deep neural network is used to estimate the current state value, and the second neural network of the dual deep neural network is used to predict the target state value; the Q-time violation risk is calculated according to the difference between the current state value and the target state value, and a scheduling action is selected from a preset action space based on the Q-time violation risk; wherein the deep reinforcement learning model is trained using a comprehensive reward function, and the comprehensive reward function includes a Q-time violation penalty item, a production efficiency reward item, an equipment utilization reward item, and a batch balance reward item;

[0011] The semiconductor manufacturing process is monitored in real time according to the selected scheduling action, and the manufacturing process data obtained by the real-time monitoring is input into a multi-objective optimization decision module. The multi-objective optimization decision module generates scheduling instructions by comprehensively considering the maximization of production capacity and the minimization of Q-time violations, and dynamically adjusts the process arrangement in the semiconductor manufacturing process according to the scheduling instructions.

[0012] In an optional embodiment,

[0013] The deep reinforcement learning model constructs a state space based on the input information, and inputs the state space into a dual deep neural network, wherein the first neural network of the dual deep neural network is used to estimate the current state value, and the second neural network of the dual deep neural network is used to predict the target state value; the steps of calculating the Q-time violation risk according to the difference between the current state value and the target state value, and selecting a scheduling action from a preset action space based on the Q-time violation risk include:

[0014] A sliding time window method is used to extract parameter features from process parameters, wherein the parameter features include mean, standard deviation and trend slope, and a time series coding method is used to obtain equipment operation status features, wherein the equipment operation status features include state duration and switching frequency. The parameter features and the equipment operation status features are combined with the waiting time between processes and batch information to construct a multidimensional state space of the deep reinforcement learning model;

[0015] Inputting the multidimensional state space into a dual deep neural network of the deep reinforcement learning model, the dual deep neural network comprising a first neural network for estimating a current state value and a second neural network for predicting a target state value, wherein the first neural network adopts a residual structure and is connected through residuals and multiple layers of neurons, and the second neural network adopts a long short-term memory network structure enhanced by an attention mechanism and is fused with features of the attention layer and memory unit;

[0016] The deep reinforcement learning model calculates the difference between the target state value and the current state value to obtain a state value difference, and constructs a bidirectional risk calculation function based on the state value difference in combination with the Q-time upper limit constraint and the lower limit constraint to calculate the Q-time violation risk;

[0017] In the action space of batch movement, batch priority adjustment, equipment switching and batch merging preset in the deep reinforcement learning model, the action value function of the candidate action is calculated based on the current state value and the Q-time violation risk, the candidate action includes all executable specific action instances in batch movement, batch priority adjustment, equipment switching and batch merging, the action value function includes immediate reward and risk penalty, and a risk-adaptive exploration strategy is adopted to select scheduling actions from the action space.

[0018] In an optional embodiment,

[0019] In the action space of batch movement, batch priority adjustment, equipment switching and batch merging preset in the deep reinforcement learning model, the action value function of the candidate action is calculated based on the current state value and the Q-time violation risk, the candidate action includes all executable specific action instances in batch movement, batch priority adjustment, equipment switching and batch merging, the action value function includes immediate reward and risk penalty, and the step of selecting a scheduling action from the action space using a risk-adaptive exploration strategy includes:

[0020] The batch movement includes a batch identifier, a starting device, a target device, and a movement time, the batch priority adjustment includes a batch identifier and a batch priority level, the device switching includes a batch identifier and a target device, and the batch merging includes a first batch identifier and a second batch identifier;

[0021] Calculating an instant reward for the candidate action based on the current state value, wherein the instant reward is obtained by a weighted combination of the capacity improvement benefit, the degree of improvement in equipment utilization, and the degree of load balancing;

[0022] The risk penalty value is calculated based on the Q-time violation risk and the action risk impact, wherein the action risk impact includes: the risk impact of batch movement is calculated according to the path change amplitude, the risk impact of batch priority adjustment is calculated according to the batch priority change amount, the risk impact of equipment switching is calculated according to the equipment difference, and the risk impact of batch merging is calculated according to the batch size ratio;

[0023] Combining the immediate reward with the risk penalty value to construct an action value function, and calculating the value of a specific action instance in the candidate action based on the action value function;

[0024] An exploration rate that is adaptively adjusted with the risk is used to select actions, and the exploration rate increases with the increase of the Q-time violation risk. According to the action value function and the exploration rate, switching is performed between the specific action instance with the greatest value and the random action instance according to a preset probability to obtain the final scheduling action.

[0025] In an optional embodiment,

[0026] The deep reinforcement learning model is trained using a comprehensive reward function, wherein the comprehensive reward function includes a Q-time violation penalty item, a production efficiency reward item, an equipment utilization reward item, and a batch balance reward item, and the steps include:

[0027] The Q-time violation penalty is calculated based on the difference between the batch waiting time and the limit time. The production efficiency reward is calculated based on the number of batches processed per unit time. The equipment utilization reward is calculated based on the ratio of the actual equipment processing time to the total time. The batch balance reward is calculated based on the standard deviation of the number of work-in-progress in each process.

[0028] Constructing a loss function based on the comprehensive reward function, the loss function including a temporal difference error term and a regularization term, wherein the temporal difference error term is calculated by subtracting the current state action value output by the first neural network from the sum of the reward value calculated by the comprehensive reward function and the discounted value of the next state maximum action value predicted by the second neural network;

[0029] A priority experience replay mechanism is used to sample training samples, wherein the priority experience replay mechanism includes experience tuple storage, sample priority calculation and sampling probability calculation;

[0030] Performing gradient descent update on the parameters of the first neural network based on the loss function, and updating the parameters of the second neural network by soft update, wherein the parameter update of the first neural network uses gradient clipping technology to limit the gradient norm, and the parameter update of the second neural network controls the update speed by soft update coefficient;

[0031] The Q-time violation rate is calculated according to the Q-time violation risk, the average reward value fluctuation is calculated according to the comprehensive reward function, and the value estimation error is calculated according to the loss function. The training process is terminated when the Q-time violation rate, the average reward value fluctuation and the value estimation error are improved by less than a preset threshold for multiple consecutive rounds.

[0032] In an optional embodiment,

[0033] The priority experience replay mechanism is used to sample training samples. The priority experience replay mechanism includes the steps of experience tuple storage, sample priority calculation and sampling probability calculation, including:

[0034] Constructing an experience tuple, wherein the experience tuple includes a current state, an execution action, a reward value calculated by the comprehensive reward function, a next state after the execution action, and a temporal difference error term, wherein the current state is composed of a process parameter vector, an equipment state vector, an inter-process waiting time vector, and a batch information vector;

[0035] The experience tuple is stored in a playback buffer pool of fixed capacity according to a first-in-first-out principle, and the sample priority of the experience tuple is calculated based on the absolute value of the temporal difference error term, wherein the sample priority ensures that all experience tuples have a non-zero sampling probability by adding a bias value;

[0036] The sampling probability of each experience tuple is calculated according to the sample priority, the sampling probability is obtained by the ratio of a single sample priority to the sum of all sample priorities in the playback buffer pool, and an importance sampling index is introduced to adjust the sampling probability, and the importance sampling index is dynamically adjusted with the training process;

[0037] extracting training samples from the playback buffer pool based on the sampling probability, and calculating the importance weight of the training samples, wherein the importance weight is inversely proportional to the sampling probability;

[0038] The importance weight is multiplied by the loss function to obtain a weighted loss function, the parameters of the first neural network are updated based on the weighted loss function, and the sample priority is updated through the temporal difference error term to achieve dynamic optimization of the priority experience replay mechanism.

[0039] In an optional embodiment,

[0040] The semiconductor manufacturing process is monitored in real time according to the selected scheduling action, and the manufacturing process data obtained by the real-time monitoring is input into a multi-objective optimization decision module, wherein the multi-objective optimization decision module generates a scheduling instruction by comprehensively considering the maximization of production capacity and the minimization of Q-time violations, and the step of dynamically adjusting the process arrangement in the semiconductor manufacturing process according to the scheduling instruction includes:

[0041] Collecting equipment status data through equipment interfaces, the equipment status data including operation status, maintenance status, fault status and equipment available time window; collecting process parameter data through sensor networks, the process parameter data including temperature, pressure and gas flow; collecting batch tracking data of the current process, waiting time and processing time of the batch through the manufacturing execution system, as well as constraint data of the maximum allowable interval time between adjacent processes;

[0042] Inputting equipment status data, process parameter data, batch tracking data and constraint data into a multi-objective optimization decision module, and constructing a multi-objective optimization model including a capacity maximization objective function and a Q-time violation minimization objective function based on the multi-objective optimization decision module, wherein the capacity maximization objective function is obtained by weighting the ratio of the number of completed batches to the total number of batches and the equipment utilization rate, and the Q-time violation minimization objective function is obtained by weighting the ratio of the number of overtime processes to the total number of processes and the overtime duration;

[0043] Based on the multi-objective optimization model, the equipment capacity constraint, the process dependency constraint and the equipment maintenance time window constraint are combined to solve, and the dynamic scheduling priority of each process to be scheduled is calculated according to the capacity maximization objective function value and the Q-time violation minimization objective function value, and equipment resources are pre-allocated to the high-priority process based on the dynamic scheduling priority, and the specific scheduling time point is determined in combination with the equipment available time window and the process processing time, and the scheduling instruction is generated;

[0044] The scheduling instructions are issued to the corresponding processes and equipment through the manufacturing execution system, and the current capacity index and Q-time violation rate are calculated in real time. When the deviation between the capacity index and the Q-time violation rate and the expected target is less than a first threshold, the weight coefficients of the capacity maximization objective function and the Q-time violation minimization objective function are adjusted for local optimization. When the deviation is greater than a second threshold, the multi-objective optimization model is re-solved to generate a new scheduling instruction.

[0045] In an optional embodiment,

[0046] Based on the multi-objective optimization model, the steps of solving the problem in combination with the equipment capacity constraint, the process dependency constraint and the equipment maintenance time window constraint include:

[0047] Constructing an equipment availability matrix, wherein the rows of the equipment availability matrix represent equipment numbers, the columns represent discrete time points, and the available status of the equipment at different time points is represented by binary variables, and the available equipment set of each process is determined based on the equipment available status; generating equipment allocation variables according to the available equipment set, wherein the equipment allocation variables represent the corresponding relationship between the process and the equipment, forming an equipment allocation plan based on the equipment allocation variables, and establishing equipment capacity constraints to ensure that the equipment allocation plan at any time does not exceed the equipment availability;

[0048] Based on the equipment allocation variables, process dependency constraints are established, the completion time and start time of consecutive processes are associated, and the maximum allowable interval time constraint between adjacent processes is introduced to ensure that the equipment allocation plan meets the process dependency requirements;

[0049] Marking the equipment maintenance time window in the equipment availability matrix to form an equipment maintenance time window constraint, setting the equipment availability status corresponding to the maintenance period to unavailable, and canceling the equipment allocation plan that has a time intersection with the equipment maintenance time window according to the non-interruptible characteristic of the maintenance activity;

[0050] The equipment capacity constraint, process dependency constraint and equipment maintenance time window constraint are constructed into a unified constraint matrix, and a Lagrangian relaxation method is used to solve the feasible solution that satisfies the constraint conditions;

[0051] The process execution progress is monitored in real time. When a violation of the equipment capacity constraint, the process pre- and post-dependency constraint, or the equipment maintenance time window constraint is detected, for the violation of the equipment capacity constraint caused by equipment unavailability, the equipment allocation variables are regenerated in the available equipment set; for the violation of the process pre- and post-dependency constraint, the maximum allowed interval time constraint between adjacent processes is adjusted; for the violation of the equipment maintenance time window constraint, the equipment maintenance time window in the equipment availability matrix is ​​updated to ensure the execution feasibility of the feasible solution.

[0052] According to a second aspect of the embodiments of the present invention,

[0053] Provides semiconductor manufacturing Q-time control system based on deep reinforcement learning, including:

[0054] The first unit is used to obtain process parameters, equipment status information, inter-process waiting time and batch information in the semiconductor manufacturing process, standardize the process parameters, extract the timing characteristics of the equipment status information, and input the standardized process parameters, timing characteristics of the equipment status information, inter-process waiting time and batch information into the deep reinforcement learning model;

[0055] The second unit is used for the deep reinforcement learning model to construct a state space based on the input information, input the state space into a dual deep neural network, the first neural network of the dual deep neural network is used to estimate the current state value, and the second neural network of the dual deep neural network is used to predict the target state value; the Q-time violation risk is calculated according to the difference between the current state value and the target state value, and a scheduling action is selected from a preset action space based on the Q-time violation risk; wherein the deep reinforcement learning model is trained using a comprehensive reward function, and the comprehensive reward function includes a Q-time violation penalty item, a production efficiency reward item, an equipment utilization reward item, and a batch balance reward item;

[0056] The third unit is used to monitor the semiconductor manufacturing process in real time according to the selected scheduling action, and input the manufacturing process data obtained by real-time monitoring into a multi-objective optimization decision module. The multi-objective optimization decision module comprehensively considers the maximization of production capacity and the minimization of Q-time violations to generate scheduling instructions, and dynamically adjusts the process arrangement in the semiconductor manufacturing process according to the scheduling instructions.

[0057] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including: a processor;

[0058] a memory for storing processor-executable instructions;

[0059] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0060] A fourth aspect of the embodiments of the present invention is:

[0061] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0062] The present invention improves the Q-time control capability of the semiconductor manufacturing process: by predicting the Q-time violation risk through a deep reinforcement learning model and selecting scheduling actions accordingly, the Q-time violation risk can be effectively reduced and product quality and yield can be improved.

[0063] The present invention improves the production efficiency and equipment utilization of semiconductor manufacturing: the introduction of the comprehensive reward function enables the model to take into account multiple objectives such as Q-time control, production efficiency, equipment utilization and batch balance during the learning process, thereby improving production efficiency and equipment utilization while ensuring product quality.

[0064] The present invention realizes dynamic scheduling optimization of the semiconductor manufacturing process: the multi-objective optimization decision module dynamically adjusts the process schedule based on real-time monitoring data, comprehensively considers maximizing production capacity and minimizing Q-time violations, and realizes refined control and optimization of the manufacturing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a flow chart of a semiconductor manufacturing Q-time control method based on deep reinforcement learning according to an embodiment of the present invention;

[0066] Figure 2 Schematic diagram of the structure of a semiconductor manufacturing Q-time control system based on deep reinforcement learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0069] Figure 1 FIG. 5 is a flow chart of a semiconductor manufacturing Q-time control method based on deep reinforcement learning according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0070] S1. Obtain process parameters, equipment status information, inter-process waiting time and batch information in the semiconductor manufacturing process, standardize the process parameters, extract the timing characteristics of the equipment status information, and input the standardized process parameters, timing characteristics of the equipment status information, inter-process waiting time and batch information into the deep reinforcement learning model;

[0071] S2. The deep reinforcement learning model constructs a state space based on the input information, and inputs the state space into a dual deep neural network, wherein the first neural network of the dual deep neural network is used to estimate the current state value, and the second neural network of the dual deep neural network is used to predict the target state value; the Q-time violation risk is calculated according to the difference between the current state value and the target state value, and a scheduling action is selected from a preset action space based on the Q-time violation risk; wherein the deep reinforcement learning model is trained using a comprehensive reward function, and the comprehensive reward function includes a Q-time violation penalty item, a production efficiency reward item, an equipment utilization reward item, and a batch balance reward item;

[0072] S3. Monitor the semiconductor manufacturing process in real time according to the selected scheduling action, and input the manufacturing process data obtained by real-time monitoring into a multi-objective optimization decision module. The multi-objective optimization decision module comprehensively considers the maximization of production capacity and the minimization of Q-time violations to generate scheduling instructions, and dynamically adjusts the process arrangement in the semiconductor manufacturing process according to the scheduling instructions.

[0073] In an optional embodiment,

[0074] The deep reinforcement learning model constructs a state space based on the input information, and inputs the state space into a dual deep neural network, wherein the first neural network of the dual deep neural network is used to estimate the current state value, and the second neural network of the dual deep neural network is used to predict the target state value; the steps of calculating the Q-time violation risk according to the difference between the current state value and the target state value, and selecting a scheduling action from a preset action space based on the Q-time violation risk include:

[0075] A sliding time window method is used to extract parameter features from process parameters, wherein the parameter features include mean, standard deviation and trend slope, and a time series coding method is used to obtain equipment operation status features, wherein the equipment operation status features include state duration and switching frequency. The parameter features and the equipment operation status features are combined with the waiting time between processes and batch information to construct a multidimensional state space of the deep reinforcement learning model;

[0076] Inputting the multidimensional state space into a dual deep neural network of the deep reinforcement learning model, the dual deep neural network comprising a first neural network for estimating a current state value and a second neural network for predicting a target state value, wherein the first neural network adopts a residual structure and is connected through residuals and multiple layers of neurons, and the second neural network adopts a long short-term memory network structure enhanced by an attention mechanism and is fused with features of the attention layer and memory unit;

[0077] The deep reinforcement learning model calculates the difference between the target state value and the current state value to obtain a state value difference, and constructs a bidirectional risk calculation function based on the state value difference in combination with the Q-time upper limit constraint and the lower limit constraint to calculate the Q-time violation risk;

[0078] In the action space of batch movement, batch priority adjustment, equipment switching and batch merging preset in the deep reinforcement learning model, the action value function of the candidate action is calculated based on the current state value and the Q-time violation risk, the candidate action includes all executable specific action instances in batch movement, batch priority adjustment, equipment switching and batch merging, the action value function includes immediate reward and risk penalty, and a risk-adaptive exploration strategy is adopted to select scheduling actions from the action space.

[0079] For example, in order to improve the efficiency and stability of production scheduling in complex production environments, a dynamic scheduling method based on deep reinforcement learning is proposed. The core of this method is to build a deep reinforcement learning model that can evaluate the risk of scheduling actions and select the optimal scheduling action based on the risk.

[0080] First, it is necessary to construct the state space of the deep reinforcement learning model. This state space is a multidimensional vector that contains key information that affects scheduling decisions. Specifically, the sliding time window method is used to extract parameter features from the process parameters. These features include the parameter mean, standard deviation, and trend slope within each time window. For example, the mean, standard deviation, and rising / falling trend of the temperature in the past hour can be extracted. At the same time, the operating state characteristics of the equipment are recorded, including the duration of each state and the frequency of state transitions. For example, the duration of each of the three states of "operation", "standby", and "maintenance" of the equipment and the frequency of transitions between states are recorded. In addition, the waiting time between processes and batch information (such as batch ID, arrival time, process requirements, etc.) need to be added to the state space. Finally, these features are combined into a multidimensional state space of the deep reinforcement learning model.

[0081] Next, a dual deep neural network for the deep reinforcement learning model is constructed. The network consists of two subnetworks: one for estimating the current state value, and one for predicting the target state value. The network for estimating the current state value uses a residual structure, which improves the learning ability of the model through residual connections and multi-layer neurons. The network for predicting the target state value uses a long short-term memory network structure enhanced by an attention mechanism, which better captures the temporal dependency and key features of the state through the feature fusion of the attention layer and the memory unit. The constructed state space is input into this dual deep neural network.

[0082] Then, calculate the Q-time violation risk. First, calculate the difference between the target state value and the current state value to obtain the state value difference. Then, pre-set the upper and lower constraints of Q-time, for example, stipulate that Q-time must be between 10-20 hours. Based on the state value difference, Q-time upper and lower constraints, construct a two-way risk calculation function. If the state value difference causes the predicted Q-time to exceed the preset range, the risk is higher; otherwise, the risk is lower. For example, if the predicted Q-time is 25 hours, which exceeds the upper limit of 20 hours, a higher risk value is calculated.

[0083] Finally, select the scheduling action. The preset action space contains four types of actions: batch movement, batch priority adjustment, equipment switching, and batch merging. For the current state, list all executable specific action instances, for example, move batch 1 from device A to device B, increase the priority of batch 2 to the highest, switch device C to produce product X, merge batch 3 and batch 4, etc. For each candidate action, calculate its action value function, which contains immediate rewards and risk penalties. The immediate reward can be an increase in the speed of completing the batch, and the risk penalty is related to the risk of Q-time violation. Adopting a risk-adaptive exploration strategy, it tends to choose low-risk actions in high-risk situations, and more actively explores potential better actions in low-risk situations. Finally, the action with the largest action value function is selected as the scheduling action.

[0084] The present invention can find a better scheduling strategy through learning and optimization of the deep reinforcement learning model, thereby reducing production time and improving production efficiency; by considering the risk of Q-time violation, it can avoid violations of time constraints, thereby improving the stability of production scheduling. For example, by giving priority to batches close to the deadline, delayed delivery can be avoided and the stable execution of the production plan can be ensured; the method can dynamically adjust the scheduling strategy according to the real-time status, so as to better adapt to changes in the production environment. For example, when a certain device fails, the batch allocation and priority can be adjusted immediately to minimize the impact of production interruptions.

[0085] In an optional embodiment,

[0086] In the action space of batch movement, batch priority adjustment, equipment switching and batch merging preset in the deep reinforcement learning model, the action value function of the candidate action is calculated based on the current state value and the Q-time violation risk, the candidate action includes all executable specific action instances in batch movement, batch priority adjustment, equipment switching and batch merging, the action value function includes immediate reward and risk penalty, and the step of selecting a scheduling action from the action space using a risk-adaptive exploration strategy includes:

[0087] The batch movement includes a batch identifier, a starting device, a target device, and a movement time, the batch priority adjustment includes a batch identifier and a batch priority level, the device switching includes a batch identifier and a target device, and the batch merging includes a first batch identifier and a second batch identifier;

[0088] Calculating an instant reward for the candidate action based on the current state value, wherein the instant reward is obtained by a weighted combination of the capacity improvement benefit, the degree of improvement in equipment utilization, and the degree of load balancing;

[0089] The risk penalty value is calculated based on the Q-time violation risk and the action risk impact, wherein the action risk impact includes: the risk impact of batch movement is calculated according to the path change amplitude, the risk impact of batch priority adjustment is calculated according to the batch priority change amount, the risk impact of equipment switching is calculated according to the equipment difference, and the risk impact of batch merging is calculated according to the batch size ratio;

[0090] Combining the immediate reward with the risk penalty value to construct an action value function, and calculating the value of a specific action instance in the candidate action based on the action value function;

[0091] An exploration rate that is adaptively adjusted with the risk is used to select actions, and the exploration rate increases with the increase of the Q-time violation risk. According to the action value function and the exploration rate, switching is performed between the specific action instance with the greatest value and the random action instance according to a preset probability to obtain the final scheduling action.

[0092] Exemplarily, first, define the action space of the deep reinforcement learning model. The action space contains four types of actions: batch movement, batch priority adjustment, device switching, and batch merging. Batch movement is defined as moving a specific batch from one device to another, and the batch identifier, starting device, target device, and moving time must be specified. For example, move batch A from device 1 to device 2, and the moving time is 1 hour. Batch priority adjustment is defined as changing the priority level of a specific batch. For example, increase the priority of batch B from level 3 to level 1. Device switching refers to switching the device to process other types of batches. For example, switch device 3 from processing type X batches to processing type Y batches. Batch merging refers to merging two batches into one batch for processing. For example, merge batch C and batch D.

[0093] Next, the immediate reward for each candidate action is calculated. For the current production state, each executable action will generate an immediate reward. The reward is a weighted combination of three factors: capacity improvement benefits, equipment utilization improvement, and load balancing. For example, if an action can increase capacity by 10%, equipment utilization by 5%, and improve load balancing by 2%, the immediate reward can be calculated based on the preset weights, such as 0.5, 0.3, and 0.2: 0.5*10%+0.3*5%+0.2*2%=6.9%.

[0094] Then, the risk penalty value of each action is evaluated. Each action may increase the risk of Q-time violation. The risk penalty value is calculated by the Q-time violation risk and the action risk impact. The risk impact of different types of actions is calculated differently. The risk impact of batch movement is calculated based on the path change. For example, an increase in the moving distance may lead to a greater risk. Assume that batch E was originally planned to be directly transported to device 5 after being processed by device 4, but now it is transported to device 6 first and then to device 5. Due to the increase in the path, the risk impact increases. The risk impact of batch priority adjustment is calculated based on the batch priority change. For example, raising a low-priority batch to a high-priority batch may cause delays for other high-priority batches. Assume that batch F had an original priority of 5 and was adjusted to 1, with a change of 4, and a large risk impact. The risk impact of equipment switching is calculated based on the equipment difference. For example, switching to an unfamiliar equipment type may lead to reduced production efficiency. Assume that equipment 7 and equipment 8 process different types of batches, the equipment difference is large, and the switching risk is high. The risk impact of batch merging is calculated based on the batch size ratio. For example, merging two batches with very different sizes may cause confusion in production plans. Assuming that the batch G size is 100, the batch H size is 10, and the size ratio is 10, the risk impact is relatively large. The higher the Q-time violation risk, the greater the risk penalty value of the action.

[0095] Construct an action value function. Combine the immediate reward and risk penalty values ​​to construct an action value function. For example, if the immediate reward of an action is 6.9% and the risk penalty value is 2%, the value of the action is 4.9%.

[0096] Adopt risk-adaptive exploration strategy to select scheduling actions. Dynamically adjust the exploration rate according to the current Q-time violation risk. The higher the risk, the higher the exploration rate, which encourages the model to try more different actions and avoid falling into local optimality. According to the action value function and the exploration rate, choose between the action with the greatest value and the random action. For example, if the exploration rate is 10%, there is a 90% probability of selecting the action with the greatest value and a 10% probability of selecting the random action. If the current Q-time violation risk is high, for example, exceeding the preset threshold of 80%, the exploration rate will be increased accordingly, for example, to 30%, to encourage the model to explore more different scheduling strategies.

[0097] Embodiments of the present invention also include:

[0098] Construct state space: obtain batch information of each process with Q-Time constraints in the current production system, including: number of batches to be released, batch urgency, Q-Time remaining time margin, process equipment occupancy and expected completion time of work-in-progress; obtain process equipment status information, including: equipment availability status, current processing batch type, remaining processing time and equipment maintenance plan time window; obtain waiting area information, including: quantity distribution, priority distribution and waiting time distribution of each type of batch, etc. Combine the above information to construct a multidimensional state space.

[0099] Establish a dual deep neural network: The dual deep neural network includes an evaluation network and a prediction network. The evaluation network uses a multi-layer convolution structure to extract time series features, and avoids the gradient vanishing problem through residual connections. Finally, the fully connected layer outputs the current state evaluation value; the prediction network uses a long short-term memory network structure with an attention mechanism. By focusing on key state features, it predicts the evolution trend of the future state of the system and outputs the target state prediction value.

[0100] Define the action space: Set N different batch types with Q-Time constraints, and construct an N+1-dimensional action space accordingly. Action 0 means not releasing any batch, and action i (i ranges from 1 to N) means releasing a batch of type i. Perform feasibility verification on each candidate action, including: checking the availability of the corresponding type of batch, verifying the equipment processing capacity constraints, considering the batch priority requirements, etc., and eliminating actions that do not meet the constraints.

[0101] Calculate risk and select actions: Calculate system risk indicators based on the current WIP status, key equipment load forecast, and Q-Time violation probability. Design a comprehensive reward function that includes capacity achievement, equipment utilization, Q-Time compliance, and batch cycle time. Adopt a risk-adaptive exploration strategy, that is, increase the exploration rate in high-risk conditions to find a better solution, and reduce the exploration rate in low-risk conditions to maintain stability. Select the final action to be executed based on the current risk and reward function calculation results.

[0102] Execute decisions and provide feedback updates: Execute the selected batch release actions, including: confirming batch release conditions, adjusting equipment scheduling plans, updating batch status information, etc. Collect actual execution results, update state space information, calculate actual reward values, and use this information as feedback for optimizing and updating neural network parameters.

[0103] Optimize training strategies: Establish an experience replay pool to store historical decision samples, and regularly perform batch learning updates to ensure sample diversity and avoid overfitting. Dynamically adjust system parameters, including: adaptive adjustment of learning rate, reward weight balance, dynamic change of exploration rate, etc., to continuously optimize the decision model.

[0104] The present invention optimizes batch scheduling, maximizes production capacity, improves revenue and equipment utilization, and thus improves overall production efficiency; by considering the Q-time violation risk and action risk impact, high-risk actions are avoided and the Q-time violation risk is reduced; a risk-adaptive exploration strategy is adopted to enable the model to dynamically adjust the exploration degree according to actual conditions, enhance the robustness of the scheduling strategy, and adapt to complex production environments.

[0105] In an optional embodiment,

[0106] The deep reinforcement learning model is trained using a comprehensive reward function, wherein the comprehensive reward function includes a Q-time violation penalty item, a production efficiency reward item, an equipment utilization reward item, and a batch balance reward item, and the steps include:

[0107] The Q-time violation penalty is calculated based on the difference between the batch waiting time and the limit time. The production efficiency reward is calculated based on the number of batches processed per unit time. The equipment utilization reward is calculated based on the ratio of the actual equipment processing time to the total time. The batch balance reward is calculated based on the standard deviation of the number of work-in-progress in each process.

[0108] Constructing a loss function based on the comprehensive reward function, the loss function including a temporal difference error term and a regularization term, wherein the temporal difference error term is calculated by subtracting the current state action value output by the first neural network from the sum of the reward value calculated by the comprehensive reward function and the discounted value of the next state maximum action value predicted by the second neural network;

[0109] A priority experience replay mechanism is used to sample training samples, wherein the priority experience replay mechanism includes experience tuple storage, sample priority calculation and sampling probability calculation;

[0110] Performing gradient descent update on the parameters of the first neural network based on the loss function, and updating the parameters of the second neural network by soft update, wherein the parameter update of the first neural network uses gradient clipping technology to limit the gradient norm, and the parameter update of the second neural network controls the update speed by soft update coefficient;

[0111] The Q-time violation rate is calculated according to the Q-time violation risk, the average reward value fluctuation is calculated according to the comprehensive reward function, and the value estimation error is calculated according to the loss function. The training process is terminated when the Q-time violation rate, the average reward value fluctuation and the value estimation error are improved by less than a preset threshold for multiple consecutive rounds.

[0112] Exemplarily, first, a comprehensive reward function is constructed. The function consists of four parts: Q-time violation penalty item, production efficiency reward item, equipment utilization reward item and batch balance reward item. The Q-time violation penalty item is used to punish batches that exceed the specified time, and its calculation method is the degree to which the batch waiting time exceeds the limit time. For example, if the limit time is 10 hours and a batch waits for 12 hours, the violation time is 2 hours, and the penalty value can be set according to the length of the violation time, such as 1 point for every hour exceeding. The production efficiency reward item is used to encourage the model to improve production efficiency, and its calculation method is the number of batches processed per unit time. For example, if 10 batches are processed in one hour, the reward value is 10. The equipment utilization reward item is used to encourage the model to make full use of the equipment, and its calculation method is the ratio of the actual processing time of the equipment to the total time. For example, if a piece of equipment works for 50 minutes in one hour, the equipment utilization rate is 50 / 60. The batch balance reward item is used to encourage the model to balance the number of work-in-progress in each process, and its calculation method is the standard deviation of the number of work-in-progress in each process. The smaller the standard deviation, the more balanced the WIP of each process is, and the higher the reward value is. For example, if the WIP of the three processes is 10, 12, and 14, the standard deviation is small and the reward value is high; if the WIP of the three processes is 5, 10, and 20, the standard deviation is large and the reward value is low.

[0113] Next, a loss function is constructed based on the comprehensive reward function. The loss function includes a temporal difference error term and a regularization term. The temporal difference error term is calculated by adding the reward value calculated by the comprehensive reward function to the discounted value of the maximum action value of the next state predicted by the second neural network, minus the current state action value output by the first neural network. The regularization term is used to prevent the model from overfitting.

[0114] Then, the priority experience replay mechanism is used to sample training samples. This mechanism includes experience tuple storage, sample priority calculation, and sampling probability calculation. Experience tuple storage is used to store experience data generated by the interaction between the model and the environment, including information such as the current state, the action performed, the reward obtained, and the next state. Sample priority calculation is used to calculate the importance of each experience tuple, and samples with high priority are more likely to be sampled. For example, experience tuples with high reward values ​​have higher priorities. Sampling probability calculation is used to calculate the probability of each experience tuple being sampled based on the sample priority.

[0115] Next, the parameters of the first neural network are updated by gradient descent based on the loss function, and the parameters of the second neural network are updated by soft update. The parameter update of the first neural network uses gradient clipping technology to limit the gradient norm to prevent gradient explosion. The parameter update of the second neural network controls the update speed through the soft update coefficient, so that the parameters of the second neural network gradually approach the parameters of the first neural network.

[0116] Finally, the Q-time violation rate is calculated based on the Q-time violation risk, the average reward value fluctuation is calculated based on the comprehensive reward function, and the value estimation error is calculated based on the loss function. The training process is terminated when the Q-time violation rate, average reward value fluctuation, and value estimation error have improved by less than the preset threshold for multiple consecutive rounds. For example, if the Q-time violation rate decreases by less than 1% for three consecutive rounds, the average reward value fluctuation is less than 2%, and the value estimation error is less than 3%, the model is considered to have converged and the training process is terminated.

[0117] Embodiments of the present invention also include:

[0118] The comprehensive reward function design method revolves around four core goals: minimizing the number of batches that violate the Q-time constraint, maximizing the number of products that successfully complete the Q-time constraint process, maximizing the production target completion rate, and maximizing equipment utilization. Specifically, it includes the following steps:

[0119] Steps for evaluating the execution effect of the Q-time constraint: Count the total number of batches that violate the Q-time constraint in the current manufacturing process as the primary goal to be minimized; at the same time, count the number of batches that successfully meet the Q-time constraint and complete production, and use it as the incentive goal to be maximized; set the penalty intensity based on the number of illegal batches, and set the reward intensity based on the number of successful batches; through a combination of rewards and punishments, guide the system to continuously improve the execution effect of the Q-time constraint.

[0120] Production target achievement assessment steps: obtain target output information set in the current production plan; count the actual number of products completed; calculate the achievement ratio of production targets; determine the assessment criteria for target achievement rate based on the production priorities corresponding to different product types; maximize production capacity efficiency by continuously improving the production target achievement rate.

[0121] Equipment utilization efficiency evaluation steps: record the actual working status of production equipment; calculate the actual processing time ratio of each equipment; consider the balance of equipment combination use; include necessary downtime such as equipment maintenance into the evaluation system; maximize the utilization of production line capacity by improving equipment utilization level.

[0122] Comprehensive evaluation steps: Combine the results of the above three evaluation dimensions; determine the importance of each dimension in the overall evaluation, among which Q-time constraint-related indicators have the highest priority, followed by production target achievement, and then equipment utilization efficiency; dynamically adjust the evaluation criteria and weight distribution of each dimension according to the actual production operation situation; guide subsequent production scheduling decisions through comprehensive evaluation results.

[0123] Through the guidance of the comprehensive reward function, the model of the present invention can learn the optimal production scheduling strategy, thereby increasing the number of batches completed in unit time; by punishing Q-time violations, the model can learn strategies to avoid violations, thereby reducing the Q-time violation rate; by rewarding equipment utilization, the model can learn strategies to make full use of the equipment, thereby improving equipment utilization.

[0124] In an optional embodiment,

[0125] The priority experience replay mechanism is used to sample training samples. The priority experience replay mechanism includes the steps of experience tuple storage, sample priority calculation and sampling probability calculation, including:

[0126] Constructing an experience tuple, wherein the experience tuple includes a current state, an execution action, a reward value calculated by the comprehensive reward function, a next state after the execution action, and a temporal difference error term, wherein the current state is composed of a process parameter vector, an equipment state vector, an inter-process waiting time vector, and a batch information vector;

[0127] The experience tuple is stored in a playback buffer pool of fixed capacity according to a first-in-first-out principle, and the sample priority of the experience tuple is calculated based on the absolute value of the temporal difference error term, wherein the sample priority ensures that all experience tuples have a non-zero sampling probability by adding a bias value;

[0128] The sampling probability of each experience tuple is calculated according to the sample priority, the sampling probability is obtained by the ratio of a single sample priority to the sum of all sample priorities in the playback buffer pool, and an importance sampling index is introduced to adjust the sampling probability, and the importance sampling index is dynamically adjusted with the training process;

[0129] extracting training samples from the playback buffer pool based on the sampling probability, and calculating the importance weight of the training samples, wherein the importance weight is inversely proportional to the sampling probability;

[0130] The importance weight is multiplied by the loss function to obtain a weighted loss function, the parameters of the first neural network are updated based on the weighted loss function, and the sample priority is updated through the temporal difference error term to achieve dynamic optimization of the priority experience replay mechanism.

[0131] Exemplarily, first, construct an experience tuple. The experience tuple is used to record the experience of the agent's interaction with the environment, including the current state, the action performed, the reward obtained, the next state after the action is performed, and the time difference error. Among them, the current state is composed of the process parameter vector, the equipment state vector, the waiting time vector between processes, and the batch information vector. For example, an experience tuple can be expressed as: current state (process temperature: 200°C, equipment A is running, waiting time for process 1: 10 minutes, batch 1), action performed (starting equipment B), reward value (+10), next state (process temperature: 220°C, equipment A and B are running, waiting time for process 2: 5 minutes, batch 1), time difference error (0.8).

[0132] Next, the experience tuple is stored in the replay buffer pool. The replay buffer pool adopts the first-in-first-out principle, that is, the newly added experience tuple will replace the earliest added experience tuple. The capacity of the replay buffer pool is fixed, for example, it is set to 10000. When the buffer pool is full, the new experience tuple will replace the oldest experience tuple. Then, the sample priority of the experience tuple is calculated based on the absolute value of the temporal difference error. In order to avoid the sampling probability of some experience tuples being zero, a small bias value, such as 0.01, is added when calculating the sample priority. Assuming that the temporal difference error of an experience tuple is 0.8, its sample priority is 0.8+0.01=0.81.

[0133] After that, the sampling probability of each experience tuple is calculated according to the sample priority. The sampling probability is obtained by the ratio of a single sample priority to the sum of all sample priorities in the replay buffer pool. In order to balance the exploration and utilization of experience replay, the importance sampling index is introduced to adjust the sampling probability. The importance sampling index is dynamically adjusted during the training process. For example, the initial value is 0.6 and gradually decays to 0.1.

[0134] Subsequently, training samples are extracted from the playback buffer pool based on the sampling probability. For example, a roulette wheel selection algorithm can be used to extract samples based on the sampling probability. Then, the importance weights of the training samples are calculated. The importance weights are inversely proportional to the sampling probability and are used to correct for the deviations caused by non-uniform sampling. For example, if the sampling probability of an experience tuple is 0.002, its importance weight is 1 / 0.002=500.

[0135] Finally, the importance weight is multiplied by the loss function to obtain the weighted loss function, and the parameters of the deep reinforcement learning network are updated based on the weighted loss function. At the same time, the sample priority is updated using the new temporal difference error to achieve dynamic optimization of the priority experience replay mechanism. For example, the gradient descent algorithm is used to update the network parameters.

[0136] The present invention accelerates the learning speed of the intelligent agent by preferentially replaying experiences that are more valuable for learning, so that it can converge to the optimal strategy more quickly; the importance sampling mechanism effectively reduces the deviation caused by non-uniform sampling, improves the stability of the training process, and avoids oscillation and divergence in the learning process; it can help the intelligent agent learn better decision-making strategies and achieve higher production efficiency and lower resource consumption in complex production scheduling scenarios.

[0137] In an optional embodiment,

[0138] The semiconductor manufacturing process is monitored in real time according to the selected scheduling action, and the manufacturing process data obtained by the real-time monitoring is input into a multi-objective optimization decision module, wherein the multi-objective optimization decision module generates a scheduling instruction by comprehensively considering the maximization of production capacity and the minimization of Q-time violations, and the step of dynamically adjusting the process arrangement in the semiconductor manufacturing process according to the scheduling instruction includes:

[0139] Collecting equipment status data through equipment interfaces, the equipment status data including operation status, maintenance status, fault status and equipment available time window; collecting process parameter data through sensor networks, the process parameter data including temperature, pressure and gas flow; collecting batch tracking data of the current process, waiting time and processing time of the batch through the manufacturing execution system, as well as constraint data of the maximum allowable interval time between adjacent processes;

[0140] Inputting equipment status data, process parameter data, batch tracking data and constraint data into a multi-objective optimization decision module, and constructing a multi-objective optimization model including a capacity maximization objective function and a Q-time violation minimization objective function based on the multi-objective optimization decision module, wherein the capacity maximization objective function is obtained by weighting the ratio of the number of completed batches to the total number of batches and the equipment utilization rate, and the Q-time violation minimization objective function is obtained by weighting the ratio of the number of overtime processes to the total number of processes and the overtime duration;

[0141] Based on the multi-objective optimization model, the equipment capacity constraint, the process dependency constraint and the equipment maintenance time window constraint are combined to solve, and the dynamic scheduling priority of each process to be scheduled is calculated according to the capacity maximization objective function value and the Q-time violation minimization objective function value, and equipment resources are pre-allocated to the high-priority process based on the dynamic scheduling priority, and the specific scheduling time point is determined in combination with the equipment available time window and the process processing time, and the scheduling instruction is generated;

[0142] The scheduling instructions are issued to the corresponding processes and equipment through the manufacturing execution system, and the current capacity index and Q-time violation rate are calculated in real time. When the deviation between the capacity index and the Q-time violation rate and the expected target is less than a first threshold, the weight coefficients of the capacity maximization objective function and the Q-time violation minimization objective function are adjusted for local optimization. When the deviation is greater than a second threshold, the multi-objective optimization model is re-solved to generate a new scheduling instruction.

[0143] Exemplarily, first, a real-time data acquisition system is built. The system obtains real-time data of the manufacturing process through various channels. Through the equipment interface, the operating status, maintenance status, fault status and time window of the equipment are collected. For example, if the status of a certain etching machine is displayed as "running", it is expected to enter the maintenance state after 1 hour, and the maintenance time is 30 minutes. Then the etching machine will be available in the next hour and will be available again after 1 hour and 30 minutes. Through the sensor network, process parameter data such as temperature, pressure, gas flow rate, etc. are collected in real time. For example, in the chemical vapor deposition process, the temperature in the reaction chamber is 700°C, the pressure is 100Pa, and the silane gas flow rate is 5sccm. Through the manufacturing execution system (MES), batch tracking data is collected, including the current process of the batch, waiting time, processing time, etc. For example, batch A is currently in the photolithography process, has been waiting for 2 hours, and the estimated processing time of the photolithography process is 1 hour. In addition, it is also necessary to obtain the maximum allowable interval time constraint data between adjacent processes from the MES. For example, the maximum allowable interval time between the photolithography process and the etching process is 4 hours.

[0144] Next, the collected data is input into the multi-objective optimization decision module. The core of this module is a multi-objective optimization model, which contains two objective functions: capacity maximization and Q-time violation minimization. The capacity maximization objective function is obtained by weighting the ratio of the number of completed batches to the total number of batches and the equipment utilization rate. For example, assuming that 100 batches are planned to be produced, 80 batches have been completed, and the equipment utilization rate is 90%, then the capacity index is (80 / 100)*0.7+0.9*0.3=0.83. The Q-time violation minimization objective function is obtained by weighting the ratio of the number of overtime processes to the total number of processes and the overtime duration. For example, assuming that the total number of processes is 500, 10 of which are overtime, and the total overtime duration is 20 hours, then the Q-time violation index is (10 / 500)*0.8+(20 / 500)*0.2=0.024.

[0145] Then, based on the multi-objective optimization model, the solution is combined with the equipment capacity constraint, the process dependency constraint and the equipment maintenance time window constraint. For example, a certain lithography machine can only process one batch at the same time, the etching process must be performed after the lithography process is completed, and the equipment maintenance time window needs to be considered. By solving the multi-objective optimization model, the dynamic scheduling priority of each process to be scheduled can be obtained. The dynamic scheduling priority of each process to be scheduled is calculated based on the objective function value of maximizing production capacity and the objective function value of minimizing Q-time violation. For example, if a process has a large impact on production capacity improvement and a low Q-time risk, the process will be given a higher priority.

[0146] Subsequently, equipment resources are pre-allocated to high-priority processes based on dynamic scheduling priorities, and the specific scheduling time point is determined in combination with the available time window of the equipment and the processing time of the process, and scheduling instructions are generated. For example, high-priority batch B is allocated to available lithography machines, and based on the available time window of the lithography machine and the lithography process processing time of batch B, the start time of the lithography process of batch B is determined to be 1 hour after the current time. Finally, the scheduling instructions are sent to the corresponding processes and equipment through MES.

[0147] Finally, the current capacity index and Q-time violation rate are calculated in real time and adjusted dynamically. When the deviation between the capacity index and Q-time violation rate and the expected target is less than the first threshold (e.g. 5%), local optimization is performed by adjusting the weight coefficients of the capacity maximization objective function and the Q-time violation minimization objective function. For example, if the current capacity index meets the requirements but the Q-time violation rate is high, the weight coefficient of the Q-time violation minimization objective function is increased. When the deviation is greater than the second threshold (e.g. 10%), the multi-objective optimization model is re-solved to generate new scheduling instructions to adjust the production plan to a greater extent.

[0148] The present invention can maximize the use of equipment resources and reduce batch waiting time through real-time monitoring and dynamic scheduling, thereby improving production efficiency; by optimizing the scheduling strategy, it can effectively control the completion time of the process, reduce the Q-time violation rate, and improve product quality and yield; the system can be dynamically adjusted according to real-time data and target deviation, thereby adapting to changes in the production environment and improving the robustness and stability of the system.

[0149] In an optional embodiment,

[0150] Based on the multi-objective optimization model, the steps of solving the problem in combination with the equipment capacity constraint, the process dependency constraint and the equipment maintenance time window constraint include:

[0151] Constructing an equipment availability matrix, wherein the rows of the equipment availability matrix represent equipment numbers, the columns represent discrete time points, and the available status of the equipment at different time points is represented by binary variables, and the available equipment set of each process is determined based on the equipment available status; generating equipment allocation variables according to the available equipment set, wherein the equipment allocation variables represent the corresponding relationship between the process and the equipment, forming an equipment allocation plan based on the equipment allocation variables, and establishing equipment capacity constraints to ensure that the equipment allocation plan at any time does not exceed the equipment availability;

[0152] Based on the equipment allocation variables, process dependency constraints are established, the completion time and start time of consecutive processes are associated, and the maximum allowable interval time constraint between adjacent processes is introduced to ensure that the equipment allocation plan meets the process dependency requirements;

[0153] Marking the equipment maintenance time window in the equipment availability matrix to form an equipment maintenance time window constraint, setting the equipment availability status corresponding to the maintenance period to unavailable, and canceling the equipment allocation plan that has a time intersection with the equipment maintenance time window according to the non-interruptible characteristic of the maintenance activity;

[0154] The equipment capacity constraint, process dependency constraint and equipment maintenance time window constraint are constructed into a unified constraint matrix, and a Lagrangian relaxation method is used to solve the feasible solution that satisfies the constraint conditions;

[0155] The process execution progress is monitored in real time. When a violation of the equipment capacity constraint, the process pre- and post-dependency constraint, or the equipment maintenance time window constraint is detected, for the violation of the equipment capacity constraint caused by equipment unavailability, the equipment allocation variables are regenerated in the available equipment set; for the violation of the process pre- and post-dependency constraint, the maximum allowed interval time constraint between adjacent processes is adjusted; for the violation of the equipment maintenance time window constraint, the equipment maintenance time window in the equipment availability matrix is ​​updated to ensure the execution feasibility of the feasible solution.

[0156] Exemplarily, in order to effectively solve the impact of equipment capacity, process dependency and equipment maintenance on production scheduling in multi-objective optimization problems, a multi-objective optimization method based on equipment availability matrix is ​​proposed.

[0157] First, construct an equipment availability matrix. The rows of the matrix represent different equipment, and the columns represent discrete time points, such as hours. The elements in the matrix are binary variables, 1 means that the equipment is available at that time point, and 0 means it is unavailable. For example, assuming there are 3 devices and the planning cycle is 24 hours, construct a matrix with 3 rows and 24 columns. In the initial state, all elements are 1, indicating that all equipment is available at all time points. If it is known that equipment 1 is shut down for maintenance from the 5th to the 8th hour, set the elements in the first row, 5th, 6th, 7th, and 8th columns of the matrix to 0. By analyzing the equipment availability matrix, the set of equipment that can be used for each process at each time point can be determined. For example, process A needs to use equipment 1 or equipment 2. At the 5th hour, since equipment 1 is unavailable, the set of available equipment for process A is {equipment 2}.

[0158] Next, generate equipment allocation variables based on the available equipment set. Equipment allocation variables are used to represent the correspondence between processes and equipment. Based on equipment allocation variables, different equipment allocation schemes can be formed. Then, establish equipment capacity constraints. This constraint ensures that at any point in time, the number of processes assigned to a device does not exceed the capacity of the device, that is, a device can only process one process at a time. For example, at the 5th hour, the capacity of device 2 is 1, then at most one process can be assigned to device 2.

[0159] Then, establish the process before and after dependency constraints. This constraint links the completion time and start time of consecutive processes. For example, process B can only start after process A is completed. Introducing the maximum allowable interval time constraint between adjacent processes can avoid long waiting time between processes. For example, after process A is completed, process B must start within 2 hours. Through these constraints, ensure that the generated equipment allocation plan meets the requirements of process before and after dependency.

[0160] Next, mark the equipment maintenance time window in the equipment availability matrix to form the equipment maintenance time window constraint. Set the corresponding equipment availability status during the maintenance period to unavailable. For example, if equipment 3 is maintained from the 10th hour to the 12th hour, the elements in the 10th, 11th, and 12th columns of the third row of the matrix are set to 0. Due to the non-interruptible nature of maintenance activities, cancel the equipment allocation plan that has a time intersection with the equipment maintenance time window. For example, if it was originally planned to allocate process C to equipment 3 at the 11th hour, since equipment 3 is undergoing maintenance at this time, the allocation plan is canceled.

[0161] The equipment capacity constraint, process dependency constraint and equipment maintenance time window constraint are constructed into a unified constraint matrix. The Lagrangian relaxation method is used to solve the problem and obtain a feasible solution that satisfies the constraints. The Lagrangian relaxation method is an optimization method that converts constraints into part of the objective function and can effectively handle complex constraints.

[0162] Finally, the progress of process execution is monitored in real time. When violations of equipment capacity constraints, process dependencies, or equipment maintenance time window constraints are detected, adjustments need to be made. For example, if the equipment capacity constraint is violated due to equipment failure, it is necessary to regenerate the equipment allocation variables in the available equipment set and reallocate the affected processes to other available equipment. If the process dependencies are violated, such as delayed completion of process A, which causes process B to be unable to start on time, it is necessary to adjust the maximum allowable interval time constraint between adjacent processes, or adjust the schedule of other processes. If the equipment maintenance time window constraint is violated, such as the equipment maintenance time is extended, it is necessary to update the equipment maintenance time window in the equipment availability matrix and regenerate the equipment allocation plan. Through these adjustments, the feasibility of the execution of the feasible solution is ensured. For example, suppose process A was originally planned to be carried out on equipment 1, but due to the failure of equipment 1, process A is reallocated to equipment 2.

[0163] The present invention optimizes equipment allocation and process sequencing by taking into account constraints such as equipment capacity, process dependency, and maintenance time windows, thereby maximizing the use of equipment resources and reducing equipment idle time; by optimizing process scheduling and equipment allocation, process waiting time and equipment switching time can be reduced, thereby shortening the entire production cycle and improving production efficiency; through real-time monitoring and adjustment, it is possible to respond to emergencies such as equipment failures and process delays, quickly adjust production plans, and enhance the flexibility and robustness of the production system.

[0164] Figure 2 Schematic diagram of the structure of the semiconductor manufacturing Q-time control system based on deep reinforcement learning according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0165] The first unit is used to obtain process parameters, equipment status information, inter-process waiting time and batch information in the semiconductor manufacturing process, standardize the process parameters, extract the timing characteristics of the equipment status information, and input the standardized process parameters, timing characteristics of the equipment status information, inter-process waiting time and batch information into the deep reinforcement learning model;

[0166] The second unit is used for the deep reinforcement learning model to construct a state space based on the input information, input the state space into a dual deep neural network, the first neural network of the dual deep neural network is used to estimate the current state value, and the second neural network of the dual deep neural network is used to predict the target state value; the Q-time violation risk is calculated according to the difference between the current state value and the target state value, and a scheduling action is selected from a preset action space based on the Q-time violation risk; wherein the deep reinforcement learning model is trained using a comprehensive reward function, and the comprehensive reward function includes a Q-time violation penalty item, a production efficiency reward item, an equipment utilization reward item, and a batch balance reward item;

[0167] The third unit is used to monitor the semiconductor manufacturing process in real time according to the selected scheduling action, and input the manufacturing process data obtained by real-time monitoring into a multi-objective optimization decision module. The multi-objective optimization decision module comprehensively considers the maximization of production capacity and the minimization of Q-time violations to generate scheduling instructions, and dynamically adjusts the process arrangement in the semiconductor manufacturing process according to the scheduling instructions.

[0168] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including: a processor;

[0169] a memory for storing processor-executable instructions;

[0170] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0171] A fourth aspect of the embodiments of the present invention is:

[0172] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0173] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semiconductor manufacturing Q-time control method based on deep reinforcement learning, characterized in that: include: Obtaining process parameters, equipment status information, inter-process waiting time and batch information in a semiconductor manufacturing process, standardizing the process parameters, extracting timing features of the equipment status information, and inputting the standardized process parameters, timing features of the equipment status information, inter-process waiting time and batch information into a deep reinforcement learning model; The deep reinforcement learning model constructs a state space based on the input information, and inputs the state space into a dual deep neural network, wherein the first neural network of the dual deep neural network is used to estimate the current state value, and the second neural network of the dual deep neural network is used to predict the target state value; the Q-time violation risk is calculated according to the difference between the current state value and the target state value, and a scheduling action is selected from a preset action space based on the Q-time violation risk; wherein the deep reinforcement learning model is trained using a comprehensive reward function, and the comprehensive reward function includes a Q-time violation penalty item, a production efficiency reward item, an equipment utilization reward item, and a batch balance reward item; Real-time monitoring of the semiconductor manufacturing process is performed according to the selected scheduling action, and the manufacturing process data obtained by the real-time monitoring is input into a multi-objective optimization decision module, the multi-objective optimization decision module generates a scheduling instruction by comprehensively considering the maximization of production capacity and the minimization of Q-time violations, and dynamically adjusts the process arrangement of the semiconductor manufacturing process according to the scheduling instruction; The deep reinforcement learning model constructs a state space based on the input information, and inputs the state space into a dual deep neural network, wherein the first neural network of the dual deep neural network is used to estimate the current state value, and the second neural network of the dual deep neural network is used to predict the target state value; the steps of calculating the Q-time violation risk according to the difference between the current state value and the target state value, and selecting a scheduling action from a preset action space based on the Q-time violation risk include: A sliding time window method is used to extract parameter features from process parameters, wherein the parameter features include mean, standard deviation and trend slope, and a time series coding method is used to obtain equipment operation status features, wherein the equipment operation status features include state duration and switching frequency. The parameter features and the equipment operation status features are combined with the waiting time between processes and batch information to construct a multidimensional state space of the deep reinforcement learning model; Inputting the multidimensional state space into a dual deep neural network of the deep reinforcement learning model, the dual deep neural network comprising a first neural network for estimating a current state value and a second neural network for predicting a target state value, wherein the first neural network adopts a residual structure and is connected through residuals and multiple layers of neurons, and the second neural network adopts a long short-term memory network structure enhanced by an attention mechanism and is fused with features of the attention layer and memory unit; The deep reinforcement learning model calculates the difference between the target state value and the current state value to obtain a state value difference, and constructs a bidirectional risk calculation function based on the state value difference in combination with the Q-time upper limit constraint and the lower limit constraint to calculate the Q-time violation risk; In the action space of batch movement, batch priority adjustment, equipment switching and batch merging preset in the deep reinforcement learning model, the action value function of the candidate action is calculated based on the current state value and the Q-time violation risk, the candidate action includes all executable specific action instances in batch movement, batch priority adjustment, equipment switching and batch merging, the action value function includes immediate reward and risk penalty, and a risk-adaptive exploration strategy is adopted to select scheduling actions from the action space.

2. The method according to claim 1, characterized in that: In the action space of batch movement, batch priority adjustment, equipment switching and batch merging preset in the deep reinforcement learning model, the action value function of the candidate action is calculated based on the current state value and the Q-time violation risk, the candidate action includes all executable specific action instances in batch movement, batch priority adjustment, equipment switching and batch merging, the action value function includes immediate reward and risk penalty, and the step of selecting a scheduling action from the action space using a risk-adaptive exploration strategy includes: The batch movement includes a batch identifier, a starting device, a target device, and a movement time, the batch priority adjustment includes a batch identifier and a batch priority level, the device switching includes a batch identifier and a target device, and the batch merging includes a first batch identifier and a second batch identifier; Calculating an instant reward for the candidate action based on the current state value, wherein the instant reward is obtained by a weighted combination of the capacity improvement benefit, the degree of improvement in equipment utilization, and the degree of load balancing; The risk penalty value is calculated based on the Q-time violation risk and the action risk impact, wherein the action risk impact includes: the risk impact of batch movement is calculated according to the path change amplitude, the risk impact of batch priority adjustment is calculated according to the batch priority change amount, the risk impact of equipment switching is calculated according to the equipment difference, and the risk impact of batch merging is calculated according to the batch size ratio; Combining the immediate reward with the risk penalty value to construct an action value function, and calculating the value of a specific action instance in the candidate action based on the action value function; An exploration rate that is adaptively adjusted with the risk is used to select actions, and the exploration rate increases with the increase of the Q-time violation risk. According to the action value function and the exploration rate, switching is performed between the specific action instance with the greatest value and the random action instance according to a preset probability to obtain the final scheduling action.

3. The method according to claim 1, characterized in that The deep reinforcement learning model is trained using a comprehensive reward function, wherein the comprehensive reward function includes a Q-time violation penalty item, a production efficiency reward item, an equipment utilization reward item, and a batch balance reward item, and the steps include: The Q-time violation penalty is calculated based on the difference between the batch waiting time and the limit time. The production efficiency reward is calculated based on the number of batches processed per unit time. The equipment utilization reward is calculated based on the ratio of the actual equipment processing time to the total time. The batch balance reward is calculated based on the standard deviation of the number of work-in-progress in each process. Constructing a loss function based on the comprehensive reward function, the loss function including a temporal difference error term and a regularization term, wherein the temporal difference error term is calculated by subtracting the current state action value output by the first neural network from the sum of the reward value calculated by the comprehensive reward function and the discounted value of the next state maximum action value predicted by the second neural network; A priority experience replay mechanism is used to sample training samples, wherein the priority experience replay mechanism includes experience tuple storage, sample priority calculation and sampling probability calculation; Performing gradient descent update on the parameters of the first neural network based on the loss function, and updating the parameters of the second neural network by soft update, wherein the parameter update of the first neural network uses gradient clipping technology to limit the gradient norm, and the parameter update of the second neural network controls the update speed by soft update coefficient; The Q-time violation rate is calculated according to the Q-time violation risk, the average reward value fluctuation is calculated according to the comprehensive reward function, and the value estimation error is calculated according to the loss function. The training process is terminated when the Q-time violation rate, the average reward value fluctuation and the value estimation error are improved by less than a preset threshold for multiple consecutive rounds.

4. The method according to claim 3, characterized in that The priority experience replay mechanism is used to sample training samples. The priority experience replay mechanism includes the steps of experience tuple storage, sample priority calculation and sampling probability calculation, including: Constructing an experience tuple, wherein the experience tuple includes a current state, an execution action, a reward value calculated by the comprehensive reward function, a next state after the execution action, and a temporal difference error term, wherein the current state is composed of a process parameter vector, an equipment state vector, an inter-process waiting time vector, and a batch information vector; The experience tuple is stored in a playback buffer pool of fixed capacity according to a first-in-first-out principle, and the sample priority of the experience tuple is calculated based on the absolute value of the temporal difference error term, wherein the sample priority ensures that all experience tuples have a non-zero sampling probability by adding a bias value; The sampling probability of each experience tuple is calculated according to the sample priority, the sampling probability is obtained by the ratio of a single sample priority to the sum of all sample priorities in the playback buffer pool, and an importance sampling index is introduced to adjust the sampling probability, and the importance sampling index is dynamically adjusted with the training process; extracting training samples from the playback buffer pool based on the sampling probability, and calculating the importance weight of the training samples, wherein the importance weight is inversely proportional to the sampling probability; The importance weight is multiplied by the loss function to obtain a weighted loss function, the parameters of the first neural network are updated based on the weighted loss function, and the sample priority is updated through the temporal difference error term to achieve dynamic optimization of the priority experience replay mechanism.

5. The method according to claim 1, characterized in that The semiconductor manufacturing process is monitored in real time according to the selected scheduling action, and the manufacturing process data obtained by the real-time monitoring is input into a multi-objective optimization decision module, wherein the multi-objective optimization decision module generates a scheduling instruction by comprehensively considering the maximization of production capacity and the minimization of Q-time violations, and the step of dynamically adjusting the process arrangement in the semiconductor manufacturing process according to the scheduling instruction includes: Collecting equipment status data through equipment interfaces, the equipment status data including operation status, maintenance status, fault status and equipment available time window; collecting process parameter data through sensor networks, the process parameter data including temperature, pressure and gas flow; collecting batch tracking data of the current process, waiting time and processing time of the batch through the manufacturing execution system, as well as constraint data of the maximum allowable interval time between adjacent processes; Inputting equipment status data, process parameter data, batch tracking data and constraint data into a multi-objective optimization decision module, and constructing a multi-objective optimization model including a capacity maximization objective function and a Q-time violation minimization objective function based on the multi-objective optimization decision module, wherein the capacity maximization objective function is obtained by weighting the ratio of the number of completed batches to the total number of batches and the equipment utilization rate, and the Q-time violation minimization objective function is obtained by weighting the ratio of the number of overtime processes to the total number of processes and the overtime duration; Based on the multi-objective optimization model, the equipment capacity constraint, the process dependency constraint and the equipment maintenance time window constraint are combined to solve, and the dynamic scheduling priority of each process to be scheduled is calculated according to the capacity maximization objective function value and the Q-time violation minimization objective function value, and equipment resources are pre-allocated to the high-priority process based on the dynamic scheduling priority, and the specific scheduling time point is determined in combination with the equipment available time window and the process processing time, and the scheduling instruction is generated; The scheduling instructions are issued to the corresponding processes and equipment through the manufacturing execution system, and the current capacity index and Q-time violation rate are calculated in real time. When the deviation between the capacity index and the Q-time violation rate and the expected target is less than a first threshold, the weight coefficients of the capacity maximization objective function and the Q-time violation minimization objective function are adjusted for local optimization. When the deviation is greater than a second threshold, the multi-objective optimization model is re-solved to generate a new scheduling instruction.

6. The method according to claim 5, characterized in that Based on the multi-objective optimization model, the steps of solving the problem in combination with the equipment capacity constraint, the process dependency constraint and the equipment maintenance time window constraint include: Constructing an equipment availability matrix, wherein the rows of the equipment availability matrix represent equipment numbers, the columns represent discrete time points, and the available status of the equipment at different time points is represented by binary variables, and the available equipment set of each process is determined based on the equipment available status; generating equipment allocation variables according to the available equipment set, wherein the equipment allocation variables represent the corresponding relationship between the process and the equipment, forming an equipment allocation plan based on the equipment allocation variables, and establishing equipment capacity constraints to ensure that the equipment allocation plan at any time does not exceed the equipment availability; Based on the equipment allocation variables, process dependency constraints are established, the completion time and start time of consecutive processes are associated, and the maximum allowable interval time constraint between adjacent processes is introduced to ensure that the equipment allocation plan meets the process dependency requirements; Marking the equipment maintenance time window in the equipment availability matrix to form an equipment maintenance time window constraint, setting the equipment availability status corresponding to the maintenance period to unavailable, and canceling the equipment allocation plan that has a time intersection with the equipment maintenance time window according to the non-interruptible characteristic of the maintenance activity; The equipment capacity constraint, process dependency constraint and equipment maintenance time window constraint are constructed into a unified constraint matrix, and a Lagrangian relaxation method is used to solve the feasible solution that satisfies the constraint conditions; The process execution progress is monitored in real time. When a violation of the equipment capacity constraint, the process pre- and post-dependency constraint, or the equipment maintenance time window constraint is detected, for the violation of the equipment capacity constraint caused by equipment unavailability, the equipment allocation variables are regenerated in the available equipment set; for the violation of the process pre- and post-dependency constraint, the maximum allowed interval time constraint between adjacent processes is adjusted; for the violation of the equipment maintenance time window constraint, the equipment maintenance time window in the equipment availability matrix is ​​updated to ensure the execution feasibility of the feasible solution.

7. A semiconductor manufacturing Q-time control system based on deep reinforcement learning, used to implement the method described in any one of claims 1 to 6, characterized in that: include: The first unit is used to obtain process parameters, equipment status information, inter-process waiting time and batch information in the semiconductor manufacturing process, standardize the process parameters, extract the timing characteristics of the equipment status information, and input the standardized process parameters, timing characteristics of the equipment status information, inter-process waiting time and batch information into the deep reinforcement learning model; The second unit is used for the deep reinforcement learning model to construct a state space based on the input information, input the state space into a dual deep neural network, the first neural network of the dual deep neural network is used to estimate the current state value, and the second neural network of the dual deep neural network is used to predict the target state value; the Q-time violation risk is calculated according to the difference between the current state value and the target state value, and a scheduling action is selected from a preset action space based on the Q-time violation risk; wherein the deep reinforcement learning model is trained using a comprehensive reward function, and the comprehensive reward function includes a Q-time violation penalty item, a production efficiency reward item, an equipment utilization reward item, and a batch balance reward item; The third unit is used to monitor the semiconductor manufacturing process in real time according to the selected scheduling action, and input the manufacturing process data obtained by real-time monitoring into a multi-objective optimization decision module. The multi-objective optimization decision module comprehensively considers the maximization of production capacity and the minimization of Q-time violations to generate scheduling instructions, and dynamically adjusts the process arrangement in the semiconductor manufacturing process according to the scheduling instructions.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images