Wafer manufacturing scheduling method and system based on dynamic reward mechanism q-learning

Abstract

The application discloses a wafer manufacturing scheduling method and system based on a dynamic reward mechanism Q-learning, and mainly solves the problems of sparse rewards, target attenuation and redundant detours in wafer manufacturing scheduling in the prior art. The scheme is as follows: based on the processing process of a wafer, a Petri net model with time constraints is constructed; based on the time Petri net model, the state-action quality evaluation value Q is iteratively updated through the interaction between an intelligent agent and a manufacturing environment, and a table Q-table storing the quality evaluation values corresponding to each state-action pair is generated; the table of quality evaluation values is used for searching to generate an initial scheduling solution; the generation of the quality evaluation table and the searching to obtain an initial scheduling solution set are cyclically executed, and secondary deep optimization is performed on the initial scheduling solution set, and a globally optimal scheduling path is output. The application can balance global exploration and local utilization, improve the quality and feasibility of the initial scheduling solution, and improve the solving performance and efficiency of global scheduling of a complex wafer manufacturing system, and can be used for production scheduling optimization of wafer manufacturing.

Description

Technical Field

[0001] This invention belongs to the field of microelectronics technology, and in particular relates to a wafer manufacturing scheduling method and system, which can be used for wafer manufacturing production scheduling optimization, and provides a Q-learning intelligent scheduling scheme with dynamic reward mechanism for semiconductor wafer manufacturing system scheduling. Background Technology

[0002] In modern microelectronics manufacturing systems, semiconductor wafer fabrication is the most crucial link determining chip yield and production capacity. As manufacturing processes evolve towards higher precision, wafer fabrication equipment is gradually developing into highly automated, modular cluster tools. These systems not only involve strictly sequential multi-stage processes but also suffer from highly frequent wafer re-entry phenomena and stringent wafer dwell time constraints. These demanding physical and timing conditions make wafer manufacturing scheduling a typical NP-hard problem. To accurately describe the complex concurrent execution, resource preemption, and deadlock avoidance mechanisms within the system, temporal Petri nets, due to their rigorous mathematical foundation and topological representation advantages, are widely used for the underlying mathematical modeling of complex wafer scheduling.

[0003] In recent years, with breakthroughs in artificial intelligence technology, reinforcement learning algorithms, represented by Q-learning, have begun to be introduced into the field of scheduling, with the aim of autonomously learning high-quality initial scheduling strategies through dynamic trial-and-error interaction between the agent and the manufacturing environment.

[0004] Patent application CN202311513142.3 discloses a workshop scheduling method, apparatus, and system based on deep reinforcement learning. It transforms the dynamic flexible workshop scheduling problem into a Markov decision process and achieves adaptive scheduling through state features, composite scheduling rules, and a deep Q-network. However, this method is primarily geared towards general dynamic flexible workshops, with reward design focusing on machine utilization. It does not optimize for reward sparsity, ineffective detours, and low-quality path selection issues under constraints such as long transition sequences, reentrant processing, and dwell time in wafer manufacturing.

[0005] Patent application CN202210406935.4 discloses an adaptive scheduling method for job shops based on deep reinforcement learning. This method improves the proximal policy optimization algorithm and combines it with graph neural networks to extract shop state features, achieving adaptive scheduling of the job shop. However, this method is mainly applicable to general job shop scheduling and does not incorporate temporal Petri nets to describe concurrency, resource contention, and deadlock constraints in wafer manufacturing systems. Furthermore, it does not address the issues of reverse reward decay and inefficient path retention in long-sequence scheduling using Q-learning.

[0006] In summary, standard Q-learning reinforcement learning exhibits the following shortcomings in wafer fabrication scheduling scenarios involving long transition sequences and strong constraints:

[0007] First, there are issues with reward sparsity and inefficient loops. Standard Q-learning often provides fixed rewards for intermediate states, causing agents to engage in inefficient loops and redundant operations—essentially taking detours to accumulate immediate rewards—and failing to progress towards the goal of minimizing the global completion time. Second, the target reward decays too rapidly in the reverse direction. In long path sequences, traditional discount factors cause the completion reward obtained by the agent upon reaching the destination to decay sharply when backpropagating to the initial state, making the algorithm prone to policy delusion in the early stages of scheduling. Third, there is a lack of a dynamic evaluation mechanism for weeding out inferior policies. Standard algorithms provide fixed positive feedback as soon as the target state is reached, failing to distinguish between high-quality scheduling paths and inefficient redundant paths. This results in a Q-table filled with mediocre policies, leading to an inconsistent quality of the initial scheduling solution set.

[0008] Therefore, it is urgent to propose a reinforcement learning initialization method with dynamic reward adjustment and anti-detour mechanism under the temporal Petri net scheduling framework, so as to give full play to the advantages of reinforcement learning adaptive decision-making, overcome the convergence defects under long sequence scheduling, and effectively improve the global optimization capability and solution efficiency of complex wafer manufacturing systems. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the prior art and propose a wafer manufacturing scheduling method and system based on dynamic reward mechanism Q-learning. This avoids the reward sparsity, target decay and redundant detours that are prone to occur when using standard Q-learning reinforcement learning algorithms to deal with long sequence scheduling, and effectively improves the global optimization capability and solution efficiency of complex wafer manufacturing systems.

[0010] The technical approach to achieve the objective of this invention is as follows: A time-based Petri net model is established based on equipment resources, process constraints, and dwell time limitations during wafer fabrication. On this basis, a dynamic reward mechanism is introduced to improve the Q-learning training process. Redundant detours are suppressed through time consumption negative feedback, high-quality complete paths are reinforced through dynamic record-breaking rewards, and reward sparsity in long sequence scheduling is alleviated. The backpropagation capability of termination rewards in long transition sequences is enhanced through a high-dimensional discount factor, and the target reward decay problem is optimized, thereby generating a high-quality Q-table. Then, feasible initial scheduling solutions are constructed in batches based on the Q-table, and these solutions are input into an evolutionary algorithm for secondary global optimization, ultimately obtaining a scheduling scheme that satisfies wafer manufacturing constraints and has a relatively optimal completion time.

[0011] Based on the above ideas, the technical solution of the present invention includes:

[0012] 1. A wafer manufacturing scheduling method based on Q-learning with a dynamic reward mechanism, characterized in that it includes:

[0013] (1) Based on the wafer fabrication process, construct a Petri net model with time constraints;

[0014] (2) Based on the time Petri net model, the state-action quality evaluation value Q is updated iteratively through the interaction between the agent and the manufacturing environment, and a table Q-table is generated to store the quality evaluation values ​​corresponding to each state-action.

[0015] (3) Use the Q-table of quality assessment values ​​to search and generate an initial scheduling solution;

[0016] (4) Repeat steps (2) and (3) to obtain the initial scheduling solution set, perform a second-order deep optimization, and output the globally optimal scheduling path.

[0017] Furthermore, in step (2), based on the time-based Petri net model, the state-action quality evaluation value is iteratively updated through the interaction between the agent and the manufacturing environment, generating a table Q-table that stores the quality evaluation values ​​corresponding to each state-action. Its implementation includes:

[0018] 2a) Based on whether the state terminates, the time consumption of the action, the comparison between the current scheduling result and the historical best result, and the difference between the current scheduling result and the target value, construct a dynamic reward function to obtain the instantaneous state reward value. ;

[0019] 2b) During the update process of the state-action quality assessment value Q, for the immediate state reward value A high-dimensional discount factor is introduced to suppress the reverse decay of the target reward value, while a dynamic exploration rate decay strategy is adopted to reduce the probability of random exploration in the later stages. ;

[0020] 2c) Output the updated Q-table used to record the quality evaluation values ​​corresponding to each state-action.

[0021] Furthermore, in step (3), the initial scheduling solution is generated by searching using the Q-table of quality assessment values. This process includes:

[0022] 3a) Obtain all legal transition actions under the current identifier state to obtain the candidate transition set;

[0023] 3b) Sort the action transitions in the candidate transition set in descending order according to the Q-table of quality assessment values;

[0024] 3c) Using a greedy strategy, prioritize actions with high state-action quality evaluation values ​​(Q) for state expansion, and generate the next identifier based on the action transition;

[0025] 3d) Validate the next identifier using time constraints and taboo constraints:

[0026] If the next identifier passes the verification, the action transition in the transition set is added to the current scheduling sequence, and the identifier is updated to the new current identifier;

[0027] If the next identifier fails the verification, the next transition action is selected from the candidate transition set for state expansion.

[0028] 3e) Repeat 3a) to 3d) until the target identifier state is reached, and record the complete scheduling sequence consisting of the transition actions as an initial scheduling solution.

[0029] 2. A wafer manufacturing scheduling system based on Q-learning with a dynamic reward mechanism, characterized in that it includes:

[0030] The model building module is used to construct a time-based Petri net model based on the sequential and parallel execution relationships in the wafer fabrication process.

[0031] The reward function construction module is used to construct a dynamic reward function based on whether the state has terminated, the time consumption of the action, the comparison between the current scheduling result and the historical best result, and the difference between the current scheduling result and the target value.

[0032] The quality assessment value Q update module is used to calculate immediate rewards based on a dynamic reward function. Furthermore, a high-dimensional discount factor and a dynamic exploration rate decay strategy are introduced to update the state-action quality assessment value;

[0033] The Q-table generation module is used to generate a Q-table to record the quality assessment values ​​Q corresponding to each state-action after the state-action quality assessment values ​​Q converge.

[0034] The initial scheduling solution generation module is used to sort and expand the candidate transition actions according to the Q-table to obtain the initial scheduling solution set;

[0035] The global optimization module is used to input the initial scheduling solution set into the evolutionary algorithm, perform a second global optimization with the goal of minimizing the global completion time, and output the globally optimal scheduling path.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] This invention introduces a dynamic reward mechanism based on the temporal Petri net model. For non-terminating states, it introduces time consumption negative feedback to suppress redundant actions and ineffective detours. For terminating states, it adjusts the reward intensity of the complete scheduling path by combining a dynamic record-breaking prize and a target gap decay mechanism. Compared to a fixed reward method, this invention can distinguish between high-quality and inefficient paths, strengthen high-quality complete scheduling paths, and reduce the impact of low-quality feasible paths on the state-action quality evaluation table Q-table, thereby improving the discriminative power of the state-action quality evaluation value Q and the effectiveness of the Q-learning algorithm.

[0038] Secondly, by introducing a high-dimensional discount factor and a dynamic exploration rate decay strategy during the update process of the state-action quality evaluation value Q, this invention ensures that the initial state-action can still be effectively guided by the long-term target reward, alleviating the problems of reward sparsity and target signal decay in long-sequence wafer scheduling. At the same time, it enables the agent to maintain a high probability of random exploration in the early stage of exploration and gradually reduce the probability of random exploration in the later stage of exploration, selecting high-quality transition actions more based on the Q-table. Therefore, it can take into account both global exploration and local utilization, reduce the interference of invalid random actions in the later stage on the convergence process, and improve the learning stability and convergence effect in long-path scheduling scenarios.

[0039] Thirdly, this invention utilizes the converged state-action quality evaluation table Q-table to sort and greedily expand candidate transition actions, and combines time constraints and taboo constraints to generate initial scheduling solutions. This ensures that the initial solutions no longer rely on completely random generation, but are guided by the converged state-action quality evaluation information. Therefore, it improves the overall quality and feasibility of the initial scheduling solution set, providing a better initial search foundation for subsequent evolutionary algorithms' secondary global optimization. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the implementation of the wafer manufacturing scheduling method based on the dynamic reward mechanism Q-learning of this invention.

[0041] Figure 2 This is a schematic diagram of the time-based Petri net model established in this invention;

[0042] Figure 3 This is a block diagram of the wafer manufacturing scheduling system based on the dynamic reward mechanism Q-learning of this invention. Detailed Implementation

[0043] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative effort should all fall within the protection scope of the present invention.

[0044] Example 1: A wafer manufacturing scheduling method based on Q-learning dynamic reward mechanism.

[0045] Reference Figure 1 The implementation steps of this embodiment include:

[0046] Step 1: Based on the wafer fabrication process, construct a Petri net model with time constraints.

[0047] Wafer fabrication is typically accomplished using modular equipment. This equipment works collaboratively with modules such as loading, robotic arms, calibration, vacuum transfer, process chambers, and unloading to enable continuous wafer transfer between different processing stages. During this process, while there are strict sequential relationships between different processes, multiple processing chambers or transfer modules may execute serially or in parallel. Furthermore, it is constrained by factors such as equipment capacity, robotic arm occupancy, process dwell time, and processing time windows.

[0048] This step establishes a Petri net model to describe the changes in the operating state and resource flow of the combined equipment, based on the serial and parallel execution relationships between the various processes in the wafer fabrication process. The model uses identifiers to represent different states of the combined equipment during operation, places to represent resources such as processing modules, transmission modules, buffer modules, and robotic arms in the equipment, and transitions to represent the handling actions performed by the robotic arms between different modules.

[0049] Furthermore, the process time window, equipment occupancy conditions, and resource constraints are mapped to the corresponding warehouses and transitions. Processing time, waiting time limits, dwell time, equipment capacity, and resource exclusivity conditions are added to the warehouses or transitions, forming a time-constrained warehouse-transition Petri net model, such as... Figure 2 As shown.

[0050] Reference Figure 2This place-transition Petri net model uses wafer cassettes, calibration modules, vacuum locks, processing chambers, transport paths, and robotic arm resources in the combined equipment as modeling objects. Specifically: LP represents the wafer cassette module used to store wafers to be processed and receive processed wafers; AL represents the calibration module used to perform position calibration and attitude adjustment before the wafers enter the formal processing flow; LLA / LLB represents the vacuum lock module used to connect the atmospheric and vacuum ends; and LLC / LLD represents the vacuum lock module that completes wafer loading / unloading and pressure state transitions. PM1 / PM2 represent the wafer first-stage process processing modules, PM7 / PM82 represent the wafer first-stage process processing modules, TM1 represents the robotic arm module in the atmospheric transfer area used to pick up and place the wafer between the wafer cassette LP, calibration module AL, and vacuum lock inlet / outlet, TM2 represents the robotic arm module in the first vacuum transfer area used to transfer and exchange the wafer between the vacuum lock module and some processing chamber modules, and TM3 represents the robotic arm module in the second vacuum transfer area used to transfer and exchange the wafer between the remaining processing chamber modules, cooling or transfer modules.

[0051] Figure 2 The process path is LP-TM1-LLA / LLB-TM2-PM7 / PM8-LLC-TM3-PM1 / PM2-LLD-LLA / LLB-TM1-LP. That is, the wafer starts from the wafer cassette LP, is taken out by the loading robot TM1 and enters the calibration and feeding process. Then it enters the vacuum side through the vacuum lock LLA / LLB. Subsequently, the robot TM2 sends the wafer into the PM7 or PM8 module to perform the first stage of parallel processing. After passing through the LLC, the robot TM3 enters the PM1 or PM2 module to perform the second stage of parallel processing. After processing, the wafer returns to the output path through the LLD and is finally sent back to the wafer cassette by the robot TM1.

[0052] Step 2: Based on the time-based Petri net model, the state-action quality assessment value Q is updated iteratively through the interaction between the agent and the manufacturing environment, generating a table Q-table that stores the quality assessment values ​​corresponding to each state-action.

[0053] This step involves using the flag state in the temporal Petri net as the agent's state, and the legal transitions that can be triggered under the current flag state as the agent's optional actions. Let the current flag state be... The agent is in the current identified state The action transition of the next selection is Changes in execution actions Then transition to the next state After each state transition, the action changes accordingly. The resulting time consumption, the next state Calculate the immediate state reward value corresponding to this state transition based on whether the state is terminated and the global completion time advantage of the current scheduling path. The reward value will be used to update the current state-action pair. Quality assessment value Its implementation includes:

[0054] 2.1) Determine the next state Calculate the immediate state reward value based on whether the state is in a terminated state. :

[0055] If the next state Assuming the agent is in a non-terminating state, let the action transition be defined. The total time taken before was Changes in execution actions The subsequent global execution time is and the negative of the difference between the two As a reward value for non-termination state Receive immediate status reward value ;

[0056] If the next state Assuming the current state is terminated, let the global completion time for the current scheduling path be... The best historical completion time is The basic reward is and set the current global completion time Best completion time in history Compare:

[0057] when When this occurs, it indicates that the current scheduling path is better than the historical best path, triggering the dynamic record-breaking prize mechanism and updating... The best historical completion time before the update is recorded as . And assign basic rewards to intelligent agents. and additional rewards That is, the reward value of the terminated state. Receive immediate status reward value ;

[0058] when When the current scheduling path is not better than the historical best path, the target gap decay mechanism is triggered, based on the current time elapsed. Best time in history The difference in the base reward The coefficient is The linear decay, i.e., the reward value at the termination state. Receive immediate status reward value .

[0059] 2.2) Obtaining instant state reward value Then, a high-dimensional discount factor is introduced. To prevent reward decay, determine whether the next state is a termination state in order to determine the target evaluation value. :

[0060] When the next state When the state is terminated, an immediate reward will be given. Sure Target evaluation value, i.e. ;

[0061] When the next state When the state is not terminated, an immediate reward will be given. And the optimal action evaluation value for the next state With discount factor Jointly determine the target evaluation value ;

[0062] 2.2) Based on the already obtained target evaluation values and learning rate The updated quality assessment value was determined. The updated quality assessment value Q is written into the quality assessment table Q-table as the basis for subsequent action selection.

[0063] 2.3) Based on the updated quality assessment table (Q-table), determine the selection method for the next action:

[0064] The range of values ​​to be generated is random numbers Let the current exploration rate be... Determine random numbers exploration rate with current state Relationship:

[0065] when The time-based exploration mechanism is used, which means that a transition action is randomly selected with equal probability from all triggerable legal transitions in the current Petri net state.

[0066] when The execution mechanism utilizes the current state to select the action transition with the highest action quality assessment value Q in the Q-table for execution;

[0067] 2.4) After each round of updating the state-action quality assessment value Q, in order to gradually shift the agent from full exploration in the early stage to stable utilization in the later stage, in conjunction with the first... exploration rate Compared with the preset exploration rate decay coefficient For the next exploration rate Perform a decay update to obtain the current exploration rate. ;

[0068] 2.5) Repeat the state transition, immediate reward calculation, target evaluation value calculation, Q value update and exploration rate decay process until the termination flag is reached, and output a Q-table to record the quality evaluation value corresponding to each state-action.

[0069] Step 3: Use the Q-table of quality assessment values ​​to search and generate an initial scheduling solution.

[0070] This step uses the Q-table obtained in step 2 as the basis for action transition selection, and gradually expands the scheduling path from the initial identifier state of the temporal Petri net to generate an initial scheduling solution. Its implementation includes:

[0071] 3.1) Under the current flag state, obtain all legal action transitions that meet the trigger conditions, as the set of candidate transitions to be selected under the current state;

[0072] 3.2) Based on the current identifier state and the state-action quality evaluation value Q between each candidate action transition recorded in the quality evaluation table Q-table, sort the candidate transition set in descending order to obtain a candidate transition sequence with high priority to low priority.

[0073] 3.3) Following a greedy strategy, select the action transition with the higher state-action quality evaluation value Q from the sorted candidate transition sequence as the transition to be executed, and trigger the generation of the next labeled state for that action transition;

[0074] 3.4) After generating the next identifier state, perform time constraint and taboo constraint checks on the next identifier state, where:

[0075] The time constraint check is used to determine whether the process time window, wafer dwell time, and equipment occupancy time requirements are met after the action transition is executed. Its implementation includes:

[0076] 3.4.1) Obtain the entry time, exit time, and dwell time of each relevant repository after the action transition is executed;

[0077] 3.4.2) Compare the entry time and exit time with the preset dwell time and processing time window of the warehouse or corresponding transition to determine whether the action transition occurs within the allowable process time range;

[0078] If the action transition is allowed to occur, then the next flag state is determined to have passed the time constraint check.

[0079] If the action transition is not allowed to occur, it is determined that it has failed the time constraint check.

[0080] The taboo constraint check is used to determine whether the next identifier state belongs to a state that has been prohibited or is not suitable for repeated expansion, and its implementation includes:

[0081] 3.4.3) Match the recorded taboo states based on whether there is a next flag state in the taboo table:

[0082] If the taboo does not exist in the taboo list, it means that the next identifier state is not restricted and can continue to be expanded;

[0083] If it exists in the taboo table, compare the residence time of the corresponding repository or the occupation time of the corresponding resource in the current next identifier state with the taboo time threshold:

[0084] If the current time has reached or exceeded the taboo time threshold, the taboo condition is lifted and expansion is allowed;

[0085] If the current time is still less than the taboo time threshold, then the next flag state is determined to have triggered a taboo constraint, and it will not be expanded further.

[0086] 3.5) Based on the time constraints and taboo constraints for the next flag state verified in 3.4), determine whether the current action transition is a valid extended action:

[0087] If the next flag state passes the verification, the action transition selected this time will be added to the current scheduling sequence and updated to the new current flag state, and the next round of state expansion will continue.

[0088] If the next identified state fails the time constraint or taboo constraint check, the selected action transition is discarded, and the next action transition is selected from the candidate transition set for state expansion.

[0089] If none of the action transitions in the current candidate transition set can pass the verification, return to the previous expandable identifier state and reselect other action transitions to avoid the scheduling path from falling into an infeasible state.

[0090] 3.6) Repeat steps 3.1) to 3.5) to obtain candidate transitions, sort Q-table in descending order, select greedy actions, generate the next identifier, and verify constraints until the current identifier state reaches the target identifier state. Then, combine the action transitions recorded in the current scheduling sequence according to the execution order into a complete scheduling sequence as an initial scheduling solution.

[0091] Step 4: Repeat steps (2) and (3) to obtain the initial scheduling solution set, perform a second-order deep optimization, and output the globally optimal scheduling path.

[0092] 4.1) Set the initial scheduling solution set size and generate the initial scheduling solution set;

[0093] To meet the initial population size requirements of the evolutionary algorithm, the expected initial scheduling solution set size is set to be... Repeat steps 2 and 3 to obtain... 1 initial scheduling solution is obtained and these solutions are aggregated to form an initial scheduling solution set;

[0094] 4.2) The initial set of scheduling solutions is used as the initial population of the evolutionary algorithm and preliminary screening is performed to form the current set of scheduling solutions. This set is then used as the initial population of the evolutionary algorithm. The global completion time of each scheduling solution in the initial set of scheduling solutions is calculated, and high-quality scheduling solutions with smaller global completion times are retained to form the current set of scheduling solutions.

[0095] 4.3) Perform an adaptive crossover operation on the current scheduling solution set, and add the crossover-generated scheduling solution to the current scheduling solution set. Adaptively determine the crossover method based on the fitness dispersion of the current scheduling solution set:

[0096] When the difference between scheduling solutions is greater than the set multi-point crossover probability At the same time, increase the probability of multiple intersections to expand the search range;

[0097] When the difference between scheduling solutions is less than the set multi-point crossover probability At this time, single-point intersection is used to enhance local optimization capabilities;

[0098] 4.4) Perform a local transition fragment mutation operation on the current scheduling solution set. Generate a new scheduling solution by adjusting the execution order of the local transition fragments or replacing feasible transition fragments, and add it to the current scheduling solution set.

[0099] 4.5) Iteratively update and output the globally optimal scheduling path.

[0100] The updated set of scheduling solutions is used as input for the next round. The fitness evaluation, selection of high-quality scheduling solutions, adaptive crossover, and mutation of local transition fragments are repeated until the program's set repetition rate is reached. The convergence condition is to select the scheduling individual with the smallest global completion time from the final scheduling solution set, and output the complete action transition sequence corresponding to the scheduling individual as the global optimal scheduling path.

[0101] It should be noted that the step numbers in this specification and claims are only for the purpose of clearly and completely describing the embodiments of the present invention and for ease of understanding, and their order is not limited.

[0102] Example 2: Wafer manufacturing scheduling system based on Q-learning dynamic reward mechanism.

[0103] Reference Figure 3This example includes: Model building module 1, Reward function building module 2, Quality evaluation value Q-table update module 3, Quality evaluation table Q-table generation module 4, Initial scheduling solution generation module 5, and Global optimization module 6. The Quality evaluation value Q-table update module 3 includes: Target evaluation value... Calculation submodule 31, Quality assessment value Q update submodule 32, Action selection submodule 33, Exploration rate Attenuation submodule 34.

[0104] The functions of each module and the data transmission methods are as follows:

[0105] The model building module 1 is used to build a time Petri net model based on the process sequence, parallel relationship, processing time, equipment capacity and resource constraints in the wafer processing flow, and then pass the model to the reward function building module 2.

[0106] The reward function construction module 2 calculates the immediate reward value R based on the time Petri net model output by the model construction module 1, taking into account whether the current state has terminated, the action time consumption, the comparison between the current scheduling result and the historical best result, and the difference between the current scheduling result and the target value. The immediate reward value R is then passed to the quality evaluation value Q update module 3.

[0107] The quality assessment value Q update module 3 updates the instantaneous reward value R output by the reward function construction module 2, combined with the high-dimensional discount factor and learning rate. A dynamic exploration rate decay strategy is used to iteratively update the state-action quality assessment value Q, wherein the target assessment value... The calculation submodule 31 calculates the target evaluation value based on the immediate reward value R output by the reward function construction module 2, combined with whether the next state is a termination state, the high-dimensional discount factor, and the optimal action quality evaluation value Q of the next state. Then the target evaluation value The quality assessment value Q is passed to the quality assessment value Q update submodule 32; the quality assessment value Q update submodule 32, based on the target assessment value... Combining the learning rate α and the original quality assessment value Q of the current state-action pair, the updated quality assessment value Q is calculated, and the updated result is written into the current Q-table and passed to the action selection submodule 33; the action selection submodule 33, based on the current Q-table and the exploration rate, calculates the updated quality assessment value Q and writes the updated result into the current Q-table and passes it to the action selection submodule 33; Identify the action transitions to be executed, and transmit the completion signal for the current action transition selection to the exploration rate. Attenuation submodule 34. This exploration rate. Attenuation submodule 34, combined with the current exploration rate and preset attenuation coefficient Calculate the next exploration rate. If the state-action quality assessment value Q does not meet the preset termination condition, update the exploration rate. Return to the action selection submodule 33 for the next round of action selection; when the state-action quality evaluation value Q meets the termination condition, the end signal is transmitted to the quality evaluation table Q-table generation module 4.

[0108] The quality assessment table Q-table generation module 4 generates a Q-table that records the quality assessment value Q corresponding to each state-action based on the state-action quality assessment value Q that meets the termination condition, and then passes the quality assessment table Q-table to the initial scheduling solution generation module 5.

[0109] The initial scheduling solution generation module 5 sorts and expands the candidate transition actions in the current identified state according to the quality evaluation table Q-table. After time constraint and taboo constraint verification, it obtains the initial scheduling solution set and passes the set to the global optimization module 6.

[0110] The global optimization module 6 uses the initial scheduling solution set output by the initial scheduling solution generation module 5 as the initial population of the evolutionary algorithm. It performs secondary global optimization through fitness evaluation, selection of high-quality individuals, adaptive crossover, and mutation of local transition segments, and outputs the scheduling path with the minimum global completion time.

[0111] It should be noted that the above functional modules can be implemented, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, as program instruction products. A program instruction product includes one or a set of program instructions. When the program instructions are loaded and executed on a computer, the described process or function is generated, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The program instructions can be stored in a computer-readable and writable storage medium, or transferred from one computer's readable and writable storage medium to another.

[0112] In this embodiment, the direct coupling or communication connection between the modules can be achieved through indirect coupling or communication connection via interfaces, devices, or modules. The functional modules and sub-modules in this embodiment can dynamically reside within a single processing unit, or each module can exist physically independently, or two or more modules can dynamically reside within a single processing unit. When these dynamic components are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable and writable storage medium. This storage medium can be a memory, disk, or optical disc, etc.

[0113] The effects of this invention can be further illustrated by the following simulation experiments:

[0114] Table 1 shows the comparison results of the mean and average completion time of the initial scheduling solution sets of the standard reinforcement learning algorithm and the method of the present invention after running 10 training cycles.

[0115] Table 1. Performance comparison results between standard reinforcement learning algorithms and the method of this invention.

[0116]

[0117] As shown in Table 1, the wafer manufacturing scheduling method based on the dynamic reward mechanism Q-learning proposed in this invention significantly outperforms the standard reinforcement learning algorithm in both the quality of the initial scheduling solution and the final average completion time. Specifically:

[0118] Regarding the quality of the initial scheduling solution, the average initial scheduling set generated by this invention is significantly reduced from 1535.91s in standard reinforcement learning to 1005.25s, a decrease of approximately 34.6%. This indicates that the time consumption negative feedback mechanism introduced in this invention completely eliminates the agent's ineffective detour behavior, providing a very high standard of optimization starting point for subsequent evolutionary algorithms.

[0119] Regarding the final optimization capability, thanks to the support of a high-quality initial population, after two rounds of deep optimization, the average completion time of this method was further reduced from 1010.16s in the standard reinforcement learning architecture to 768.62s, a reduction of approximately 23.9%. The experimental results fully verify that this method can not only raise the quality baseline of the initial scheduling solution, but also effectively improve the overall solution performance of global scheduling in complex wafer manufacturing systems.

[0120] The above description is merely a specific embodiment of the present invention and does not constitute a limitation thereof. Those skilled in the art, after understanding the content and principles of the present invention, can make formal or detailed modifications without departing from the principles and structure of the present invention. For example, in this example, the reward parameter, high-dimensional discount factor, learning rate, and exploration rate decay coefficient can all be adjusted according to the actual scheduling strategy. However, these modifications, equivalent substitutions, and changes based on the ideas of the present invention should all fall within the protection scope of the claims of the present invention.

Claims

1. A wafer manufacturing scheduling method based on Q-learning with a dynamic reward mechanism, characterized in that, include: (1) Based on the wafer fabrication process, construct a Petri net model with time constraints; (2) Based on the time Petri net model, the state-action quality evaluation value Q is updated iteratively through the interaction between the agent and the manufacturing environment, and a table Q-table is generated to store the quality evaluation values ​​corresponding to each state-action. (3) Use the Q-table of quality assessment values ​​to search and generate an initial scheduling solution; (4) Repeat steps (2) and (3) to obtain the initial scheduling solution set, perform a second-order deep optimization, and output the globally optimal scheduling path.

2. The method according to claim 1, characterized in that, The Petri net model with time constraints, based on the wafer fabrication process in (1), is implemented as follows: Based on the processing relationship of wafers executed serially or in parallel, a Petri net model is established to describe the process state changes and resource flow through places, transitions and their connections; The process time window, equipment occupancy conditions, and resource constraints are mapped to the corresponding locations and transitions to form a Petri net model with time constraints for locations and transitions.

3. The method according to claim 1, characterized in that, In (2), based on the time-based Petri net model, the state-action quality evaluation value is updated iteratively through the interaction between the agent and the manufacturing environment, generating a table Q-table that stores the quality evaluation values ​​corresponding to each state-action. Its implementation includes: 2a) Based on whether the state terminates, the time consumption of the action, the comparison between the current scheduling result and the historical best result, and the difference between the current scheduling result and the target value, construct a dynamic reward function to obtain the instantaneous state reward value. ; 2b) During the update process of the state-action quality assessment value Q, for the immediate state reward value A high-dimensional discount factor is introduced to suppress the reverse decay of the target reward value, while a dynamic exploration rate decay strategy is adopted to reduce the probability of random exploration in the later stages. ; 2c) Output the updated Q-table used to record the quality evaluation values ​​corresponding to each state-action.

4. The method according to claim 1, characterized in that, The initial scheduling solution is generated by searching the Q-table of quality assessment values ​​in (3), and its implementation includes: 3a) Obtain all legal transition actions under the current identifier state to obtain the candidate transition set; 3b) Sort the action transitions in the candidate transition set in descending order according to the Q-table of quality assessment values; 3c) Using a greedy strategy, prioritize actions with high state-action quality evaluation values ​​(Q) for state expansion, and generate the next identifier based on the action transition; 3d) Validate the next identifier using time constraints and taboo constraints: If the next identifier passes the verification, the action transition in the transition set is added to the current scheduling sequence, and the identifier is updated to the new current identifier; If the next identifier fails the verification, the next transition action is selected from the candidate transition set for state expansion. 3e) Repeat 3a) to 3d) until the target identifier state is reached, and record the complete scheduling sequence consisting of the transition actions as an initial scheduling solution.

5. The method according to claim 1, characterized in that, In step (4), steps (2) and (3) are executed cyclically to perform a second-order depth search on the initial scheduling solution set and output the globally optimal scheduling path. The implementation includes: 4a) Set the expected initial set size of scheduling solutions. Within a separate external control loop, the process of generating and searching the Q-table of quality assessment values ​​is repeatedly executed. Next, generate One initial scheduling solution; 4b) Based on the collected The initial scheduling solutions are obtained by obtaining the initial scheduling solution set; 4c) Input the initial scheduling solution set into the evolutionary algorithm, and perform a second global optimization exploration on the initial scheduling solution set with the goal of minimizing the global time, and output the globally optimal scheduling path.

6. The method according to claim 3, characterized in that, In step 2a), a dynamic reward function is constructed based on whether the state has terminated, the action time consumption, the comparison between the current scheduling result and the historical best result, and the difference between the current scheduling result and the target value, to obtain the instantaneous state reward value. Its implementation includes: 2a1) For non-terminating states, introduce negative feedback on time consumption to obtain the immediate state reward value. To obtain the global time taken before the action transition when the agent performs an action transition from the current state to the next non-terminating state. Global execution time after action transition The negative of the time consumption value Feedback is given to the agent to penalize it for ineffective stops or redundant jumps on non-critical paths, forcing the update of the state-action quality evaluation value Q to converge toward the action transition with the shortest time. 2a2) For the terminated state, combine the dynamic record-breaking prize and the target gap decay mechanism to obtain the instant state reward value. This achieves a strategy of survival of the fittest. When an agent transitions to the termination state corresponding to the target identifier, it obtains the current global completion time. And compare it with the best historical completion time. Compare: when At that time, the dynamic record-breaking prize mechanism is triggered, and an update is performed. The best historical completion time before the update is recorded as . And assign basic rewards to intelligent agents. Additional Rewards Receive the termination state reward value ; when When the target gap decay mechanism is triggered, it is adjusted based on the current time elapsed. Best time in history The difference in the base reward The coefficient is The linear decay yields the decayed state reward value R. t : 。 7. The method according to claim 3, characterized in that, In step 2b), during the update of the state-action quality assessment value Q, the immediate state reward value is... A high-dimensional discount factor is introduced to suppress the reverse decay of the target reward value, while a dynamic exploration rate decay strategy is adopted to reduce the probability of random exploration in the later stages. Its implementation includes: 2b1) For immediate state reward values Introducing a high-dimensional discount factor To prevent reward decay, first determine if the next state is a termination state to determine the target evaluation value. : The next state When the state is terminated, an immediate reward will be given. Sure Target evaluation value, i.e. ; The next state When the state is not terminated, an immediate reward will be given. And the optimal action evaluation value for the next state With discount factor Jointly determine the target evaluation value ; 2b2) Based on the target evaluation value and learning rate For current state-action pairs The quality assessment value is updated to obtain the updated quality assessment value. ; 2b3) Determine the selection method for the next action based on the updated quality assessment value Q: The range of values ​​to be generated is random numbers Determine random numbers exploration rate with current state Relationship: when The time-based exploration mechanism is used, which means that a transition action is randomly selected with equal probability from all triggerable legal transitions in the current Petri net state. when The execution mechanism is based on the current quality assessment table Q-table, which selects the action transition with the highest action quality assessment value in the current state for execution. 2b4) After each round of updating the state-action quality assessment value Q, combined with the first... exploration rate For the next exploration rate Perform a decay update to obtain the current exploration rate. ,in This is the preset exploration rate decay coefficient.

8. The method according to claim 5, characterized in that, In step 4c), the initial scheduling solution set is input into the evolutionary algorithm. A secondary global optimization search is performed on the initial scheduling solution set with the objective of minimizing global time, and the globally optimal scheduling path is output. This process includes: 4c1) Calculate the global completion time of each initial scheduling solution, and select and retain high-quality scheduling solutions based on the global completion time to form the current scheduling solution set; 4c2) Adaptively determine the crossover method based on the dispersion of the current scheduling solution set, perform single-point or multi-point crossover on individuals in the scheduling solution set, and add the scheduling solution generated by the crossover to the current scheduling solution set; 4c3) Select candidate scheduling solutions from the current scheduling solution set, perform local transition fragment mutation, and add the generated scheduling solution to the current scheduling solution set; 4c4) The updated scheduling solution set is used as the input for the next round. The filtering, crossover and mutation processes from 4c1) to 4c3) are repeated until the preset termination condition is met. The scheduling individual with the minimum global completion time in the scheduling solution set is output.

9. A wafer manufacturing scheduling system based on Q-learning dynamic reward mechanism, characterized in that, include: The model building module is used to construct a time-based Petri net model based on the sequential and parallel execution relationships in the wafer fabrication process. The reward function construction module is used to construct a dynamic reward function based on whether the state has terminated, the time consumption of the action, the comparison between the current scheduling result and the historical best result, and the difference between the current scheduling result and the target value. The quality assessment value Q update module is used to calculate immediate rewards based on a dynamic reward function. Furthermore, a high-dimensional discount factor and a dynamic exploration rate decay strategy are introduced to update the state-action quality assessment value; The Q-table generation module is used to generate a Q-table to record the quality assessment values ​​Q corresponding to each state-action after the state-action quality assessment values ​​Q converge. The initial scheduling solution generation module is used to sort and expand the candidate transition actions according to the Q-table to obtain the initial scheduling solution set; The global optimization module is used to input the initial scheduling solution set into the evolutionary algorithm, perform a second global optimization with the goal of minimizing the global completion time, and output the globally optimal scheduling path.

10. The system according to claim 9, characterized in that, The quality assessment value Q update module includes: Target evaluation value The calculation submodule is used to obtain the instant reward value based on the dynamic reward function. And based on whether the next state is a termination state, the target evaluation value is calculated in conjunction with the high-dimensional discount factor. ; The quality assessment value Q update submodule is used to update the target assessment value. and learning rate Update the quality assessment value Q of the current state-action pair; The action selection submodule is used to select actions based on the exploration rate. Determine whether to implement an exploration mechanism or a utilization mechanism, and select a corresponding transition action; Exploration rate The decay submodule is used to adjust the exploration rate after completing a round of state-action quality assessment value Q update. Perform decay updates to reduce the probability of random exploration in later stages.

Images