A dynamic scheduling method and system based on dual-domain trust optimization

Abstract

The application relates to the technical field of scheduling optimization, and discloses a dynamic scheduling method and system based on double-domain trust optimization, which comprises the following steps: training an initial strategy network, calculating an initial discount occupancy measure, generating a comprehensive state vector, a mixed action vector, a multi-target immediate reward signal, a process and resource constraint set and a working condition context vector; based on the estimated discount occupancy measure and the calculated OAD value, the behavior drift amount is detected and the strategy update step is scaled; a double-domain collaborative optimization objective function is constructed and the constraint weight is dynamically adjusted, the strategy parameters are updated to generate a candidate strategy, and the return increment and the risk change amount are calculated; the behavior trust domain radius and the double-domain constraint coefficient are adjusted; a dynamic feasible region is constructed to map illegal actions into legal actions, and the projection deviation is calculated; the performance of the new strategy is verified, and a scheduling instruction is output to a wafer manufacturing line. The method can improve the efficiency of semiconductor wafer manufacturing while improving safety and stability.

Description

Technical Field

[0001] This invention relates to the field of scheduling optimization technology, and in particular to a dynamic scheduling method and system based on dual-domain trust optimization. Background Technology

[0002] As semiconductor manufacturing processes evolve towards the nanometer scale, front-end wafer fabrication lines have become one of the most complex discrete manufacturing systems in the industrial field. Their production processes typically involve over 600 highly intertwined steps, encompassing various heterogeneous equipment groups such as photolithography, etching, and deposition. In actual production, due to frequent order insertions, random equipment failures, and tight coupling between processes, the production environment exhibits significant non-stationarity. Currently, wafer fab scheduling mainly relies on rule-based heuristic methods or traditional mathematical programming methods. However, heuristic methods struggle to obtain optimal solutions in multi-objective balancing, while mathematical programming methods suffer from excessive computational complexity when facing dynamically changing, ultra-large-scale scenarios, making true real-time scheduling difficult. In recent years, scheduling methods based on deep reinforcement learning have gradually emerged. These methods generate policy networks by learning from historical trajectories, demonstrating great potential in terms of real-time performance and decision quality. Existing reinforcement learning scheduling methods based on Proximal Policy Optimization (PPO) or Trust Region Policy Optimization (TRPO) mainly constrain policy parameter updates through Kullback-Leibler divergence (KL). However, these methods only focus on changes in the parameter space and cannot directly constrain the drift of the policy's behavioral distribution in the state-action space. When production line conditions change abruptly, even small parameter changes can lead to significant shifts in behavioral patterns. Existing reinforcement learning-based dynamic scheduling techniques still face the following key challenges in practical deployment: **Instability of Multi-Objective Advantage Signals:** When considering multiple objectives such as throughput, energy consumption, and yield, the reward functions for different objectives have vastly different dimensions and are affected by fluctuations in operating conditions. This results in high noise and inconsistent values ​​in the original advantage signals during training, making it difficult for the strategy to converge stably. **Behavioral Blind Zones in Non-Stationary Environments:** Traditional methods primarily constrain the update magnitude of policy parameters. However, during abrupt changes in operating conditions, even small hyperparameter changes can cause drastic shifts in the policy's behavior distribution in the state-space, leading to unverified behavioral blind zones and posing risks to production line operations. **Imbalance Between Performance Improvement and Physical Reliability:** Existing algorithms often excessively pursue reward maximization, lacking a measure of the reliability of policy updates and a dynamic correction mechanism for physical hard constraints. This can lead to a situation where improving efficiency may sacrifice system security and stability. Summary of the Invention

[0003] This invention provides a dynamic scheduling method and system based on dual-domain trust optimization to solve the problem that efficiency and stability in semiconductor wafer manufacturing cannot coexist in the existing wafer fab production scheduling process.

[0004] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, the present invention provides a dynamic scheduling method based on dual-domain trust optimization, comprising: S1. Collect equipment status data, work-in-process and order data, process and resource constraint data, and historical performance data of the semiconductor wafer manufacturing system. Generate multiple historical scheduling trajectories using the earliest due date priority rule. Based on all historical scheduling trajectories, train the initial policy network using the behavior cloning method, calculate the initial discount occupancy measure, and generate a fixed-dimensional comprehensive state vector, a hybrid action vector containing discrete batch-equipment assignment and continuous power adjustment, a multi-objective instant reward signal, a process and resource constraint set, and a working condition context vector. The historical scheduling trajectory includes the system's actions at each time point in the historical operation. The initial discount occupancy measure is used to describe the access probability distribution of each state-action pair in the initial policy network. The multi-objective instant reward signal is used to describe the reward for action execution. The process and resource constraint set is used to describe the constraints of dynamic system scheduling at the process and resource level. The working condition context vector is used to describe the current system state. S2. Receive the multi-objective real-time reward signal, comprehensive state vector, historical performance data, and historical behavioral drift output from step S1. Perform context-aware normalization in the target value space and introduce a risk-drift joint calibration factor in the behavioral distribution space to generate a normalized comprehensive advantage signal. The behavioral drift is used to describe the degree of deviation in the distribution of scheduling actions, the risk-drift joint calibration factor is used to quantify the comprehensive suppression strength of production line risk and strategy behavioral drift, and the comprehensive advantage signal is used to describe the quality of the actions. S3. Receive the initial discount occupancy measure of the old strategy output in step S1 and the comprehensive advantage signal output in step S2. Estimate the discount occupancy measure of the temporary candidate strategy based on importance sampling. Calculate the OAD value between the old and new strategies. Detect the behavior drift and adaptively scale the strategy update step size. The OAD value is used to quantify the distribution difference of the discounted occupancy measure between the old and new strategies. S4. Receive the policy update step size and historical dual-domain constraint coefficients output from step S3. Combine the geometric trust domain natural gradient and behavioral trust domain OAD constraints based on the K-FAC approximation to construct a dual-domain collaborative optimization objective function. Detect parameter-behavioral cross-sensitivity and dynamically adjust constraint weights. Update policy parameters to generate candidate policies and calculate reward increment and risk change. The dual-domain collaborative optimization objective function describes the expected reward in the geometric and behavioral trust domains. The reward increment describes the increase in return due to dynamically adjusting constraint weights, and the risk change describes the decrease in return due to dynamically adjusting constraint weights. S5. Receive the working context vector output from step S1, the reward increment and risk change output from step S4, and the behavior drift output from step S3. Select the OAD operator type according to the adaptive selection mechanism, and dynamically adjust the behavior trust domain radius and dual-domain constraint coefficient. The OAD operator is used to calculate the OAD value. S6. Receive the candidate strategy output from step S4 and the set of process and resource constraints output from step S1, construct the dynamic feasible region, and use a two-stage decomposition projection method to map illegal actions to legal actions, and calculate the projection deviation; the projection deviation is used to describe the severity of the action violating the constraints. S7. Receive the projection deviation output from step S6, the reward increment and update statistics output from step S4, calculate the comprehensive risk score, execute a soft rollback or hard rollback mechanism based on the comprehensive risk score, verify the performance of the new strategy through canary release, and output the final safe scheduling instruction to the wafer manufacturing line; the update statistics include the strategy update step size when updating the strategy in step S4, and the final safe scheduling instruction is used to instruct the actions of each device in the wafer manufacturing line at future moments.

[0005] Secondly, the present invention also provides a dynamic scheduling system based on dual-domain trust optimization, including a processor and a memory; Memory, used to store computer programs; The processor, when executing a program stored in memory, implements the method steps of the first aspect.

[0006] Beneficial effects: This invention provides a dynamic scheduling method based on dual-domain trust optimization. It eliminates dimensional differences and scale inconsistencies among multiple objectives such as throughput, energy consumption, and latency by performing context-aware normalization on the original advantage functions of different performance objectives in the target value space. Simultaneously, it introduces behavioral occupancy alignment divergence (OAD) drift and tail risk index (CVaR) into the behavioral distribution space to construct a risk-drift joint calibration factor. Compared to traditional static normalization or fixed-weight methods, this invention achieves dynamic closed-loop coupling of advantage scale, target weights, and risk constraints. The output unified advantage signal possesses both multi-objective balance and risk perception capabilities, effectively eliminating the interference of environmental noise and distribution drift on policy gradients under non-stationary conditions, ensuring numerical stability during training. Furthermore, it constructs a full-link security barrier from policy output to physical execution. A two-stage decompositional projection method is employed to forcibly correct illegal assignments and excessive power in candidate actions to within the feasible region that satisfies the process matrix, capacity limit, and power range, while minimizing deviations from the original decision intent. This achieves a decoupled architecture of "soft policy guidance and hard constraint projection." A comprehensive risk score based on multi-dimensional risk coupling is introduced to monitor behavioral drift, tail risk changes, and performance decline in real time. When risks exceed limits, soft / hard rollbacks are automatically executed, and A / B testing is conducted through a canary release mechanism for verification. This dual-layer protection mechanism ensures that while pursuing throughput improvement, the strategy always operates within the system's acceptable safety boundaries. It effectively avoids production line operation risks caused by overexploration or sudden environmental changes in traditional reinforcement learning methods. By considering the multi-objective balance of system scheduling, it ensures consistency in behavioral distribution and physical safety constraints, thereby improving both the efficiency and safety and stability of semiconductor wafer manufacturing. Attached Figure Description

[0007] Figure 1 This is a flowchart of a dynamic scheduling method based on dual-domain trust optimization, which is a preferred embodiment of the present invention. Detailed Implementation

[0008] The technical solution of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0009] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms "an" or "a" and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms "connected" or "linked" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up," "down," "left," "right," etc., are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship also changes accordingly.

[0010] Please see Figure 1 This application provides a dynamic scheduling method based on dual-domain trust optimization, comprising: S1. Collect equipment status data, work-in-process and order data, process and resource constraint data, and historical performance data of the semiconductor wafer manufacturing system. Generate multiple historical scheduling trajectories using the earliest due date priority rule. Based on all historical scheduling trajectories, train the initial policy network using the behavior cloning method, calculate the initial discount occupancy measure, and generate a fixed-dimensional comprehensive state vector, a hybrid action vector containing discrete batch-equipment assignment and continuous power adjustment, a multi-objective instant reward signal, a process and resource constraint set, and a working condition context vector. The historical scheduling trajectory includes the system's actions at each time point in the historical operation. The initial discount occupancy measure is used to describe the access probability distribution of each state-action pair in the initial policy network. The multi-objective instant reward signal is used to describe the reward for action execution. The process and resource constraint set is used to describe the constraints of dynamic system scheduling at the process and resource level. The working condition context vector is used to describe the current system state. S2. Receive the multi-objective real-time reward signal, comprehensive state vector, historical performance data, and historical behavioral drift output from step S1. Perform context-aware normalization in the target value space and introduce a risk-drift joint calibration factor in the behavioral distribution space to generate a normalized comprehensive advantage signal. The behavioral drift is used to describe the degree of deviation in the distribution of scheduling actions, the risk-drift joint calibration factor is used to quantify the comprehensive suppression strength of production line risk and strategy behavioral drift, and the comprehensive advantage signal is used to describe the quality of the actions. If the semiconductor wafer manufacturing system has been scheduled at a previous historical moment, the behavior drift amount at that historical moment is obtained; otherwise, the behavior drift amount at that historical moment is a preset value.

[0011] S3. Receive the initial discount occupancy measure of the old strategy output from step S1 and the comprehensive advantage signal output from step S2. Estimate the discount occupancy measure of the temporary candidate strategy based on importance sampling. Calculate the behavior occupancy alignment divergence (OAD) value between the old and new strategies. Detect the behavior drift and adaptively scale the strategy update step size. The OAD value is used to quantify the distribution difference of the discounted occupancy measure between the old and new strategies. S4. Receive the policy update step size and historical dual-domain constraint coefficients output from step S3. Combine the geometric trust domain natural gradient and behavioral trust domain OAD constraints based on the K-FAC approximation to construct a dual-domain collaborative optimization objective function. Detect parameter-behavioral cross-sensitivity and dynamically adjust constraint weights. Update policy parameters to generate candidate policies and calculate reward increment and risk change. The dual-domain collaborative optimization objective function describes the expected reward in the geometric and behavioral trust domains. The reward increment describes the increase in return due to dynamically adjusting constraint weights, and the risk change describes the decrease in return due to dynamically adjusting constraint weights. If the bidomain constraint coefficients have been adjusted at the current historical time, then the adjusted bidomain constraint coefficients at the historical time are obtained; otherwise, the historical bidomain constraint coefficients are the preset values.

[0012] S5. Receive the working context vector output from step S1, the reward increment and risk change output from step S4, and the behavior drift output from step S3. Select the OAD operator type according to the adaptive selection mechanism, and dynamically adjust the behavior trust domain radius and dual-domain constraint coefficient. The OAD operator is used to calculate the OAD value. S6. Receive the candidate strategy output from step S4 and the set of process and resource constraints output from step S1, construct the dynamic feasible region, and use a two-stage decomposition projection method to map illegal actions to legal actions, and calculate the projection deviation; the projection deviation is used to describe the severity of the action violating the constraints. S7. Receive the projection deviation output from step S6, the reward increment and update statistics output from step S4, calculate the comprehensive risk score, execute a soft rollback or hard rollback mechanism based on the comprehensive risk score, verify the performance of the new strategy through canary release, and output the final safe scheduling instruction to the wafer manufacturing line; the update statistics include the strategy update step size when updating the strategy in step S4, and the final safe scheduling instruction is used to instruct the actions of each device in the wafer manufacturing line at future moments.

[0013] The future moment is the moment after the current moment. The actions of the equipment include discrete actions (decision-making for batch assignment to the equipment) and continuous actions (decision-making for resource allocation of the equipment).

[0014] In this embodiment, a complete seven-step closed-loop method is defined, encompassing data acquisition, advantage calibration, behavioral trust domain constraints, dual-domain collaborative optimization, divergence scheduling, feasible domain projection, and safe rollback. This technical solution systematically addresses the three core issues raised in the background technology: First, step S2 generates a unified advantage signal by normalizing the target value space and calibrating the behavioral distribution space risk, eliminating the dimensional differences and environmental noise of multi-target rewards and resolving the problem of "instability of multi-target advantage signals." Second, steps S3 and S4 introduce behavioral occupancy alignment divergence (OAD) based on discounted occupancy measure and combine it with the geometric trust domain to simultaneously constrain behavioral distribution drift and parameter update amplitude during policy updates, effectively avoiding "behavioral blind spots in non-stationary environments." Third, steps S6 and S7 embed physical hard constraints and risk monitoring into the decision-making chain through feasible domain projection and safe rollback mechanisms, resolving the problem of "imbalance between performance improvement and physical reliability." This closed-loop architecture enables reinforcement learning strategies to evolve in a "small-step, controllable, and reliable" manner under non-stationary operating conditions in semiconductor wafer fabs (equipment failures, emergency order insertions, etc.), significantly improving the stability, security, and online scheduling reliability of the scheduling system.

[0015] In some embodiments of this application, in step S1, equipment status data, work-in-process and order data, process and resource constraint data, and historical performance data of the semiconductor wafer manufacturing system can be collected through sensors and other devices. The equipment status data includes the operating status and utilization rate of equipment such as lithography and etching equipment in the semiconductor wafer manufacturing system. The work-in-process and order data includes data on the products currently being manufactured by the equipment in the system and the order quantity. The process and resource constraint data includes data on equipment power constraints, energy consumption limits, and safe temperature ranges. The historical performance data includes the throughput, energy cost, late delivery rate, and tail risk value of the equipment during historical operation.

[0016] Specifically, (1) Equipment status data Used to describe the real-time operating state of devices, it is the core input for constructing state vectors and feasible region constraints. Define the set of devices:

[0017] Where: m = number of devices. For any device... Define state variables Define state variables for each device: ① Equipment number , indicating the first A unique identifier for each device, used to uniquely identify the device and index the device feature vector in status coding.

[0018] ② Processing stage, identified by process family here. To indicate, Based on the semiconductor front-end manufacturing (Fab) equipment family classification principles, this application divides the processing equipment involved in wafer manufacturing into nine typical process families according to their process functions. Therefore, the legality mask used for calculating the action space represents the current processing stage of the equipment. Among them: 1 is photolithography equipment, 2 is etching equipment, 3 is thin film deposition equipment, 4 is ion implantation equipment, 5 is thermal processing equipment, 6 is cleaning equipment, 7 is chemical mechanical polishing equipment, 8 is defect detection equipment, and 9 is metrology equipment.

[0019] Because the scheduling system manages equipment types rather than individual micro-processes, the above classification is based on the basic physical processing behaviors of wafer manufacturing (material formation, pattern definition, material removal, electrical property adjustment, and quality control), which can comprehensively cover the main equipment types in front-end manufacturing.

[0020] Although advanced logic chip manufacturing may involve 600 to 2000 specific processing steps, these steps can usually be mapped to the limited number of device process families mentioned above. Therefore, this classification can significantly reduce state space complexity and improve reinforcement learning training stability while ensuring the integrity of scheduling modeling.

[0021] ③ Equipment operating status :

[0022] 0 represents idle, 1 represents running, and 2 represents fault.

[0023] Equipment utilization rate for equipment status characteristics calculate:

[0024] Its function is as a counter switch. At a certain moment, if the device... The state is running, and the function value is 1. If the device... The function value is 0 when the state is idle or faulty. The summation symbol ∑ is used to set the time window. The numerator is the sum of the function values ​​at all sampling times. Overall, it represents the total duration (or number of sampling points) the device was in operation.

[0025] ④ Equipment availability variables :

[0026] The set of optional devices in the action space, used for constructing the set of optional devices in the action space.

[0027] ⑤ Equipment power This indicates the device's real-time power.

[0028] Define power range .

[0029] ⑥ Equipment energy consumption : The cumulative energy consumption within the time window ΔT is used for reward function calculation:

[0030] Energy cost used in the reward function: .

[0031] ⑧ Fault History Record Record the historical failure times and repair times of the equipment to calculate reliability metrics such as mean time between failures (MTBF), and use them as contextual features in status coding.

[0032] The device state vector is defined as:

[0033] Full device status matrix: , used for status coding.

[0034] (2) Work-in-process and order data, which describes the production task structure and is the core basis for action decision-making, while also affecting reward calculation and risk modeling.

[0035] Define the WIP set:

[0036] For batch quantity, for batch Define the following variables: Batch ID is (Unique identifier) ​​used for trajectory construction.

[0037] Batch quantity This indicates the number of wafers, used for reward calculations and status statistics.

[0038] Process Path , which represents the processing path sequence.

[0039] The current process is This indicates the current stage of the process.

[0040] Standard working hours Indicates batch In the process The standard processing time is used for time estimation in status coding.

[0041] Time attribute: Arrival time is Delivery time is relaxation time .

[0042] Order Attributes: Order Priority is Order type (Including planned orders and urgent orders, used to calculate the order insertion rate), the remaining number of processes is: .

[0043] Batch feature vector Defined as:

[0044] (3) Process and resource constraint data, which is used to define the action feasible domain Feasible(a) and is the core input of the projection module.

[0045] Define the following data variable: process sequence matrix ,in . Indicates batch Available on device Processing. Machine Capability Matrix , indicating equipment Maximum number of parallel batches. Batch parallelism constraint. This indicates the number of processes allowed to be processed in parallel. Power constraint. Energy consumption limit Temperature safety range Constraint set definition .

[0046] (4) Performance and Risk Indicators, which are used for reward calculation and risk control.

[0047] definition: Throughput This represents the output per unit time:

[0048] Energy consumption cost , representing the current total energy consumption, and .

[0049] Late delivery rate This indicates the percentage of currently delayed batches:

[0050] Tail risk is , representing the conditional value of risk calculated based on the historical return distribution.

[0051] Risk vector is .

[0052] After obtaining the above data, preprocessing such as cleaning missing values ​​and standardization can be performed on the data performance. Then, based on this data, multiple historical scheduling trajectories are generated according to the earliest expiration priority rule (this trajectory includes data of the device at multiple time points and is sorted in chronological order from earliest to latest). Based on all historical scheduling trajectories, the behavior cloning method is used to train the initial policy network: a rule-based heuristic initial policy is defined, and the earliest expiration priority rule is selected. ) rules, for state Its action decision logic is expressed as:

[0053] Prioritize the allocation of delivery dates The tightest batches are sent to compatible equipment, while also incorporating a process sequence matrix. A compatibility check is performed, and priority-based power regulation logic is set for resource allocation. Next, an initial trajectory is generated by sampling from historical scheduling records. A rule-based generation method is used to apply rules to each state in the sampled trajectory to generate a new action, while retaining historical rewards and the next state, forming a new trajectory list.

[0054] Then, a parameterized initial policy model is fitted, and the behavior cloning method is used to train a simple neural network model by using the state-action pairs (s, a) in the trajectory as supervision data. To fit rule-based behavior and predict actions for a given state, the optimization objective is to minimize the negative log-likelihood loss.

[0055] Training was performed for 100 epochs using cross-entropy loss and the Adam optimizer.

[0056] Finally, the discount occupancy measure is calculated using the initial trajectory. Based on the sampled trajectory, an empirical averaging method is used for estimation:

[0057] in, Let be the indicator function. Discount-weighted frequencies are statistically analyzed from the trajectories and normalized using hash states and actions to obtain the probability distribution of the policy in the state-action space.

[0058] Preferably, in step S1, the formula for calculating the initial discount occupancy measure is: ; In the formula, As a measure of initial discount usage, For state-action pairs in the trajectory. For state, For action, For the time in the historical scheduling trajectory The following state, For the time in the historical scheduling trajectory The following action, As a discount factor, Let D be the indicator function, D be the set of historical scheduling trajectories, and N be the number of trajectories. For historical scheduling trajectory, Historical scheduling trajectory The length of , in the indicator function, when the ... Access status at each time step And perform the action The value is 1 if the action is active and 0 otherwise. By statistically analyzing the discount-weighted access frequency of each state-action pair across all sampled trajectories and normalizing the data, the discount occupancy measure of the policy in the state-action space is estimated.

[0059] The formula for calculating multi-objective instant reward signals is: ; In the formula, The value of the multi-objective instant reward signal. This indicates an encouragement strategy to improve production efficiency. This indicates a limitation on energy consumption. Indicates the risk of late delivery. Let C represent the constraint set.

[0060] In this embodiment, the discount occupancy measurement formula uses a discount factor γ to weight historical trajectories, enabling the measurement to reflect the long-term behavioral distribution of the strategy, rather than focusing solely on the instantaneous state, thus providing a time-extended behavioral baseline for subsequent OAD calculations. The strategy, updated every 100 steps, ensures a balance between computational efficiency and distribution tracking accuracy. In the multi-objective reward formula, four factors work together: positive incentive for throughput, negative penalties for energy consumption and latency risks, and penalties for constraint violations. Furthermore, each indicator is standardized through historical distribution, ensuring the comparability of reward signals even under non-stationary operating conditions. This formula allows the strategy optimization process to simultaneously consider production efficiency, energy conservation and consumption reduction, on-time delivery, and safety compliance, avoiding system imbalances caused by a single dominant objective.

[0061] Preferably, in step S1, the comprehensive state vector is generated by concatenating equipment-side features, work-in-process (WIP)-side features, and time context features. The equipment-side features include equipment utilization, real-time power, one-hot encoding of operating status, and mean time between failures (MTBF). The WIP-side features include global statistical features and Top-K key batch features sorted by priority × 1 / delivery date. The time context features include the mean, variance, and trend slope within the sliding window. Historical performance data includes throughput, energy costs, late delivery rate, and tail risk value.

[0062] In this embodiment, utilization, real-time power, one-hot encoding of operating status, and mean time between failures (MTBF) in the equipment-side features jointly characterize the physical state and reliability of the production line. The global statistical features (mean, range, histogram) in the work-in-process features are combined with Top-K key batch features to capture both the global load distribution and focus on the most urgent tasks, while a fixed K value ensures a constant input dimension for the neural network. The mean, variance, and trend slope in the time context features transform non-stationary time series into fixed-dimensional trend features, enabling the model to perceive the direction and intensity of changes in operating conditions. This multi-dimensional heterogeneous feature fusion method allows the policy network to extract stable and distinguishable state representations from high-dimensional, dynamic, and non-stationary industrial data, significantly improving the policy's perception and generalization capabilities in complex environments.

[0063] For example, the integrated state vector is:

[0064] in, For time context vectors, This is the device state vector. This is a global statistical vector. Micro-batch characteristics:

[0065] ① Equipment-side feature encoding: Based on device state vector Extract key features: Operational characteristics: utilization rate Real-time power .

[0066] Status coding: for running status Perform one-hot encoding and mark fault flag bits. ; Reliability characteristics: based on failure history Calculate the mean time between failures. As a contextual feature:

[0067] Offline processing and masking: If a device is offline or in a maintenance state, its corresponding feature bit is filled with 0. At the same time, a device mask vector is introduced , when the device is online, =1, and when it is offline . This mask is used in the attention mechanism to shield the noise interference of offline devices:

[0068] ② WIP / order side feature encoding: Global statistical features: Calculate the macro indicators of the current workshop to construct a vector , the specific operations are as follows: Numerical statistics: Calculate the mean and range of continuous variables to capture the overall load and fluctuation:

[0069] Perform binning statistics on discrete or interval variables to capture the distribution pattern. Divide the slack time into L intervals , and count the frequency of batches falling into each interval:

[0070] Similarly, count the WIP distribution histogram of each process section .

[0071] 3. Proportion statistics: Calculate the overall late delivery rate and the proportion of urgent orders:

[0072] Finally, the global statistical vector is the concatenation of the above features:

[0073] Top-K key object sampling: In order to capture the details of the most urgent tasks, the system sorts according to the order priority and due time , and the sorting criteria are as follows:

[0074] Extract the top K key batches. For each selected batch j, construct a feature vector, where the single batch feature vector is: .

[0075] Top-K feature matrix:

[0076] If the actual number of WIP n < K, zero-padding is performed on the missing bits.

[0077]

[0078] If n ≥ K, only the top K batches with the highest priority are retained; information for the remaining batches is implicitly covered by "global statistical features". The output dimension is fixed. .

[0079] ③Time context features: Sliding window statistics are used for non-stationary signals to transform time series information into trend features with fixed dimensions.

[0080] Mean characteristics (reflecting the current level):

[0081] Variance characteristics (reflecting the intensity of fluctuations):

[0082] Trend slope (reflecting the direction of change): The slope of the data within the window fitted by linear regression.

[0083] Finally, the time context vector is defined as:

[0084] It should be noted that the dimension of the integrated state vector is... .

[0085] in, The WIP / order side feature dimension consists of two parts: the first part is the global statistical features, let the global statistical vector dimension be: the second part is the Top-K key batch features, let the feature dimension of a single batch be: selecting K, then ,therefore , Because the mean, variance, and trend slope are all one-dimensional. For example, the composite state vector also includes a context-dependent mean and a context-dependent standard deviation. The context-dependent mean is the average value of the production environment calculated using a sliding window (Size=K) or an exponential moving average (EMA), reflecting the average performance level under the current operating conditions.

[0086] It should be noted that the hybrid action vector includes discrete actions (decision-making for batch assignment to devices) and continuous actions (decision-making for device resource allocation), expressed as: Discrete actions This represents the assignment decisions from n batches to m devices, denoted by the assignment matrix. , Its legality is limited by the process sequence matrix defined above. : Continuous action This represents the resource allocation for m devices. The adjustment range is limited by the power range defined above. Continuous actions are normalized and mapped to [ The interval is 1,1. Discrete actions are encoded using integer mapping, while continuous actions are normalized and mapped to the interval [-1, 1]. The validity of an action is constrained by the process matrix and the upper limit of production capacity, and is verified through the feasible region projection module.

[0087] The set of process and resource constraints includes the process sequence matrix. (Define batch path dependencies), machine capability matrix (Capacity limit, parallelism), power range[ , Static or dynamic constraints, such as […].

[0088] The operating context vector is a vector composed of environmental data from the semiconductor wafer manufacturing system.

[0089] in, This represents the variance of the relaxation time (i.e., the intensity of the work-in-process fluctuation). Indicates the average utilization rate of the equipment; Indicates the proportion of urgent orders submitted; It represents the conditional risk value (the tail risk level of the current strategy).

[0090] Preferably, step S2 specifically includes: S21. Advantage function calculation: Calculate the advantage function value of each objective based on the multi-objective real-time reward signal and the state value function estimation results.

[0091] Specifically, the advantage function A(s, a) is defined as the difference between the action-value function Q(s, a) and the state-value function V(s), i.e., A(s, a) = Q(s, a) - V(s), where: Q(s, a) represents the expected cumulative reward (i.e., the cumulative value of multi-objective immediate reward signals) that can be obtained by following the policy after performing action a in state s. V(s) represents the expected cumulative reward that can be obtained by following the policy in state s (without specifying a particular action). A positive value of the advantage function indicates that the action is better than the average (a negative value indicates that it is worse than the average).

[0092] S22. Target Value Space Normalization: Using a sliding window, the advantage function values ​​of each target are calibrated based on the mean and standard deviation of context dependencies in the integrated state vector. The calculation formula is as follows: ; In the formula, Indicates the first The calibrated advantage function value of each target. Indicates the first The original advantage function values ​​for each objective. Indicates the first The context-dependent mean of each objective, calculated using a sliding window (Size=K) or exponential moving average (EMA), represents the average performance level under current operating conditions. Indicates the first The context-dependent standard deviation of each target reflects the scale of fluctuation in the dominance value, if the value of the "energy consumption" target fluctuates greatly (e.g., from 0 to 10,000). This will increase, thus reducing its advantage value and preventing gradient dominance; if the numerical fluctuation of the "delay" target is small (e.g., from 0.1 to 0.5). It is very small, thus amplifying its advantage value.

[0093] S23. Behavioral Distribution Spatial Risk Calibration: A risk-drift joint calibration factor is calculated based on the tail risk value and behavioral drift, introducing a projection bias correction term. The calculation formula is as follows: ; In the formula, For the risk-drift joint calibration factor, Indicates the amount of behavioral drift. Indicates the tail risk value. , The sensitivity (initial values ​​are set to 0.5 and 0.3 respectively) reflects the system's sensitivity to distribution drift and tail risk.

[0094] This mechanism achieves adaptive nonlinear mapping: when the environment is in a high-risk region (increased CVaR) or the operating conditions change abruptly (increased OAD), It automatically reduces and suppresses the intensity of abnormal fluctuations; under stable operating conditions... Approaching 1, the signal gain is preserved to accelerate convergence.

[0095] CVaR stands for Tail Risk Indicator, which measures extreme losses. (Distribution drift): If the strategy changes too much in this round, leading to unpredictable behavior, Increase. CVaR (Tail Risk): If the current action sequence may lead to extreme delay risk, CVaR increases.

[0096] S24. Unified Driving Signal Generation: Combining context-aware target weights, weighted fusion is used to generate a normalized comprehensive advantage signal. ; In the formula, To output the normalized comprehensive advantage signal, Indicates the first Context-aware weights for each target.

[0097] In this embodiment, target value space normalization uses a sliding window to statistically analyze the context-dependent mean and standard deviation of the advantages of each target, eliminating dimensional differences and scale inconsistencies between different targets (throughput, energy consumption, latency). This ensures that policy updates are not dominated by the gradient direction due to the excessively large numerical range of a single target. When the behavior drift... Increased or tail risk When it rises, Automatically reduces the strength of abnormal advantages, thereby preventing aggressive policy updates in risky scenarios. Unified advantage signal. The normalized multi-objective advantages are integrated with the risk calibration factor, and context-aware weights are introduced. This mechanism achieves dynamic coupling of the dominant scale, target weights, and risk constraints. It enables rapid convergence of policy updates under stable conditions and automatic conservatism under risky scenarios, significantly improving the numerical stability and safety of training.

[0098] For example, a projection bias calibration term can be introduced in this step. :

[0099] This deviation calibration item Feedback from the previous execution indicates that if a certain action is frequently and significantly corrected by the projection module (indicating that the action seriously violates physical constraints), then... This increases the advantage strength of the action and automatically suppresses it. This mechanism ensures the consistency between the policy gradient update direction and the direction of the physically feasible region.

[0100] Preferably, in step S3, the OAD value is selected from the slice Wasserstein distance (SWD) or the maximum mean difference (MMD). The formula for calculating SWD is: ; The formula for calculating MMD is: ; In the formula, This represents the discount occupancy measure of the old strategy. Discount occupancy measurement with the new strategy The Wasserstein distance between the slices express and The maximum mean difference between them This represents the integral from 0 to 1. Represents the square of the absolute value. It represents the inverse function of the cumulative distribution function of the old distribution. The cumulative distribution function of the new distribution is the inverse function, and dp represents the integration variable. In the regenerating nucleus Hilbert space Norms in This represents the kernel mean embedding of the old distribution. This represents the kernel mean embedding of the new distribution. Let x represent the kernel function and x represent the sample point. When detected At that time, the step size contraction mechanism is triggered, and the updated formula is:

[0101] In the formula, η∈(0,1) is the step size scaling factor. For the updated policy parameters, This represents the original policy parameters. These are the candidate strategy parameters.

[0102] Specifically, the calculation of OAD occurs during the internal iteration of policy optimization, rather than after the policy update is complete. For each temporary candidate policy... BTR evaluates the behavior distribution of each policy. It should be noted that the temporary candidate policy has not yet been implemented in a real production environment, and its discounted occupancy measure cannot be directly obtained from the actual trajectory. Therefore, in one embodiment, the present invention uses an empirical approximation method based on the trajectory of the old policy to estimate the occupancy measure of the temporary candidate policy, specifically including but not limited to the following methods: Based on the old strategy sampling trajectory, the strategy probability ratio is used. Perform importance-weighted estimation; Alternatively, based on the policy network output, a reweighted approximation can be performed on the historical state-action samples.

[0103] Therefore, the occupancy measure of temporary candidate strategies can be obtained without real-world interaction. A feasible approximation is used for OAD calculation.

[0104] In each internal iteration of strategy optimization, the specific steps of BTR are executed according to the following process: 1. Obtain the occupancy metric samples corresponding to the old strategy. ; 2. Receive unified drive signal We weight the state-action samples to focus on behavior drift calculations in high-value areas; 3. In each iteration of policy optimization, estimate the occupancy measure of the current temporary candidate policy based on an approximation method. ; 4. Based on the divergence operator type and parameters from the previous iteration, calculate the OAD value for the current iteration: .in Indicates the current iteration The OAD value at that location is used for real-time constraints. This represents a metric operator (such as SWD or MMD). This represents the occupancy measure of the old strategy. This represents the occupancy measure of a temporary strategy.

[0105] When detected: ,but( = ( =0.8), scaling update step size .

[0106] in When the behavior drift threshold is preset or adaptive, BTR triggers an update strength suppression mechanism to scale the current policy update step size:

[0107] in ∈(0,1) is the step scaling factor. The step scaling mechanism is equivalent to dynamically amplifying the weight of the behavior domain penalty term in the joint optimization objective, thereby achieving a soft constraint on excessive behavior drift from an optimization perspective, rather than a simple empirical pruning rule.

[0108] The constrained update step size and constraint signal are synchronously transmitted to S4 and work in conjunction with the geometric trust domain constraint.

[0109] Behavioral drift The calculation formula is:

[0110] in and These are the discount occupancy measures for the old and new strategies estimated above, respectively; Ψ This is a defined occupancy metric divergence operator. This increment is used to measure whether policy updates introduce excessive behavioral distribution shifts.

[0111] The adaptive scaling strategy updates the step size using the step size scaling factor. .

[0112] In the above embodiments, SWD captures the global geometric drift of the behavior distribution through slice averaging, making it sensitive to the overall shift of the distribution; MMD captures local statistical feature differences through kernel mean embedding, making it sensitive to changes in the shape of the distribution. The complementarity of these two methods allows the system to select the most suitable metric under different operating conditions. When OAD_t exceeds the threshold δ=0.05, the step size scaling factor η decreases sequentially by a shrinkage rate of 0.8, achieving adaptive suppression of "the larger the behavior drift, the smaller the update step size." This mechanism ensures that the strategy does not deviate from the historically validated high-value behavior region in each optimization iteration, fundamentally solving the nonlinear instability problem of "small parameter changes, large behavior changes" in traditional methods, and significantly improving the update reliability and behavior consistency of the strategy in non-stationary environments.

[0113] Preferably, in step S4, the natural gradient of the geometric trust domain is calculated using the K-FAC approximation of the Fisher information matrix, as follows: ; In the formula, This indicates the parameter update increment, and ≈ represents the approximation sign. Represents the inverse of the Fisher information matrix. Describe the objective function Relative to parameters gradient, Indicates the strategy parameters, Indicates the parameterization strategy, This represents the objective function for policy optimization. The objective function for dual-domain collaborative optimization is: ; In the formula, This represents the total loss function for bi-domain collaborative optimization. Represents the mathematical expectation symbol. This indicates the current strategy to be optimized. These are the learnable parameters of a neural network. Indicates the old strategy, Indicates the importance sampling weight, This indicates a normalized overall advantage signal. Represents the behavioral domain constraint coefficients. This represents the occupancy measure divergence operator. This represents the discount occupancy measure of the old strategy. This represents the discount occupancy measure of the current strategy. Represents the geometric domain constraint coefficients. Indicates the Kullback-Leibler divergence. This represents the expectation of the above KL divergence under the state access distribution of the old strategy; Calculation parameter - Behavioral cross-sensitivity factor: ; When the parameter - behavior cross sensitivity factor Nonlinearly increasing the behavioral domain constraint coefficient ,in For the preset threshold, It is a constant.

[0114] Calculate the incremental return With risk change :

[0115] in, The expected cumulative reward defined in the objective function for dual-domain collaborative optimization. and These are the updated new policy parameters and the old policy parameters, respectively. They are used to measure whether the OAD type used in the previous policy update truly improved scheduling efficiency.

[0116]

[0117] CVaR is a defined tail risk metric used to penalize OAD instances that worsen tail risk while increasing returns.

[0118] It should be noted that the Geometric Trust Region (GTR) is used to constrain the update magnitude of policy parameters on the parameter manifold, ensuring the geometric smoothness and numerical stability of the optimization process. The core function of the GTR is to suppress high-frequency oscillations and discontinuous jumps in the parameter space, ensuring that parameter updates proceed along the geometrically smoothest and most reliable direction, thus providing stability guarantees for policy optimization at the parameter level. The natural gradient constraint based on Fisher information in the GTR and the KL divergence constraint in the joint optimization objective (described later) have different levels of effect: the former constrains the geometric distance of parameter updates on the parameter manifold, while the latter restricts the probability distribution of the policy output. and The degree of offset is complementary rather than repetitive.

[0119] The Behavioral Trust Region (BTR) constrains the difference in behavioral distribution between old and new policies in the environment through the Behavioral Occupation Alignment Divergence (OAD). It focuses on the long-term access patterns of policies in the state-action space, i.e., "where the policy went." BTR limits the extent to which policy exploration deviates from the historical high-value behavioral subspace at the behavioral distribution level, providing behavioral consistency and risk control for policy updates.

[0120] In the dual-domain collaborative framework, BTR serves as a "behavioral-level navigation mechanism," based on the discount occupancy measure calculated in step one. and The OAD value is calculated using the defined divergence operator Ψ. This OAD value, along with the step size scaling factor, is then used to calculate the OAD value. η It is not only used to directly constrain the drift of behavior distribution, but also indirectly adjusts the upper limit of the effective step size of natural gradient updates in the geometric trust domain by influencing the weight of the behavior domain penalty term in the joint optimization objective. This allows the parameter update magnitude to adaptively shrink with behavior stability, thereby achieving synergy with GTR at the mechanism level.

[0121] BTR and GTR work together to form a dual-domain trust optimization structure (DLTO), simultaneously imposing constraints in both the behavior distribution domain and the parameter geometry domain. This prevents the failure of a single trust domain in the nonlinear policy space, i.e., small parameter changes that could lead to drastic behavior changes, or controlled behavior that could result in unstable parameter updates. This significantly improves the reliability and stability of policy updates. The collaboration between GTR and BTR is achieved through a unified joint optimization objective function that balances performance improvement, behavioral constraints, and parameter stability.

[0122] Specifically, in this step, the policy update performs the following process: Gradient calculation and geometric correction: Calculating the joint optimization objective The gradient is calculated, and natural gradient correction is performed based on the Fisher information matrix approximated by K-FAC to ensure the smoothness of the update direction on the parametric manifold. Parameter linkage update: based on dynamic coefficients and Solve the joint optimization objective to generate updated parameters. This step is the core of dual-domain collaboration, specifically the behavioral domain. and geometric domain Together, they determine the final placement of the parameters. (Executed after the first round); Output and Feedback: The output is a set of updated policy parameters. It defines a point in the space of candidate policy functions.

[0123] Output candidate strategy: The candidate strategy defined by the new parameters The feasible domain projection is then passed to subsequent steps for final security verification.

[0124] Output statistical metrics: This includes updated step size, convergence rate, and other statistical metrics, as well as the magnitude of behavioral change (i.e., Ψ( , The divergence value and KL divergence value are used as signal feedback and are dynamically adjusted based on optimization performance (such as whether the convergence is too fast or too slow, or whether the behavior changes too much). and This enables the entire system to achieve adaptive adjustment and reflux closed loop.

[0125] In the above embodiments, the geometric trust domain uses the Fisher information matrix approximated by K-FAC for natural gradient correction, ensuring that the parameter update direction remains geometrically smooth on the parameter manifold and suppressing gradient explosion and high-frequency oscillations. The dual-domain collaborative optimization objective function includes three terms: a performance objective term (driving performance improvement), a behavioral constraint term, and a geometric constraint term. By simultaneously limiting behavioral distribution drift and parameter update amplitude, dual-domain joint control is achieved. The cross-sensitivity factor is used to detect nonlinear failure modes characterized by "small parameter changes and large behavioral changes": when... When the threshold τ is exceeded, the behavior constraint coefficient is automatically increased. This forces the optimizer to find a smoother path in the parameter space. This collaborative mechanism ensures that policy updates maintain parameter stability while also ensuring that the evolution of behavior distribution always remains within a reliable neighborhood, thus achieving truly "small-step, controllable, and reliable" policy evolution in nonlinear deep networks.

[0126] Preferably, step S5 specifically includes: S51. Constructing the OAD candidate set: ; Among them, SWD (Sliced ​​Wasserstein Distance) is an optimal transport metric that excels at capturing the drift of global geometry; MMD (Maximum Mean Discrepancy) is an integral probability metric that is computationally stable, has smooth gradients, and excels at measuring differences in local statistical features.

[0127] S52. Calculate the overall reward for each operator: ; in, This represents the value of the overall reward. Indicates the incremental return. Indicates the change in risk. Indicates the amount of behavioral drift. As the weight of the change in risk, Weights for behavioral drift, such as A value of 0.5 is used to penalize changes in CVaR. Take 0.3 for penalty The drift increment and weight parameters can be adaptively adjusted based on the operating condition context vector or historical stability. S53. Calculate the operator selection probability using the Softmax mechanism: ; In the formula, Indicates the current context of the working condition. Next choice The probability value of using the OAD operator in the next round is the alignment divergence operator. Select the temperature parameter in the Softmax mechanism; It should be noted that the range of values ​​for the above probabilities is [missing information]. It satisfies the normalization condition:

[0128] k is the index number of the Occupied Alignment Divergence (OAD) candidate operator. K represents the number of candidates for the Occupied Alignment Divergence operator. In this invention, K=2. exp represents the exponential function used to map the composite reward to non-negative weights. Temperature parameters are selected in the Softmax selection mechanism to control the exploration process during divergence operator selection – utilizing trade-offs. When the value is large, the selection probability of each OAD operator tends to be uniform, enhancing the exploratory nature; when When the value is small, the probability of selection is concentrated on operators with larger reward values ​​to enhance utilization. The summation term in the denominator is used to normalize the exponential rewards of all candidate OAD operators, ensuring that the final selection probabilities form a valid probability distribution.

[0129] S54. Adjust the dual-domain constraint coefficients according to the OAD operator, and update the trust domain radius for the next round of behavior based on ΔOAD and ΔCVaR. : ; In the formula, δ is used to control the maximum allowable behavioral offset range of the temporary policy in step three, BTR. Mapping function The following monotonicity constraint is satisfied: as ΔOAD or ΔCVaR increases, the radius δ of the behavioral trust domain monotonically decreases, thus making the policy update more conservative.

[0130] For example, after calculating the selection probabilities of the two operators, the operator to be used in the next iteration can be determined based on the selection probabilities. For instance, the operator with the highest selection probability can be used as the operator in the next iteration.

[0131] It should be noted that, regarding the steps described above for adjusting the bi-domain constraint coefficients based on the OAD operator: the bi-domain constraint coefficients are the behavioral domain constraint coefficients. With geometric domain constraint coefficients Dynamic adjustment in this step: when the system detects extremely small but When the change is drastic (e.g., the change in the OAD value calculated by the OAD operator exceeds a preset threshold), the behavior domain constraint coefficient... It will increase dynamically, for the geometric domain constraint coefficient After each round of policy updates, the system calculates the behavioral drift change ΔOAD and the tail risk change ΔCVaR between the current round and the previous round, and adjusts the geometric domain constraint coefficients based on both. :

[0132] in: Represents the geometric domain constraint coefficients at the t-th round of policy update; Indicates the geometric domain constraint coefficients for the next policy update; This indicates the change in alignment divergence between adjacent rounds of behavior; This indicates the change in tail risk between two adjacent rounds; , Indicates the sensitivity adjustment parameters; , These represent the minimum and maximum allowable values ​​of the geometric domain constraint coefficients, respectively. Clip(·) represents the truncation function, used to ensure... It always remains within the preset safety range.

[0133] Behavioral domain constraint coefficients It is primarily responsible for controlling the behavioral drift of the policy in the state-action space, while the geometric domain constraint coefficients... It is primarily responsible for controlling the update magnitude in the policy parameter space. Together, they constitute a two-domain trust optimization mechanism.

[0134] When the system exhibits a situation where "KL divergence is small but OAD increases dramatically," it indicates that although the parameter changes are not significant, the policy behavior has already shifted noticeably. In this case, the system should prioritize increasing the behavioral domain constraint coefficient. When the system simultaneously detects both increased behavioral drift and increased tail risk, the geometric domain constraint coefficient is increased accordingly. With behavioral domain constraint coefficient This will help to tighten the boundaries of the trust domain and achieve stability, reliability, and security during the policy update process.

[0135] In the above embodiments, the OAD operator selection is modeled as a multi-armed gambling machine problem, and the operator type (SWD or MMD) for the next round is dynamically selected based on historical comprehensive rewards using a Softmax mechanism. This mechanism enables the system to adaptively select the most suitable distribution metric based on current operating conditions (such as equipment utilization, order insertion ratio, CVaR) and the effects of the previous round's strategy update (reward increment, risk change, behavioral drift). Simultaneously, based on... and By updating the behavior trust domain radius δ, a conservative scheduling mechanism is implemented where "the greater the drift, the higher the risk and the tighter the constraint." Compared with the static fixed divergence method, this adaptive scheduler significantly improves the robustness, adaptability, and cross-scenario generalization ability of the policy in non-stationary industrial environments, avoiding the problems of being too conservative or too aggressive due to the mismatch between divergence type and operating conditions.

[0136] Preferably, in step S6, the two-stage decomposition projection method specifically is as follows: Phase 1: Using the Hungarian algorithm to solve for the optimal matching of discrete batch-equipment assignment; Second stage: Optimize the continuous power regulation value using the projected gradient method, with direct truncation of the box constraint. ; In the formula, Indicates the action after truncation. This represents the truncation function. To truncate the preceding action, Indicates the lower limit of power. Indicates the upper limit of power; The overall projection optimization goal is: ; In the formula, This represents the legal actions after projection, and argmin represents minimizing the independent variable. Indicates action It belongs to the feasible region constructed based on the constraint set C. This represents the square of the Euclidean distance. This represents the risk sensitivity coefficient. This indicates an optional regularization term.

[0137] It should be noted that in this step, based on the hybrid action vector defined in S1, the core of feasible region projection is to solve an optimization problem: to project candidate actions... Projecting the action onto the feasible region Feasible(a) defined by the constraint set C, while minimizing the deviation from the original action. That is, preserving the original decision intent given by policy optimization to the greatest extent possible while satisfying all hard constraints. This can be formalized as:

[0138] in: This represents the legal action after projection. argmin represents minimizing the independent variable. Indicates action It belongs to the feasible region constructed based on the constraint set C; represents the squared Euclidean distance, and represents the deviation from the candidate action. This indicates that the risk sensitivity adjustment coefficient output above increases when the risk is high. This indicates an optional regularization term.

[0139] feasible region It is a set defined by linear / nonlinear constraints; if it is not convex, it is solved using SQP (Sequential Quadratic Programming) or an approximate convex hull. Specifically, it includes: Process sequence constraints: determined by a matrix The decision ensures that batches can only be assigned to the next process equipment allowed by their process path; Machine capability and parallelism constraints: limited by the machine capability matrix This indicates that the number of batches processed simultaneously by a single machine does not exceed the current capacity limit; Power range constraint: The physical power corresponding to continuous action must satisfy... ; Batch exclusivity constraint: The same batch cannot be split into multiple machines.

[0140] The above form is a unified expression. Considering that the action space defined in step one has a discrete-continuous hybrid structure:

[0141] in, Represents the complete action vector, [ , ] indicates vector concatenation. Indicates the discrete action part. Indicates a continuous action.

[0142] In practical solutions, this module adopts a decompositional projection strategy to process discrete and continuous subspaces separately, in order to ensure computational feasibility and real-time performance.

[0143] For example, the execution flow of this step is as follows: 1. Receive candidate actions and constraint data: For candidate actions... Perform fast constraint verification (such as process sequence matrix) (Verification). If the constraint set C is fully satisfied, then proceed directly without projection.

[0144] 2. Define the feasible region Based on the current process matrix Machine capability matrix Based on device status and context information, a feasible region for actions is constructed. For non-convex constraint cases, convex approximation or sequential quadratic programming (SQP) methods are used.

[0145] 3. Perform projection optimization: To ensure real-time performance (≤10ms), this invention employs a two-stage solution structure: (1) Discrete assignment part: The Hungarian algorithm is used to process batch-device matching.

[0146] (2) Continuous power section: Power regulation is optimized using the Projected Gradient Difference (ProjGD) method.

[0147] in, This indicates the next iteration of the continuous actions. This represents the operation of projecting onto the feasible range of physical power. This indicates the continuous actions in the current iteration. This indicates the step size (0.01) and the maximum number of iterations is 20. This represents the gradient of the objective function.

[0148] Box constraints (such as power range) are handled with simple truncation:

[0149] in, This indicates the action after truncation. This represents a truncation function that restricts values ​​to a range. This indicates the lower limit of power. Indicates the upper limit of power.

[0150] (3) Dynamic boundary adjustment: Before projection, based on the contextual features of S1 and the signal preprocessing constraints of S5: If the device utilization rate If the value is greater than 0.8, then its power limit will be tightened.

[0151] If the CVaR calculated in step one is greater than the threshold, then the adjustment signal in step five is triggered to make ρ+=0.2.

[0152] Deviation feedback: calculation ,like If the threshold is thresh (preset threshold), the following closed loop will be executed: Feedback to S5: Tighten the trust domain radius for the next round; Feedback to S2: as a projection deviation calibration item Increase, automatically suppress this action in the overall advantage signal The intensity in the reward function is equivalent to temporarily amplifying the penalty weight of ViolationPenalty in step one.

[0153] 4. Output: Projection action It interacts with the environment to generate the next state, and participates in the next round of state coding and discounted occupancy measurement in S1. The update process achieves a closed loop.

[0154] In the above embodiments, the first stage uses the Hungarian algorithm to solve the discrete batch-device assignment problem, ensuring that process sequence constraints and machine parallelism constraints are met. The second stage uses the projection gradient method to optimize the continuous power adjustment value, and directly truncates it through bin constraints to ensure that the power does not exceed the limit. This decomposition strategy breaks down the safety correction problem of the hybrid action space into two sub-problems that can be solved efficiently, meeting the millisecond-level real-time requirements of semiconductor production line scheduling. The overall projection optimization objective is minimized, preserving the decision intent of the original strategy to the maximum extent while satisfying all physical hard constraints. The projection deviation is fed back to steps S2 and S5, forming a closed-loop correction, enabling the strategy learning to perceive the boundaries of physical constraints, thereby gradually reducing the generation of illegal actions. This module, as the safety foundation of "soft strategy guidance and hard constraint projection", ensures the physical feasibility of AI scheduling instructions and eliminates violations caused by strategy exploration or sudden environmental changes.

[0155] Preferably, in step S7, the formula for calculating the comprehensive risk score is:

[0156]

[0157] In the formula, , , , These are the coupling weights for behavioral drift, risk increment, performance decline, and projection bias, respectively. For behavior-occupied alignment divergence OAD, In order to compensate for the loss, This refers to projection deviation; It should be noted that, The Behavior Occupation Alignment Divergence (OAD) defined for S3 represents the offset of the new policy relative to the old policy in the "behavior space". A surge in OAD indicates that the policy has entered a historically unverified "behavior blind zone". The tail risk variation defined for S5 is used to capture the probability of extreme losses occurring in the scheduling sequence; For the return loss term, only if the expected return of the new strategy calculated by S4 is... Below the old strategy benchmark The penalty will be triggered at that time.

[0158] When the comprehensive risk score Exceeding the preset threshold When this happens, the system triggers a soft rollback. The soft rollback mechanism integrates the old and new strategies through soft gating weights. ; In the formula, This indicates the integrated security policy. For soft-gating weights:

[0159] This shows that when the risk score When the rise exceeds the threshold, The system rapidly reduces its decision-making focus, shifting towards validated older strategies. Offset.

[0160] At the deployment level, an A / B testing paradigm is adopted. This will be applied to the total capacity. Proportion allocation to new strategy (The action output after projection correction in step six), the rest maintains the old strategy. Run. Compare the consistency of the occupation measure divergence (OAD) and the incremental return between the two groups using statistical significance tests. To ensure industrial-grade reliability before full deployment.

[0161] After executing the above process, the final security scheduling instruction, i.e., the security policy of each device, is sent to the wafer fab scheduling system. or .

[0162] For example, the execution process of this step is as follows: 1. Calculate the comprehensive risk score based on the data from S1 and S3 at each scheduling cycle. .

[0163] 2. If Immediately activate the rollback mechanism; if Abnormal (i.e.) ) abnormal surge (even with expected returns) (Stable), judged as "risk of exploring unknown areas", and a soft rollback was executed to shrink the exploration radius.

[0164] 3. Rollback execution path: Soft rollback: Based on soft-gated weights Dynamically integrate old and new strategies to achieve a gradual shift in decision-making weights.

[0165] Hard rollback: If the constraint set C defined in S1 is severely violated or a device malfunction occurs, the new strategy link is forcibly severed, instantly restoring the system to the most stable historical baseline. .

[0166] 4. At the end of the observation period, analyze the performance differences between the test group and the baseline group. If the dual criteria of "significantly improved reward increment ΔJ and controlled behavioral occupancy divergence OAD" are met, then full deployment is approved.

[0167] 5. The reasons for the rollback and risk characteristics will be reverse-synchronized to the C-DS module of S5 as an update reward. The penalty item guides the next round of operator scheduling to enhance targeted stability.

[0168] In the above embodiments, the comprehensive risk score couples the three dimensions of behavioral drift, risk increment, and performance decline into a single risk assessment indicator. When the threshold is exceeded, the soft rollback mechanism dynamically merges the old and new strategies through soft-gated weights, achieving a gradual shift in the decision-making focus and avoiding scheduling abrupt changes and equipment shocks caused by hard switching. The core innovation of this mechanism is that even if the current reward has not yet decreased significantly, once the behavior drift (OAD) surges or the tail risk (CVaR) deteriorates, the system will still trigger a preventative rollback, upgrading the traditional "post-event remediation" to "risk-driven predictive defense." The canary release mechanism verifies the behavioral consistency and benefit significance of the new strategy through A / B testing, ensuring that only strategies that have undergone dual verification are fully deployed, thereby guaranteeing the industrial-grade reliability of reinforcement learning in continuous production environments.

[0169] This application embodiment also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor, when executing programs stored in memory, implements a dynamic scheduling method based on dual-domain trust optimization.

[0170] The aforementioned electronic device can implement various embodiments of the battery replacement method described above and achieve the same beneficial effects, which will not be elaborated upon here.

[0171] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A dynamic scheduling method based on dual-domain trust optimization, characterized in that, include: S1. Collect equipment status data, work-in-process and order data, process and resource constraint data, and historical performance data of the semiconductor wafer manufacturing system. Generate multiple historical scheduling trajectories using the earliest due date priority rule. Train the initial policy network based on all historical scheduling trajectories using the behavior cloning method, calculate the initial discount occupancy measure, and generate a fixed-dimensional comprehensive state vector, a hybrid action vector containing discrete batch-equipment assignment and continuous power adjustment, a multi-objective instant reward signal, a process and resource constraint set, and a working condition context vector. The historical scheduling trajectory includes the system's actions at each time point in the historical operation. The initial discount occupancy measure is used to describe the access probability distribution of each state-action pair in the initial policy network. The multi-objective instant reward signal is used to describe the reward for action execution. The process and resource constraint set is used to describe the constraints of dynamic system scheduling at the process and resource level. The working condition context vector is used to describe the current system state. S2. Receive the multi-objective real-time reward signal, comprehensive state vector, historical performance data, and historical behavioral drift output from step S1. Perform context-aware normalization in the target value space and introduce a risk-drift joint calibration factor in the behavioral distribution space to generate a normalized comprehensive advantage signal. The behavioral drift is used to describe the degree of deviation in the distribution of scheduling actions. The risk-drift joint calibration factor is used to quantify the comprehensive suppression strength of production line risk and strategy behavioral drift. The comprehensive advantage signal is used to describe the quality of the actions performed by the equipment. S3. Receive the initial discount occupancy measure of the old strategy output in step S1 and the comprehensive advantage signal output in step S2. Estimate the discount occupancy measure of the temporary candidate strategy based on importance sampling. Calculate the OAD value between the old and new strategies. Detect the behavior drift and adaptively scale the strategy update step size. The OAD value is used to quantify the distribution difference of the discounted occupancy measure between the old and new strategies. S4. Receive the policy update step size and historical dual-domain constraint coefficients output in step S3. Combine the geometric trust domain natural gradient and behavioral trust domain OAD constraint based on K-FAC approximation to construct a dual-domain collaborative optimization objective function. Detect parameter-behavior cross sensitivity and dynamically adjust constraint weights. Update policy parameters to generate candidate policies and calculate reward increment and risk change. The dual-domain collaborative optimization objective function is used to describe the expected returns in the geometric trust domain and the behavioral trust domain, the return increment is used to describe the increase in returns from dynamically adjusting constraint weights, and the risk change is used to describe the decrease in returns from dynamically adjusting constraint weights. S5. Receive the working context vector output from step S1, the reward increment and risk change output from step S4, and the behavior drift output from step S3. Select the OAD operator type according to the adaptive selection mechanism, and dynamically adjust the behavior trust domain radius and the dual-domain constraint coefficient; the OAD operator is used to calculate the OAD value. S6. Receive the candidate strategy output from step S4 and the set of process and resource constraints output from step S1, construct a dynamic feasible region, and use a two-stage decomposition projection method to map illegal actions to legal actions, and calculate the projection deviation; the projection deviation is used to describe the severity of the action violating the constraints. S7. Receive the projection deviation output from step S6, the reward increment and update statistics output from step S4, calculate the comprehensive risk score, execute a soft rollback or hard rollback mechanism based on the comprehensive risk score, verify the performance of the new strategy through canary release, and output the final safe scheduling instruction to the wafer manufacturing line; the update statistics include the strategy update step size during strategy update in step S4, and the final safe scheduling instruction is used to indicate the actions of each device in the wafer manufacturing line at future moments.

2. The dynamic scheduling method based on dual-domain trust optimization according to claim 1, characterized in that, In step S1, the calculation formula for the initial discount occupancy measure is as follows: In the formula, As a measure of initial discount usage, For state-action pairs in the trajectory. For state, For action, For time in historical scheduling trajectory The following state, For time in historical scheduling trajectory The following action As a discount factor, For indicator functions, A collection of historical scheduling trajectories. For the number of trajectories, For historical scheduling trajectory, Historical scheduling trajectory Length; The calculation formula for the multi-target instant reward signal is as follows: ； In the formula, The value of the multi-objective instant reward signal. This indicates an encouragement strategy to improve production efficiency. This indicates a limitation on energy consumption. Indicates the risk of late delivery. Let C represent the constraint set.

3. The dynamic scheduling method based on dual-domain trust optimization according to claim 1, characterized in that, In step S1, the comprehensive state vector is generated by concatenating equipment-side features, work-in-process-side features, and time context features. Among them, the equipment-side features include equipment utilization, real-time power, one-hot encoding of operating status, and mean time between failures; the work-in-process-side features include global statistical features and Top-K key batch features sorted by priority × 1 / delivery date; and the time context features include the mean of context dependencies, the standard deviation of context dependencies, and the trend slope within the sliding window.

4. The dynamic scheduling method based on dual-domain trust optimization according to claim 3, characterized in that, The historical performance data includes throughput, energy cost, late delivery rate, and tail risk value; Step S2 specifically includes: S21. Advantage function calculation: Calculate the advantage function value of each objective based on the multi-objective real-time reward signal and the state value function estimation result; S22. Target Value Space Normalization: Using a sliding window, the advantage function values ​​of each target are calibrated based on the mean and standard deviation of context dependencies in the integrated state vector. The calculation formula is as follows: ； In the formula, Indicates the first The calibrated advantage function value of each target. Indicates the first The original advantage function values ​​for each objective. Indicates the first The mean of context dependencies for each objective. Indicates the first The standard deviation of the context dependency of each objective; S23. Behavioral Distribution Spatial Risk Calibration: A risk-drift joint calibration factor is calculated based on the tail risk value and behavioral drift, introducing a projection bias correction term. The calculation formula is as follows: ； In the formula, For the risk-drift joint calibration factor, Indicates the amount of behavioral drift. Indicates the tail risk value. , For sensitivity; S24. Unified Driving Signal Generation: Combining context-aware target weights, weighted fusion is used to generate a normalized comprehensive advantage signal. ； In the formula, To output a normalized comprehensive advantage signal, Indicates the first Context-aware weights for each target.

5. The dynamic scheduling method based on dual-domain trust optimization according to claim 1, characterized in that, In step S3, the OAD value is selected from slice Wasserstein distance (SWD) or maximum mean difference (MMD). The formula for calculating SWD is as follows: ； The formula for calculating MMD is as follows: ； In the formula, This represents the discount occupancy measure of the old strategy. Discount occupancy measurement with the new strategy The Wasserstein distance between the slices express and The maximum mean difference between them This represents the integral from 0 to 1. Represents the square of the absolute value. This represents the inverse function of the cumulative distribution function of the old distribution. The cumulative distribution function of the new distribution is the inverse function, and dp represents the integration variable. In the regenerating nucleus Hilbert space Norms in This represents the kernel mean embedding of the old distribution. This represents the kernel mean embedding of the new distribution. Let x represent the kernel function and x represent the sample point; When OAD value is detected At that time, the step size contraction mechanism is triggered, and the updated formula is: In the formula, where ∈(0,1) is the step size scaling factor. For the updated policy parameters, This represents the original policy parameters. These are the candidate strategy parameters.

6. The dynamic scheduling method based on dual-domain trust optimization according to claim 1, characterized in that, In step S4, the natural gradient of the geometric trust domain is calculated using the K-FAC approximation of the Fisher information matrix, as shown in the formula: ； In the formula, This indicates the parameter update increment, and ≈ represents the approximation sign. Represents the inverse of the Fisher information matrix. Describe the objective function Relative to parameters gradient, Indicates the strategy parameters, Indicates the parameterization strategy, This represents the objective function for policy optimization. The objective function for dual-domain collaborative optimization is: ； In the formula, This represents the total loss function for bi-domain collaborative optimization. Represents the mathematical expectation symbol. This indicates the current strategy to be optimized. These are the learnable parameters of a neural network. Indicates the old strategy, Indicates the importance sampling weight, This indicates a normalized overall advantage signal. Indicates the behavioral domain constraint coefficient. This represents the occupancy measure divergence operator. This represents the discount occupancy measure of the old strategy. This represents the discount occupancy measure of the current strategy. Represents the geometric domain constraint coefficients. Indicates the Kullback-Leibler divergence. This represents the expectation of the above KL divergence under the state access distribution of the old strategy; Calculation parameter - Behavioral cross-sensitivity factor: ； When the parameter - behavior cross sensitivity factor Nonlinearly increasing the behavioral domain constraint coefficient ,in For the preset threshold, It is a constant.

7. The dynamic scheduling method based on dual-domain trust optimization according to claim 1, characterized in that, Step S5 specifically includes: S51. Constructing the OAD candidate set: ； S52. Calculate the overall reward for each operator: ； in, This represents the value of the overall reward. Indicates the incremental return. Indicates the change in risk. Indicates the amount of behavioral drift. As the weight of the change in risk, The weights for behavioral drift; S53. Calculate the operator selection probability using the Softmax mechanism: ； In the formula, Indicates the current context of the working condition. Next choice The probability value of using the OAD operator in the next round is the alignment divergence operator. Select the temperature parameter in the Softmax mechanism; S54. Adjust the dual-domain constraint coefficients according to the OAD operator, and update the trust domain radius for the next round of behavior based on ΔOAD and ΔCVaR. : ； In the formula, δ is used to control the maximum range of behavioral offsets allowed by the temporary strategy in step three, BTR.

8. The dynamic scheduling method based on dual-domain trust optimization according to claim 1, characterized in that, In step S6, the two-stage decomposition projection method specifically refers to: Phase 1: Using the Hungarian algorithm to solve for the optimal matching of discrete batch-equipment assignment; Second stage: Optimize the continuous power regulation value using the projected gradient method, with direct truncation of the box constraint. ； In the formula, Indicates the action after truncation. This represents the truncation function. To truncate the preceding action, Indicates the lower limit of power. Indicates the upper limit of power; The overall projection optimization goal is: ； In the formula, Indicates the legal action after projection. This indicates minimizing the independent variable. Indicates action It belongs to the feasible region constructed based on the set of process and resource constraints C. This represents the square of the Euclidean distance. This represents the risk sensitivity coefficient. This indicates an optional regularization term.

9. The dynamic scheduling method based on dual-domain trust optimization according to claim 1, characterized in that, In step S7, the formula for calculating the comprehensive risk score is: In the formula, , , , These are the coupling weights for behavioral drift, risk increment, performance decline, and projection bias, respectively. For behavior-occupied alignment divergence OAD, In order to compensate for the loss, This refers to projection deviation; The soft rollback mechanism integrates the old and new strategies through soft-gated weights: ； In the formula, This indicates the integrated security policy. This is a soft-gating weight.

10. A dynamic scheduling system based on dual-domain trust optimization, comprising a processor and a memory, characterized in that... ； Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the dynamic scheduling method based on dual-domain trust optimization as described in any one of claims 1 to 9.

Images