A shared bottleneck reentrant scheduling reinforcement learning training method based on digital twin operation feedback

Abstract

The application discloses a shared bottleneck reentrant scheduling reinforcement learning training method based on digital twin operation feedback, which generates a unified order scene by a reinforcement learning training end, dynamically creates orders and products according to order release time by a digital twin operation end, and generates a global aggregation state and a candidate product real-time snapshot when a shared bottleneck device has dispatching conditions; the reinforcement learning training end outputs a dynamic weight according to the global aggregation state, and the digital twin operation end executes a dispatching action and promotes a production process according to the dynamic weight; the digital twin operation end counts running increments such as a newly added canceled order number, a time advancing amount and a shared bottleneck device idle capacity area after the dispatching action is executed, and the reinforcement learning training end calculates an instant reward according to the running increments, forms experience samples by combining a state, an action, a reward and a new state, and updates a reinforcement learning model, thereby improving consistency between training feedback and a manufacturing system operation target.

Description

Technical Field

[0001] This invention relates to the field of intelligent workshop scheduling and simulation optimization technology, and in particular to a reinforcement learning training method for shared bottleneck reentrancy scheduling based on digital twin operational feedback. Background Technology

[0002] Shared bottleneck reentrant manufacturing systems are widely used in large-scale equipment manufacturing, discrete manufacturing, and multi-process collaborative production scenarios. Taking a wind turbine tower manufacturing workshop as an example, a four-roll mill performs both the initial milling process and the reentry milling process, forming a shared bottleneck process. These systems exhibit complex dynamic characteristics such as dynamic order release, dynamic product arrival, dynamic equipment idle time, reentry return waiting, and order cancellations, posing a significant challenge to production scheduling in terms of multi-objective coordination and optimization. Order delivery dates, equipment utilization, and order delivery rates are mutually constrained, while the shared bottleneck equipment faces resource competition between initial and reentry processing tasks, increasing the difficulty of scheduling decisions. Therefore, how to achieve an efficient and robust scheduling scheme under these complex dynamic conditions is the core technical problem that shared bottleneck reentrant manufacturing systems need to solve.

[0003] In existing technologies, the following technical approaches are commonly used to solve the scheduling problem in manufacturing workshops. One category is rule-based methods, such as heuristic rules like first-come-first-served, earliest delivery date priority, or shortest processing time priority, where work assignments are executed manually or by a system according to fixed rules. Another category is methods based on mathematical programming or metaheuristic search, such as genetic algorithms, particle swarm optimization, and simulated annealing, which generate static scheduling plans through offline optimization. In recent years, deep reinforcement learning technology has been introduced into the scheduling field. By modeling the scheduling problem as a Markov decision process, an agent learns scheduling strategies through interaction with the environment. Typical implementations include using deep Q-networks or proximal policy optimization algorithms, taking workshop state as input, scheduling rules or machine allocation as action outputs, and using production indicators such as maximum completion time or equipment utilization rate as reward signals for training. Digital twin technology has also been widely used in the manufacturing field, constructing virtual models corresponding to the physical workshop to achieve simulation, monitoring, and visualization of the production process.

[0004] However, the aforementioned existing technologies still have the following shortcomings when facing the dynamic scheduling of shared bottleneck re-entry manufacturing systems. First, Chinese patent CN117669984B discloses a workshop scheduling method based on reinforcement learning using digital twins and knowledge graphs. This scheme utilizes a digital twin model to provide state representation and combines it with a knowledge graph to enhance decision interpretability. However, its reinforcement learning reward signal still relies on static rule calculation. The digital twin environment used acts as a state supply and performance verification mechanism, but does not undertake the actual execution of work assignments, nor does it statistically analyze the results after execution and feed them back to the training process. Therefore, there is a disconnect between the reward signal and the actual production progress. Second, Chinese patent CN116821223B discloses an industrial visualization control platform and method based on digital twins. This scheme uses digital twin technology to achieve the visualization and alarm prompts of industrial scenarios. The digital twin model is only used for monitoring and display purposes and does not form a closed loop with reinforcement learning training. It does not use the operating results of the controlled object as training feedback to participate in the agent's policy update, limiting the role of digital twins in modeling complex dynamic manufacturing processes during training. Third, Chinese patent CN119740681A discloses a dynamic scheduling method for a digital twin flexible workshop. This scheme uses a multi-agent PPO algorithm for training and scheduling in a virtual workshop. However, the training and evaluation processes are not explicitly isolated, and the model may still update its parameters during evaluation. This makes it difficult to objectively compare the performance results of different scheduling strategies or different training stages under the same conditions, weakening the credibility of the model evaluation. Therefore, it is necessary to design a shared bottleneck reentrant scheduling reinforcement learning training method based on digital twin operational feedback to solve the above problems. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a shared bottleneck reentrancy scheduling reinforcement learning training method based on digital twin operation feedback. It aims to overcome the technical problems in existing reinforcement learning scheduling training, such as the disconnect between reward signals and actual operation processes, the fact that the digital twin environment is only used for display and does not participate in action execution and feedback training, and the difficulty in synchronously modeling order release and multi-resource collaboration in shared bottleneck reentrancy scenarios. It has the characteristics of using the digital twin operation terminal as a closed-loop feedback environment for dispatch action execution and operation increment statistics, and constructing training rewards based on actual operation results and maintaining consistency with the core objectives of shortening completion time, improving bottleneck equipment utilization, and reducing order cancellation rate.

[0006] To achieve the above technical solution, the technical solution adopted by the present invention is as follows: A shared bottleneck reentrant scheduling reinforcement learning training method based on digital twin operational feedback includes the following steps: S1, an order scenario generated by the reinforcement learning training end for a shared bottleneck reentrancy manufacturing system; S2 loads order scenarios from the digital twin runtime and dynamically creates orders and products according to the order release time; S3 is driven by the digital twin operation terminal to advance the production process and triggers a dispatch decision request when the shared candidate queue is not empty and there are idle devices in the shared bottleneck device. S4 generates a global aggregated state and real-time snapshots of candidate products from the digital twin runtime and sends them to the reinforcement learning training end; S5, the reinforcement learning training end outputs dynamic weights based on the global aggregate state; S6, the digital twin operation terminal executes candidate product dispatching actions and promotes the production process based on dynamic weights; S7, the digital twin runtime terminal calculates the runtime increment after the dispatch action is executed, and sends the new state and runtime increment to the reinforcement learning training terminal; S8. The reinforcement learning training end calculates the instant reward based on the running increment, and updates the reinforcement learning model based on the global aggregate state, dynamic weights, instant reward, and new state.

[0007] Furthermore, the order scenario includes order number, product number, process route, processing time of each process, order delivery date, order release time, order size, delivery pressure parameters, and random seed information.

[0008] Preferably, the digital twin runtime dynamically creates orders and products according to the order release time, and controls the products to sequentially enter the pre-processing process, the first bottleneck processing process, the intermediate processing process, the re-entry bottleneck processing process, and the subsequent processing process according to the current process status of the products; the shared bottleneck re-entry manufacturing system includes the pre-processing process, the first bottleneck processing process, the intermediate processing process, the re-entry bottleneck processing process, and the subsequent processing process connected in sequence, wherein the first bottleneck processing process and the re-entry bottleneck processing process use the same group of shared bottleneck equipment, and the products return to the shared bottleneck equipment group to perform the re-entry bottleneck processing process after completing the intermediate processing process.

[0009] Furthermore, when the digital twin operating system advances the production process, it simulates the flow of orders and products within the shared bottleneck re-entry manufacturing system through a discrete event simulation mechanism, specifically including: Scan the equipment occupancy status and queue status of each process according to the current simulation time; When a device completes processing, the product is released from the current device to the waiting queue for the next process, and the device status is updated to idle. For non-bottleneck processes, when the equipment is idle, products are selected from the waiting queue of that process according to the first-in-first-out rule for processing; For shared bottleneck processes, a work assignment decision request is triggered when the equipment is idle and the shared candidate queue is not empty, and the processing order of the products is determined by the dynamic weights output by the reinforcement learning training terminal. The execution of dispatching actions or the completion of equipment processing events are used as the driving events for time progression, advancing the simulation time to the moment when the next event occurs.

[0010] Preferably, the reinforcement learning training objectives during the training process revolve around the following three core optimization metrics: by This represents minimizing the maximum completion time, where Indicates the maximum completion time; by This represents maximizing the utilization of shared bottleneck devices, where Indicates the utilization rate of shared bottleneck equipment; by This represents minimizing the order cancellation rate, where Indicates the number of canceled orders. This indicates the initial number of orders.

[0011] Furthermore, the global aggregated status includes the current time, completion rate, shared queue length, number of initial bottleneck processing waits, number of re-entry bottleneck processing waits, percentage of re-entry bottleneck processing waits, percentage of high-risk orders, average waiting time, maximum waiting time, shared bottleneck equipment idle rate, shared bottleneck equipment utilization rate, and downstream hunger.

[0012] Furthermore, the real-time snapshot of the candidate product includes delivery pressure, cancellation risk, waiting status, re-entry status, current process priority, product contribution, downstream hunger, and bottleneck adaptability.

[0013] Preferably, the reinforcement learning training end adopts a soft actor-critic reinforcement learning model, and outputs dynamic weights of fixed dimensions according to the global aggregation state. The dynamic weights are used by the digital twin running end to decode actions and assign tasks to candidate products. The soft actor-critic reinforcement learning model includes a policy network, a first Q-network, a second Q-network, and a temperature parameter. The policy network takes the global aggregated state as input and maps it to the mean and logarithmic standard deviation of actions in each dimension through a multi-layer fully connected network. Dynamic weights for the fixed dimensions are obtained through reparameterized sampling. The first Q-network and the second Q-network estimate the state-action value respectively, and take the smaller of the two to suppress overestimation. The temperature parameter is used to adjust the strength of policy entropy regularization, which is automatically adjusted by the reinforcement learning training end during training by minimizing the entropy objective function or set to a preset fixed value. During the training phase, the policy network outputs dynamic weights using random sampling to maintain exploration, and during the evaluation phase, it outputs dynamic weights using the mean to achieve deterministic assignment. The digital twin runtime scores and sorts the candidate products according to the dynamic weights, and then allocates the candidate products to idle shared bottleneck devices according to the sorting results.

[0014] Preferably, the operational increment includes the number of newly cancelled orders, the time progress, and the idle capacity area of ​​shared bottleneck equipment between the execution of the dispatch action and the next dispatch decision point; the operational increment also includes at least one of the following: the number of shared bottleneck equipment, the cumulative busy time of shared bottleneck equipment, the cumulative idle area of ​​shared bottleneck equipment, the current utilization rate of shared bottleneck equipment, the current simulation time or the maximum completion time, the number of completed orders, and the number of cancelled orders.

[0015] Preferably, the instant reward is calculated as follows: Based on the number of newly cancelled orders Compared with the initial number of orders The ratio yields the increment of cancellation loss: ; According to the Time advance of each decision interval Average processing workload per unit order The ratio yields the time-advancing loss increment: ; Based on the idle capacity area of ​​shared bottleneck equipment Number of shared bottleneck devices and average processing workload per unit order The ratio of the products yields the increment of idle loss: ; Average processing workload per unit order Total processing workload for the scene Compared with the initial number of orders The ratio, that is ; The immediate reward is determined based on the weighted negative value of the cancellation loss increment, the time advancement loss increment, and the idle loss increment: ; in , and These are non-negative weighting coefficients.

[0016] Preferably, the first The start time of each decision interval The time when the dispatch decision request is triggered by the digital twin operating terminal, and the termination time. After the digital twin operation terminal executes the dispatch action, the production process is advanced to the earliest moment when any of the following conditions are met: a shared bottleneck device is idle again and the shared candidate queue is not empty, all orders are completed, or the preset simulation termination time is reached.

[0017] Furthermore, the non-negative weighting coefficients , and The magnitude of loss increments is determined by hyperparameter search or by normalization based on the magnitude of each running increment. The normalization is to divide each running increment by its statistical mean in the entire training scenario so that the three loss increments are of the same magnitude.

[0018] Furthermore, reentry waiting time, average delay, total delay, average waiting time in the shared queue, and reentry waiting area are statistically analyzed as auxiliary evaluation indicators or mechanism analysis indicators, rather than as the main reward item of the instant reward.

[0019] Preferably, the reinforcement learning training end constructs an experience sample from the current global aggregation state, dynamic weights, immediate reward, new global aggregation state, and scene termination flag, and writes the experience sample into the experience replay cache for updating the soft actor-critic reinforcement learning model.

[0020] Preferably, the digital twin runtime and the reinforcement learning training end interact with each other through a communication protocol that includes event number, current time, global aggregation state, candidate product number, real-time snapshot of candidate product, model version, dynamic weight, dispatch sequence and runtime increment, and perform idempotency verification through event number. When communication between the digital twin runtime and the reinforcement learning training end is interrupted, the training process is resumed based on the most recently confirmed event number, scene state and model version.

[0021] Preferably, the training process and the evaluation process are isolated from each other. The training process allows the reinforcement learning model parameters to be updated, while the evaluation process loads the specified model parameters and performs the scheduling policy evaluation without updating the model parameters.

[0022] Furthermore, it also includes a shared bottleneck reentrancy scheduling reinforcement learning training system based on digital twin operational feedback, comprising: The order scenario generation module is used to generate order scenarios for the shared bottleneck re-entry manufacturing system. The digital twin order release module, connected to the order scenario generation module, is used to receive the order scenario and dynamically create orders and products according to the order release time; The digital twin production advancement module, connected to the digital twin order release module, is used to receive created orders and products, and simulate product processing, equipment occupation, queue waiting, re-entry return, and order completion. The status and snapshot acquisition module is connected to the digital twin production advancement module and is used to collect data from the production advancement process and generate global aggregated status and real-time snapshots of candidate products. The reinforcement learning action generation module is connected to the state and snapshot acquisition module, and is used to receive the global aggregated state and output dynamic weights; The action execution module is connected to the reinforcement learning action generation module and the digital twin production promotion module, respectively, and is used to receive the dynamic weights and execute the dispatching action, and to feed back the dispatching result to the digital twin production promotion module. The incremental statistics module is connected to the digital twin production advancement module to receive production advancement data and to count the number of new and canceled orders, time advancement, and idle capacity area of ​​shared bottleneck equipment. The reward calculation module, connected to the running increment statistics module, is used to receive the running increment and calculate the instant reward; The experience replay and model update module is connected to the reward calculation module and the reinforcement learning action generation module, respectively, and is used to receive the instant reward, construct experience samples, write them to the experience replay cache and update the reinforcement learning model. The communication protocol and anomaly recovery module are connected to the status and snapshot acquisition module, the reinforcement learning action generation module, and the runtime incremental statistics module, respectively, to realize message interaction, idempotency verification, and disconnection recovery.

[0023] Furthermore, an electronic device includes a processor and a memory, the memory storing a computer program that, when executed by the processor, implements the aforementioned shared bottleneck reentrancy scheduling reinforcement learning training method based on digital twin runtime feedback.

[0024] Furthermore, a computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the aforementioned shared bottleneck reentrancy scheduling reinforcement learning training method based on digital twin operational feedback.

[0025] The beneficial effects of this invention are as follows: 1. This invention constructs a closed-loop training framework integrating a reinforcement learning training end and a digital twin operation end. This enables scheduling actions to be executed in the digital twin environment and training feedback to be generated from the results. This solves the problem of the disconnect between reward signals and actual production operations in traditional methods, ensuring that training feedback aligns with the core objectives of shortening maximum completion time, improving the utilization rate of shared bottleneck equipment, and reducing order cancellation rates. Specifically, the reinforcement learning training end is responsible for generating a unified order scenario and outputting dynamic weights, while the digital twin operation end is responsible for dynamically creating products and executing dispatch actions according to order release times. After each dispatch, the digital twin operation end calculates operational increments such as the number of newly cancelled orders, time progress, and idle capacity area of ​​shared bottleneck equipment. The reinforcement learning training end constructs real-time rewards and updates the model based on these operational increments. This ensures that reward calculation is based on the actual operational results after action execution rather than manually preset rule evaluations, improving the consistency between the training process and the manufacturing system's operational goals.

[0026] 2. This invention solves the problem of traditional reward functions' difficulty in unifying and coordinating multiple optimization objectives by constructing an immediate reward based on the operational increments of the number of newly cancelled orders, time advancement, and the idle capacity area of ​​shared bottleneck equipment. This allows the three core objectives—reducing maximum completion time, improving bottleneck equipment utilization, and reducing order cancellation rate—to be uniformly incorporated into the training and optimization process. Specifically, the cancellation loss increment guides the model to reduce order cancellations, the time advancement loss increment guides the model to shorten the production cycle, and the idle loss increment guides the model to improve the utilization efficiency of shared bottleneck equipment. The three loss increments are weighted, summed, and negatively evaluated to form a unified immediate reward signal, enabling the reinforcement learning model to simultaneously optimize multiple production operation indicators within a comprehensive reward framework. Auxiliary evaluation indicators, such as reentry waiting time and average delay, are used as mechanism analysis tools to explain the internal coordination effect of the scheduling strategy but do not participate in the main reward calculation, thus avoiding the negative impact of coupling interference between multiple objectives on training convergence.

[0027] 3. This invention addresses the issues of unreliable comparison of model performance and insufficient stability in the interaction process by implementing an isolation mechanism between the training and evaluation processes and an event-numbered communication protocol between the digital twin runtime and the reinforcement learning training end. This improves the reliability and comparability of results for long-term training tasks. Specifically, the training process allows the reinforcement learning model to continuously optimize its strategy through experience replay and parameter updates. The evaluation process loads a specified model version and outputs dynamic weights in a deterministic manner. After completing a full scene simulation, various evaluation metrics are calculated without updating model parameters. The digital twin runtime and the reinforcement learning training end interact with each other through a communication protocol containing event numbers, handling three types of messages: dispatch requests, action responses, and operational feedback. Idempotency checks are performed using event numbers to prevent duplicate execution. In the event of a communication interruption, the training process resumes based on the most recently confirmed event number, scene state, and model version, reducing the impact of communication anomalies on training continuity. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the closed-loop training process between the reinforcement learning training terminal and the digital twin operating terminal in an embodiment of the present invention. Figure 2 This is a flowchart illustrating the dynamic creation of orders based on release time in an embodiment of the present invention; Figure 3 This is a timing diagram of the communication protocol message interaction in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the instant reward calculation based on addition / cancellation, time progression, and idle loss of shared bottleneck devices in an embodiment of the present invention; Figure 5 This is a flowchart illustrating the experience replay caching and reinforcement learning model update process in an embodiment of the present invention. Figure 6 This is a schematic diagram illustrating the isolation between the training and evaluation processes in an embodiment of the present invention; Figure 7 This is a flowchart of the disconnection recovery and idempotency verification process in an embodiment of the present invention. Detailed Implementation

[0029] Example 1: A shared bottleneck reentrant scheduling reinforcement learning training method based on digital twin operational feedback includes the following steps: S1, an order scenario generated by the reinforcement learning training end for a shared bottleneck reentrancy manufacturing system; S2 loads order scenarios from the digital twin runtime and dynamically creates orders and products according to the order release time; S3 is driven by the digital twin operation terminal to advance the production process and triggers a dispatch decision request when the shared candidate queue is not empty and there are idle devices in the shared bottleneck device. S4 generates a global aggregated state and real-time snapshots of candidate products from the digital twin runtime and sends them to the reinforcement learning training end; S5, the reinforcement learning training end outputs dynamic weights based on the global aggregate state; S6, the digital twin operation terminal executes candidate product dispatching actions and promotes the production process based on dynamic weights; S7, the digital twin runtime terminal calculates the runtime increment after the dispatch action is executed, and sends the new state and runtime increment to the reinforcement learning training terminal; S8. The reinforcement learning training end calculates the instant reward based on the running increment, and updates the reinforcement learning model based on the global aggregate state, dynamic weights, instant reward, and new state.

[0030] Furthermore, the order scenario includes order number, product number, process route, processing time of each process, order delivery date, order release time, order size, delivery pressure parameters, and random seed information.

[0031] Preferably, the digital twin runtime dynamically creates orders and products according to the order release time, and controls the products to sequentially enter the pre-processing process, the first bottleneck processing process, the intermediate processing process, the re-entry bottleneck processing process, and the subsequent processing process according to the current process status of the products; the shared bottleneck re-entry manufacturing system includes the pre-processing process, the first bottleneck processing process, the intermediate processing process, the re-entry bottleneck processing process, and the subsequent processing process connected in sequence, wherein the first bottleneck processing process and the re-entry bottleneck processing process use the same group of shared bottleneck equipment, and the products return to the shared bottleneck equipment group to perform the re-entry bottleneck processing process after completing the intermediate processing process.

[0032] Furthermore, when the digital twin operating system advances the production process, it simulates the flow of orders and products within the shared bottleneck re-entry manufacturing system through a discrete event simulation mechanism, specifically including: Scan the equipment occupancy status and queue status of each process according to the current simulation time; When a device completes processing, the product is released from the current device to the waiting queue for the next process, and the device status is updated to idle. For non-bottleneck processes, when the equipment is idle, products are selected from the waiting queue of that process according to the first-in-first-out rule for processing; For shared bottleneck processes, a work assignment decision request is triggered when the equipment is idle and the shared candidate queue is not empty, and the processing order of the products is determined by the dynamic weights output by the reinforcement learning training terminal. The execution of dispatching actions or the completion of equipment processing events are used as the driving events for time progression, advancing the simulation time to the moment when the next event occurs.

[0033] Preferably, the reinforcement learning training objectives during the training process revolve around the following three core optimization metrics: by This represents minimizing the maximum completion time, where Indicates the maximum completion time; by This represents maximizing the utilization of shared bottleneck devices, where Indicates the utilization rate of shared bottleneck equipment; by This represents minimizing the order cancellation rate, where Indicates the number of canceled orders. This indicates the initial number of orders.

[0034] Furthermore, the global aggregated status includes the current time, completion rate, shared queue length, number of initial bottleneck processing waits, number of re-entry bottleneck processing waits, percentage of re-entry bottleneck processing waits, percentage of high-risk orders, average waiting time, maximum waiting time, shared bottleneck equipment idle rate, shared bottleneck equipment utilization rate, and downstream hunger.

[0035] Furthermore, the real-time snapshot of the candidate product includes delivery pressure, cancellation risk, waiting status, re-entry status, current process priority, product contribution, downstream hunger, and bottleneck adaptability.

[0036] Preferably, the reinforcement learning training end adopts a soft actor-critic reinforcement learning model, and outputs dynamic weights of fixed dimensions according to the global aggregation state. The dynamic weights are used by the digital twin running end to decode actions and assign tasks to candidate products. The soft actor-critic reinforcement learning model includes a policy network, a first Q-network, a second Q-network, and a temperature parameter. The policy network takes the global aggregated state as input and maps it to the mean and logarithmic standard deviation of actions in each dimension through a multi-layer fully connected network. Dynamic weights for the fixed dimensions are obtained through reparameterized sampling. The first Q-network and the second Q-network estimate the state-action value respectively, and take the smaller of the two to suppress overestimation. The temperature parameter is used to adjust the strength of policy entropy regularization, which is automatically adjusted by the reinforcement learning training end during training by minimizing the entropy objective function or set to a preset fixed value. During the training phase, the policy network outputs dynamic weights using random sampling to maintain exploration, and during the evaluation phase, it outputs dynamic weights using the mean to achieve deterministic assignment. The digital twin runtime scores and sorts the candidate products according to the dynamic weights, and then allocates the candidate products to idle shared bottleneck devices according to the sorting results.

[0037] Preferably, the operational increment includes the number of newly cancelled orders, the time progress, and the idle capacity area of ​​shared bottleneck equipment between the execution of the dispatch action and the next dispatch decision point; the operational increment also includes at least one of the following: the number of shared bottleneck equipment, the cumulative busy time of shared bottleneck equipment, the cumulative idle area of ​​shared bottleneck equipment, the current utilization rate of shared bottleneck equipment, the current simulation time or the maximum completion time, the number of completed orders, and the number of cancelled orders.

[0038] Preferably, the instant reward is calculated as follows: Based on the number of newly cancelled orders Compared with the initial number of orders The ratio yields the increment of cancellation loss: ; According to the Time advance of each decision interval Average processing workload per unit order The ratio yields the time-advancing loss increment: ; Based on the idle capacity area of ​​shared bottleneck equipment Number of shared bottleneck devices and average processing workload per unit order The ratio of the products yields the increment of idle loss: ; Average processing workload per unit order Total processing workload for the scene Compared with the initial number of orders The ratio, that is ; The immediate reward is determined based on the weighted negative value of the cancellation loss increment, the time advancement loss increment, and the idle loss increment: ; in , and These are non-negative weighting coefficients.

[0039] Preferably, the first The start time of each decision interval The time when the dispatch decision request is triggered by the digital twin operating terminal, and the termination time. After the digital twin operation terminal executes the dispatch action, the production process is advanced to the earliest moment when any of the following conditions are met: a shared bottleneck device is idle again and the shared candidate queue is not empty, all orders are completed, or the preset simulation termination time is reached.

[0040] Furthermore, the non-negative weighting coefficients , and The magnitude of loss increments is determined by hyperparameter search or by normalization based on the magnitude of each running increment. The normalization is to divide each running increment by its statistical mean in the entire training scenario so that the three loss increments are of the same magnitude.

[0041] Furthermore, reentry waiting time, average delay, total delay, average waiting time in the shared queue, and reentry waiting area are statistically analyzed as auxiliary evaluation indicators or mechanism analysis indicators, rather than as the main reward item of the instant reward.

[0042] Preferably, the reinforcement learning training end constructs an experience sample from the current global aggregation state, dynamic weights, immediate reward, new global aggregation state, and scene termination flag, and writes the experience sample into the experience replay cache for updating the soft actor-critic reinforcement learning model.

[0043] Preferably, the digital twin runtime and the reinforcement learning training end interact with each other through a communication protocol that includes event number, current time, global aggregation state, candidate product number, real-time snapshot of candidate product, model version, dynamic weight, dispatch sequence and runtime increment, and perform idempotency verification through event number. When communication between the digital twin runtime and the reinforcement learning training end is interrupted, the training process is resumed based on the most recently confirmed event number, scene state and model version.

[0044] Preferably, the training process and the evaluation process are isolated from each other. The training process allows the reinforcement learning model parameters to be updated, while the evaluation process loads the specified model parameters and performs the scheduling policy evaluation without updating the model parameters.

[0045] Furthermore, it also includes a shared bottleneck reentrancy scheduling reinforcement learning training system based on digital twin operational feedback, comprising: The order scenario generation module is used to generate order scenarios for the shared bottleneck re-entry manufacturing system. The digital twin order release module, connected to the order scenario generation module, is used to receive the order scenario and dynamically create orders and products according to the order release time; The digital twin production advancement module, connected to the digital twin order release module, is used to receive created orders and products, and simulate product processing, equipment occupation, queue waiting, re-entry return, and order completion. The status and snapshot acquisition module is connected to the digital twin production advancement module and is used to collect data from the production advancement process and generate global aggregated status and real-time snapshots of candidate products. The reinforcement learning action generation module is connected to the state and snapshot acquisition module, and is used to receive the global aggregated state and output dynamic weights; The action execution module is connected to the reinforcement learning action generation module and the digital twin production promotion module, respectively, and is used to receive the dynamic weights and execute the dispatching action, and to feed back the dispatching result to the digital twin production promotion module. The incremental statistics module is connected to the digital twin production advancement module to receive production advancement data and to count the number of new and canceled orders, time advancement, and idle capacity area of ​​shared bottleneck equipment. The reward calculation module, connected to the running increment statistics module, is used to receive the running increment and calculate the instant reward; The experience replay and model update module is connected to the reward calculation module and the reinforcement learning action generation module, respectively, and is used to receive the instant reward, construct experience samples, write them to the experience replay cache and update the reinforcement learning model. The communication protocol and anomaly recovery module are connected to the status and snapshot acquisition module, the reinforcement learning action generation module, and the runtime incremental statistics module, respectively, to realize message interaction, idempotency verification, and disconnection recovery.

[0046] Furthermore, an electronic device includes a processor and a memory, the memory storing a computer program that, when executed by the processor, implements the aforementioned shared bottleneck reentrancy scheduling reinforcement learning training method based on digital twin runtime feedback.

[0047] Furthermore, a computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the aforementioned shared bottleneck reentrancy scheduling reinforcement learning training method based on digital twin operational feedback.

[0048] Example 2: like Figure 1 As shown, this embodiment takes the shared bottleneck re-entry scheduling of a four-roller rolling mill in a wind turbine tower manufacturing workshop as an example. The reinforcement learning training end is implemented using a Python program, and the digital twin runtime end is implemented using a Unity environment. The Python end first generates an order scenario based on the order size, delivery pressure, and random seed. The scenario includes the process route of the order number, product number, pre-processing process, initial bottleneck processing process, intermediate processing process, re-entry bottleneck processing process, and subsequent processing process, as well as the processing time of each process, order delivery date, and order release time.

[0049] like Figures 2-7As shown, after loading the order scene in Unity, orders and products are dynamically created according to the order release time. Products sequentially undergo pre-processing, initial bottleneck processing, intermediate processing, re-entry bottleneck processing, and post-processing. The initial bottleneck processing and re-entry bottleneck processing use the same set of shared bottleneck equipment. In this embodiment, the shared bottleneck equipment set is a four-roll mill set; the initial bottleneck processing corresponds to the rolling process, and the re-entry bottleneck processing corresponds to the rounding process. After completing the intermediate processing, the product returns to the same set of four-roll mills for re-entry bottleneck processing. Products processed by the initial bottleneck and products processed by the re-entry bottleneck enter the same shared candidate queue.

[0050] When the shared candidate queue is not empty and at least one shared bottleneck device is idle, the Unity client triggers a dispatch request. The Unity client generates the current 12-dimensional global aggregation state and generates an 8-dimensional real-time snapshot of the candidate products in the shared candidate queue. At the same time, it sends the event number, current time, candidate product number, global aggregation state, and real-time product snapshot to the Python client.

[0051] After receiving the dispatch request, the Python side inputs the 12-dimensional global aggregate state into the soft actor-critic model and outputs 8-dimensional dynamic weights. The Python side then returns the event number, model version, and 8-dimensional dynamic weights to the Unity side.

[0052] After receiving the 8-dimensional dynamic weights, Unity decodes the dynamic weights and real-time snapshots of candidate products to obtain a ranking of the candidate products. Based on this ranking, the candidate products are then assigned to idle shared bottleneck devices. After the action is executed, Unity continues the production process until the next dispatch decision point or scene termination.

[0053] During the process, the Unity client tracks the number of newly added and canceled orders, the time progress, and the idle capacity area of ​​the four-roll mill within the current decision interval. It also simultaneously records the number of four-roll mills, cumulative busy time, cumulative idle area, current four-roll mill utilization rate, current simulation time, number of completed orders, and number of canceled orders. Subsequently, the Unity client sends the new status, runtime increment, and scene termination flag to the Python client.

[0054] The Python side calculates the immediate reward based on the runtime increment and writes the current global aggregation state, 8-dimensional dynamic weights, immediate reward, new global aggregation state, and scene termination flag into the experience sample cache. The Python side samples the experience sample cache from the experience sample cache and updates the parameters of the soft actor-critic model.

[0055] Example 3: In this embodiment, each dispatch request is assigned a unique event number. After sending a dispatch request, the Unity client records the event number and the current scene state. The Python client returns the action response with the same event number. The Unity client executes the action only if the event number matches and the event has not yet been executed. If the Unity client receives the same action response with the same event number repeatedly due to a network error, the dispatch will not be repeated.

[0056] When communication is interrupted, the Unity and Python sides resume training based on the most recently confirmed event number, scene state, and model version. If an event has been executed on the Unity side but the Python side has not confirmed its inclusion in the experience samples, the run feedback can be resent based on the event log; if an event has not been executed, the dispatch request is resent.

[0057] In training mode, the Python side allows the reinforcement learning model to explore and update parameters. The Unity side returns the number of new and canceled orders, time progress, and idle capacity area of ​​the four-roll mill, which are used to calculate immediate rewards and write them to the experience replay cache. In evaluation mode, the Python side loads the specified model version and outputs dynamic weights in a deterministic manner; the Unity side performs a full scene simulation and calculates the maximum completion time, four-roll mill utilization, number of canceled orders or cancellation rate, and auxiliary evaluation metrics, but the Python side does not update model parameters. By isolating the training and evaluation modes, comparable evaluations can be achieved under different algorithms, order sizes, and delivery pressure scenarios.

Claims

1. A reinforcement learning training method for shared bottleneck reentrant scheduling based on digital twin operational feedback, characterized in that, Includes the following steps: S1, an order scenario generated by the reinforcement learning training end for a shared bottleneck reentrancy manufacturing system; S2 loads order scenarios from the digital twin runtime and dynamically creates orders and products according to the order release time; S3 is driven by the digital twin operation terminal to advance the production process and triggers a dispatch decision request when the shared candidate queue is not empty and there are idle devices in the shared bottleneck device. S4 generates a global aggregated state and real-time snapshots of candidate products from the digital twin runtime and sends them to the reinforcement learning training end; S5, which outputs dynamic weights from the reinforcement learning training end based on the global aggregate state; S6, the digital twin operation terminal executes candidate product dispatching actions and promotes the production process based on dynamic weights; S7, the digital twin runtime terminal calculates the runtime increment after the dispatch action is executed, and sends the new state and runtime increment to the reinforcement learning training terminal; S8. The reinforcement learning training end calculates the instant reward based on the running increment, and updates the reinforcement learning model based on the global aggregate state, dynamic weights, instant reward, and new state.

2. The shared bottleneck reentrant scheduling reinforcement learning training method based on digital twin operational feedback as described in claim 1, characterized in that, The digital twin runtime dynamically creates orders and products according to the order release time, and controls the products to sequentially enter the pre-processing process, the first bottleneck processing process, the intermediate processing process, the re-entry bottleneck processing process, and the subsequent processing process according to the current process status of the products. The shared bottleneck re-entry manufacturing system includes the pre-processing process, the first bottleneck processing process, the intermediate processing process, the re-entry bottleneck processing process, and the subsequent processing process connected in sequence. The first bottleneck processing process and the re-entry bottleneck processing process use the same group of shared bottleneck equipment. After completing the intermediate processing process, the product returns to the shared bottleneck equipment group to perform the re-entry bottleneck processing process.

3. The shared bottleneck reentrant scheduling reinforcement learning training method based on digital twin operational feedback as described in claim 1, characterized in that, During the training process, reinforcement learning training objectives revolve around the following three core optimization metrics: by This represents minimizing the maximum completion time, where Indicates the maximum completion time; by This represents maximizing the utilization of shared bottleneck equipment, where Indicates the utilization rate of shared bottleneck equipment; by This represents minimizing the order cancellation rate, where Indicates the number of canceled orders. This indicates the initial number of orders.

4. The shared bottleneck reentrant scheduling reinforcement learning training method based on digital twin operational feedback as described in claim 1, characterized in that, The reinforcement learning training end adopts a soft actor-critic reinforcement learning model, and outputs dynamic weights with fixed dimensions according to the global aggregation state. The dynamic weights are used by the digital twin running end to decode actions and assign tasks to candidate products. The soft actor-critic reinforcement learning model includes a policy network, a first Q-network, a second Q-network, and a temperature parameter. The policy network takes the global aggregated state as input and maps it to the mean and logarithmic standard deviation of actions in each dimension through a multi-layer fully connected network. Dynamic weights for the fixed dimensions are obtained through reparameterized sampling. The first Q-network and the second Q-network estimate the state-action value respectively, and take the smaller of the two to suppress overestimation. The temperature parameter is used to adjust the strength of policy entropy regularization, which is automatically adjusted by the reinforcement learning training end during training by minimizing the entropy objective function or set to a preset fixed value. During the training phase, the policy network outputs dynamic weights using random sampling to maintain exploration, and during the evaluation phase, it outputs dynamic weights using the mean to achieve deterministic assignment. The digital twin runtime scores and sorts the candidate products according to the dynamic weights, and then allocates the candidate products to idle shared bottleneck devices according to the sorting results.

5. The shared bottleneck reentrant scheduling reinforcement learning training method based on digital twin operational feedback as described in claim 1, characterized in that, The operational increment includes the number of newly cancelled orders, the time progress, and the idle capacity area of ​​shared bottleneck equipment between the execution of the dispatch action and the next dispatch decision point; the operational increment also includes at least one of the following: the number of shared bottleneck equipment, the cumulative busy time of shared bottleneck equipment, the cumulative idle area of ​​shared bottleneck equipment, the current utilization rate of shared bottleneck equipment, the current simulation time or the maximum completion time, the number of completed orders, and the number of cancelled orders.

6. The shared bottleneck reentrant scheduling reinforcement learning training method based on digital twin operational feedback according to claim 5, characterized in that, Instant rewards are calculated as follows: Based on the number of newly cancelled orders Compared with the initial number of orders The ratio yields the increment of cancellation loss: ； According to the Time advance of each decision interval Average processing workload per unit order The ratio yields the time-advancing loss increment: ； Based on the idle capacity area of ​​shared bottleneck equipment Number of shared bottleneck devices and average processing workload per unit order The ratio of the products yields the increment of idle loss: ； Average processing workload per unit order Total processing workload for the scene Compared with the initial number of orders The ratio, that is ; The immediate reward is determined based on the weighted negative value of the cancellation loss increment, the time advancement loss increment, and the idle loss increment: ； in , and These are non-negative weighting coefficients.

7. The shared bottleneck reentrant scheduling reinforcement learning training method based on digital twin operational feedback as described in claim 6, characterized in that, The first The start time of each decision interval The time when the dispatch decision request is triggered by the digital twin operating terminal, and the termination time. After the digital twin operation terminal executes the dispatch action, the production process is advanced to the earliest moment when any of the following conditions are met: a shared bottleneck device is idle again and the shared candidate queue is not empty, all orders are completed, or the preset simulation termination time is reached.

8. The shared bottleneck reentrant scheduling reinforcement learning training method based on digital twin operational feedback as described in claim 1, characterized in that, The reinforcement learning training end constructs experience samples from the current global aggregation state, dynamic weights, immediate rewards, new global aggregation state, and scene termination flags, and writes these experience samples into the experience replay cache for updating the soft actor-critic reinforcement learning model.

9. The shared bottleneck reentrant scheduling reinforcement learning training method based on digital twin operational feedback as described in claim 8, characterized in that, The digital twin runtime and the reinforcement learning training end interact with each other through a communication protocol that includes event number, current time, global aggregation state, candidate product number, real-time snapshot of candidate product, model version, dynamic weight, dispatch sequence and runtime increment. Idempotency verification is performed through event number. When communication between the digital twin runtime and the reinforcement learning training end is interrupted, the training process is resumed based on the most recently confirmed event number, scene state and model version.

10. The shared bottleneck reentrant scheduling reinforcement learning training method based on digital twin operational feedback according to claim 1, characterized in that, The training and evaluation processes are isolated from each other. The training process allows for the updating of reinforcement learning model parameters, while the evaluation process loads specified model parameters and performs scheduling policy evaluation without updating the model parameters.

Images