A multi-agent reinforcement learning-based job shop scheduling system and method

Abstract

This invention relates to the field of data processing technology and discloses a workshop scheduling system and method based on multi-agent reinforcement learning. The system includes a task database, a global business evaluation module, a distributed scheduling control module, and a feedback correction module. The system determines the degree of deviation of the sequence of tasks to be processed from the expected delivery date of the production order and generates a global order value gap matrix. The feedback correction module matches the business default penalty feature values ​​associated with the tasks and uses linear weighted logic and dynamic adaptive factors to generate the final credit allocation signal for adjusting the network weights of the strategy. The system monitors the rate of change of the data variance of the global order value gap matrix. When the rate of change exceeds a preset threshold, the dynamic adaptive factor is reduced to decrease the weight ratio of the local efficiency reward signal. This invention transforms the overall order delivery risk into a constraint boundary for intervening in the convergence of the underlying strategy, eliminating the delivery target deviation caused by the scheduling agent pursuing local physical efficiency.

Description

Technical Field

[0001] This invention relates to a workshop scheduling system and method based on multi-agent reinforcement learning, belonging to the field of data processing technology. Background Technology

[0002] Currently, for high-concurrency processing scenarios with diverse task attributes, establishing an efficient data processing system is key to improving overall delivery efficiency. Discrete manufacturing enterprises generally adopt a distributed multi-agent architecture, configuring independent intelligent decision-making nodes for each processing unit and using reinforcement learning networks to handle local resource allocation tasks. This approach demonstrates a certain degree of flexibility in responding to dynamically changing production requests. However, when faced with the pressure of large-scale emergency order insertions or high-priority task concurrency, existing methods expose deep-seated technical limitations. Discrete manufacturing processes include management attributes such as order delivery deadlines and default penalty bases. When setting the feedback reward logic for multi-agent reinforcement learning, existing systems generally use physical dimension status data such as equipment idle rate or process flow time. This design logic, when handling high-concurrency tasks, causes each intelligent agent module to prioritize distributing tasks with simple processes or short processing times to improve single-machine utilization. When faced with an order matrix with delivery constraints, local self-interest decision-making logic cuts off the overall scheduling intention of the business management layer, and high-default-risk orders are repeatedly delayed in the queuing sequence. Due to the disconnect between the underlying data flow logic and the upper-level management objectives, the system lacks reliable monitoring capabilities under extreme conditions.

[0003] Addressing the aforementioned issues, improvement approaches such as increasing computing resource stacking or adopting a single global reward function face practical challenges. On the one hand, simply increasing computing power cannot eliminate the dimensionality explosion resistance under massive concurrency; on the other hand, static global reward signals are difficult to adapt to the high-frequency dynamic evolution of management data, easily leading to non-stationarity in policy training. For example, Chinese invention patent CN119494504B discloses a distributed flexible job shop scheduling method and system based on dual deep reinforcement learning and multi-layer intelligent agents. Although this technology utilizes a hierarchical intelligent agent architecture to achieve the distribution of the flexible job shop problem... While the decision-making process is layered, its underlying feedback mechanism is highly dependent on the physical efficiency of a single dimension, such as machine load and processing time. Under extreme pressure conditions such as large-scale emergency order insertion or high-priority task concurrency, the lack of quantitative perception of overall operational attributes such as penalties for commercial defaults in existing technologies leads to each intelligent agent prioritizing the distribution of simple tasks in order to pursue single-machine utilization indicators when resources are limited. This disconnects the overall scheduling intention of the business management layer. This non-stationary problem caused by the disconnect between the underlying action evaluation function and the upper-level management objectives can easily result in high-default-risk orders being repeatedly delayed in the queuing sequence.

[0004] Therefore, how to establish an asymmetric credit allocation architecture with management value perception capabilities, so that local resource decision-making actions converge in real time to the global delivery target, and solve the strategy oscillation and delivery risk in large-scale discrete task processing, becomes the technical problem to be solved by this invention. Summary of the Invention

[0005] To address the problems in the background art, the technical solution of this invention is as follows: A workshop scheduling system based on multi-agent reinforcement learning, comprising a task database, a global business evaluation module, a distributed scheduling control module, and a feedback correction module: The global business assessment module is used to obtain the sequence of tasks to be processed and the expected delivery of production orders, calculate the degree of deviation of the delivery date of the sequence of tasks to be processed relative to the expected delivery date of production orders, and generate a global order value gap matrix. The distributed scheduling control module includes multiple scheduling agent modules. Each scheduling agent module is used to obtain the availability status and characteristics of the tasks to be processed of the corresponding resource nodes, determine the scheduling actions through the policy network, and generate local performance reward signals based on the degree of improvement of the resource node processing efficiency by the scheduling actions. The feedback correction module is used to extract the task identifier pointed to by the scheduling action and match the business default penalty feature value associated with the task identifier from the global order value gap matrix. The feedback correction module is used to perform a weighted summation of the local performance reward signal and the business default penalty feature value using linear weighted logic to generate the final credit allocation signal used to adjust the weights of the strategy network. The dynamic adaptive factor used to balance processing performance and business constraints ranges from 0.8 to 0.9. The feedback correction module monitors the rate of change of the data variance of the global order value gap matrix and, when the rate of change of the data variance exceeds a preset threshold of 0.15, lowers the value of the dynamic adaptive factor to reduce the weight ratio of the local performance reward signal in the final credit allocation signal. Each scheduling agent module is used to adjust the parameter gradient of the policy network based on the final credit allocation signal and generate node scheduling instructions corrected by business constraints.

[0006] Preferably, when generating the global order value gap matrix, the global business assessment module obtains the remaining delivery time period and the preset default penalty base for each pending order in the task database, and uses the product of the remaining delivery time period and the preset default penalty base as the basis for determining the value gap weight of each task node.

[0007] Preferably, the system adopts an asynchronous parameter update architecture; the global business evaluation module is used to monitor the distribution variance of the global order value gap matrix; when the rate of change of the distribution variance exceeds the preset task deviation fluctuation threshold, the global business evaluation module sends a recalculation instruction to the feedback correction module to recalibrate the credit allocation weight of each scheduling agent module; during the steady-state window period when no recalculation instruction is received, each scheduling agent module calls the historical credit allocation coefficient in the local memory to correct the scheduling action.

[0008] Preferably, the feedback correction module further includes a marginal contribution calculation unit; the marginal contribution calculation unit is used to calculate the marginal contribution of each scheduling action to the convergence of the global order value gap matrix based on the resource node task matching vector reported by each scheduling agent module, and convert the marginal contribution into an asymmetric credit adjustment signal to suppress logical conflicts between scheduling agent modules.

[0009] Preferably, the scheduling agent module is used to monitor the congestion length ratio of the task queue to be processed on the corresponding resource node, and when the congestion length ratio shows an increasing trend, it synchronously reduces the value of the dynamic adaptive factor to improve the scheduling priority of the scheduling agent module in handling tasks with associated high default penalty bases.

[0010] Preferably, the global business evaluation module is used to update the weight distribution in the global order value gap matrix based on the order priority change information in the task database; the feedback correction module sets the dynamic adaptive factor to a preset lower limit value in the first control cycle after the order priority change, forcing the search space of the strategy network to converge to the new business constraint boundary.

[0011] Preferably, the feedback correction module is used to normalize the extracted commercial default penalty feature value so that the commercial default penalty feature value and the local performance reward signal are on the same order of magnitude; the feedback correction module obtains the expected action benefit value of each scheduling agent module and uses the normalized commercial default penalty feature value to perform nonlinear modulation on the expected action benefit value.

[0012] Preferably, each scheduling agent module includes a logic simulation component, which is used to pre-simulate the correlation effect between the characteristics of the task to be processed and the scheduling action, and calculate the virtual contribution of the pre-simulation result to the global business objective; and only when the virtual contribution is positive, the scheduling agent module confirms the corresponding scheduling action as a valid scheduling decision.

[0013] Preferably, the system is connected to the production management center via a data bus and receives intervention level signals from the production management center; the feedback correction module adjusts the weight of the dynamic adaptive factor in a step-by-step manner according to the intervention level signal to constrain the scheduling logic of each resource node.

[0014] A workshop scheduling method based on multi-agent reinforcement learning, implemented by a workshop scheduling system based on multi-agent reinforcement learning, includes the following steps: Step S1: The global business assessment module obtains the sequence of tasks to be processed and the expected delivery of production orders, calculates the degree of deviation of the delivery date of the sequence of tasks to be processed relative to the expected delivery date of production orders, and generates a global order value gap matrix. In step S2, each scheduling agent module obtains the availability status and characteristics of the tasks to be processed of the corresponding resource nodes, determines the scheduling action through the policy network, and generates a local performance reward signal based on the degree of improvement of the resource node processing efficiency by the scheduling action. Step S3: The feedback correction module extracts the task identifier pointed to by the scheduling action and matches the commercial default penalty feature value associated with the task identifier from the global order value gap matrix; In step S4, the feedback correction module performs a weighted summation of the local efficiency reward signal and the commercial default penalty feature value using linear weighted logic to generate the final credit allocation signal used to adjust the weights of the strategy network; wherein, the value range of the dynamic adaptive factor used to balance physical efficiency and commercial constraints is 0.8 to 0.9. Step S5: The feedback correction module monitors the data variance change rate of the global order value gap matrix, and when the data variance change rate exceeds the preset threshold of 0.15, it lowers the value of the dynamic adaptive factor to reduce the weight ratio of the local performance reward signal in the final credit allocation signal. In step S6, each scheduling agent module adjusts the parameter gradient of the policy network according to the final credit allocation signal and generates node scheduling instructions corrected by business constraints.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. In multi-agent reinforcement learning workshop scheduling, the global business evaluation module and the distributed feedback update module collaborate to directly transform the overall order delivery risk data into the constraint boundary for the convergence of the underlying intervention strategy. This eliminates the global delivery target deviation caused by the distributed scheduling agent blindly pursuing local physical efficiency under complex working conditions. Under this mechanism, the system can generate a global order value gap matrix in real time, representing the overall business deviation state, based on the remaining delivery time cycle and default penalty base of each production order. This matrix is ​​then mapped to management intervention penalty items for specific scheduling actions. This data flow path forces the reward function space of the underlying intelligent decision-making nodes to be forcibly modified by the overall management intent when executing task distribution, thereby establishing a necessary connection between the scheduling intent of the business management layer and the action sequence of processing nodes at the logical level.

[0016] 2. The strategy evaluation updater constructs an asymmetric reward distribution topology by performing linear weighted calculations on the local physical base reward value and the commercial default penalty feature value. This solves the policy non-stationarity problem commonly found in multi-agent reinforcement learning in workshop scheduling scenarios. When the system faces massive concurrent orders or resource preemption conflicts, the global commercial evaluation module monitors the data variance changes of the global order value gap matrix and dynamically reduces the weight ratio of the local physical base reward value in the credit allocation signal. This feedback adjustment mechanism forcibly deprives the processing equipment of the optimization authority for the single physical operation efficiency under resource-constrained conditions, causing the system weights to spontaneously converge towards the strategy space that minimizes the global commercial default risk, ensuring the certainty of the flow of core high-value orders in a dynamic disturbance environment. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the overall implementation of the multi-agent reinforcement learning-based workshop scheduling method of the present invention. Figure 2 This is a decision path diagram for the scheduling instructions in the state monitoring and logic pre-simulation of the present invention.

[0018] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0020] A workshop scheduling system based on multi-agent reinforcement learning includes a task database, a global business evaluation module, a distributed scheduling control module, and a feedback correction module. The global business assessment module is used to obtain the sequence of tasks to be processed and the expected delivery of production orders, calculate the degree of deviation of the delivery date of the sequence of tasks to be processed relative to the expected delivery date of production orders, and generate a global order value gap matrix. The distributed scheduling control module includes multiple scheduling agent modules. Each scheduling agent module is used to obtain the availability status and characteristics of the tasks to be processed of the corresponding resource nodes, determine the scheduling actions through the policy network, and generate local performance reward signals based on the degree of improvement of the resource node processing efficiency by the scheduling actions. The feedback correction module is used to extract the task identifier pointed to by the scheduling action and match the business default penalty feature value associated with the task identifier from the global order value gap matrix. The feedback correction module is used to perform a weighted summation of the local performance reward signal and the business default penalty feature value using linear weighted logic to generate the final credit allocation signal used to adjust the weights of the strategy network. The dynamic adaptive factor used to balance processing performance and business constraints ranges from 0.8 to 0.9. The feedback correction module monitors the rate of change of the data variance of the global order value gap matrix and, when the rate of change of the data variance exceeds a preset threshold of 0.15, lowers the value of the dynamic adaptive factor to reduce the weight ratio of the local performance reward signal in the final credit allocation signal. Each scheduling agent module is used to adjust the parameter gradient of the policy network based on the final credit allocation signal and generate node scheduling instructions corrected by business constraints.

[0021] Preferably, when generating the global order value gap matrix, the global business assessment module obtains the remaining delivery time period and the preset default penalty base for each pending order in the task database, and uses the product of the remaining delivery time period and the preset default penalty base as the basis for determining the value gap weight of each task node.

[0022] Preferably, the system adopts an asynchronous parameter update architecture; the global business evaluation module is used to monitor the distribution variance of the global order value gap matrix; when the rate of change of the distribution variance exceeds the preset task deviation fluctuation threshold, the global business evaluation module sends a recalculation instruction to the feedback correction module to recalibrate the credit allocation weight of each scheduling agent module; during the steady-state window period when no recalculation instruction is received, each scheduling agent module calls the historical credit allocation coefficient in the local memory to correct the scheduling action.

[0023] Preferably, the feedback correction module further includes a marginal contribution calculation unit; the marginal contribution calculation unit is used to calculate the marginal contribution of each scheduling action to the convergence of the global order value gap matrix based on the resource node task matching vector reported by each scheduling agent module, and convert the marginal contribution into an asymmetric credit adjustment signal to suppress logical conflicts between scheduling agent modules.

[0024] Preferably, the scheduling agent module is used to monitor the congestion length ratio of the task queue to be processed on the corresponding resource node, and when the congestion length ratio shows an increasing trend, it synchronously reduces the value of the dynamic adaptive factor to improve the scheduling priority of the scheduling agent module in handling tasks with associated high default penalty bases.

[0025] Preferably, the global business evaluation module is used to update the weight distribution in the global order value gap matrix based on the order priority change information in the task database; the feedback correction module sets the dynamic adaptive factor to a preset lower limit value in the first control cycle after the order priority change, forcing the search space of the strategy network to converge to the new business constraint boundary.

[0026] Preferably, the feedback correction module is used to normalize the extracted commercial default penalty feature value so that the commercial default penalty feature value and the local performance reward signal are on the same order of magnitude; the feedback correction module obtains the expected action benefit value of each scheduling agent module and uses the normalized commercial default penalty feature value to perform nonlinear modulation on the expected action benefit value.

[0027] Preferably, each scheduling agent module includes a logic simulation component, which is used to pre-simulate the correlation effect between the characteristics of the task to be processed and the scheduling action, and calculate the virtual contribution of the pre-simulation result to the global business objective; and only when the virtual contribution is positive, the scheduling agent module confirms the corresponding scheduling action as a valid scheduling decision.

[0028] Preferably, the system is connected to the production management center via a data bus and receives intervention level signals from the production management center; the feedback correction module adjusts the weight of the dynamic adaptive factor in a step-by-step manner according to the intervention level signal to constrain the scheduling logic of each resource node.

[0029] A workshop scheduling method based on multi-agent reinforcement learning includes the following steps: Step S1: The global business assessment module obtains the sequence of tasks to be processed and the expected delivery of production orders, calculates the degree of deviation of the delivery date of the sequence of tasks to be processed relative to the expected delivery date of production orders, and generates a global order value gap matrix. In step S2, each scheduling agent module obtains the availability status and characteristics of the tasks to be processed of the corresponding resource nodes, determines the scheduling action through the policy network, and generates a local performance reward signal based on the degree of improvement of the resource node processing efficiency by the scheduling action. Step S3: The feedback correction module extracts the task identifier pointed to by the scheduling action and matches the commercial default penalty feature value associated with the task identifier from the global order value gap matrix; In step S4, the feedback correction module performs a weighted summation of the local efficiency reward signal and the commercial default penalty feature value using linear weighted logic to generate the final credit allocation signal used to adjust the weights of the strategy network; wherein, the value range of the dynamic adaptive factor used to balance physical efficiency and commercial constraints is 0.8 to 0.9. Step S5: The feedback correction module monitors the data variance change rate of the global order value gap matrix, and when the data variance change rate exceeds the preset threshold of 0.15, it lowers the value of the dynamic adaptive factor to reduce the weight ratio of the local performance reward signal in the final credit allocation signal. In step S6, each scheduling agent module adjusts the parameter gradient of the policy network according to the final credit allocation signal and generates node scheduling instructions corrected by business constraints.

[0030] Example 1: In a discrete manufacturing workshop with 50 processing units and handling 200 concurrent tasks, when the system receives 5 urgent task flows carrying penalty coefficients and with less than 24 hours remaining in the delivery cycle, the scheduling logic using a static reward function drives each agent to prioritize low-priority tasks with shorter execution times to maintain equipment utilization. This causes the urgent tasks to lag in the queuing sequence, resulting in an increase in the characteristic variance of the global order value gap matrix over three consecutive sampling periods. The global business evaluation module extracts the remaining delivery cycle of the urgent tasks and the preset penalty base, and calculates the business penalty characteristic value in the global order value gap matrix. The global order value gap matrix is ​​a two-dimensional algebraic matrix with defined boundaries. Row indices correspond to the physical numbers of distributed processing units deployed within the workshop, while column indices correspond to the serial numbers of currently resident concurrent production orders in the task database. The values ​​of elements at the intersections of rows and columns directly reflect the overdue penalty quantification value calculated by the system when the processing unit in the corresponding row undertakes the production task in the corresponding column. The feedback correction module matches the task identifiers in the action sequences generated by each scheduling agent module, determines the order dependency data, and the strategy evaluation updater updates the local performance reward signal. Characteristic of penalties for commercial default Implementing linear weighting, the feedback correction module will dynamically adapt the factor. Adjust to 0.2 to reduce local performance reward signals. The weighting ratio is adjusted to suppress decision-making biases that solely pursue equipment utilization.

[0031] To prevent the Markov decision process from becoming unstable and gradient explosion caused by sudden changes in the weights of the underlying reward function, the policy evaluation updater simultaneously activates a gradient truncation mechanism based on time difference error when executing the instruction to lower the dynamic adaptive factor α. This mechanism forcibly constrains the L2 norm of the policy network parameter update gradient within a preset pruning threshold envelope, ensuring the continuity and physical consistency of the policy evolution direction as the neural network's search space converges toward the new business constraint boundary. Each scheduling agent module adjusts the parameter gradient based on the final credit allocation signal, generating node scheduling instructions corrected by business constraints. Resource nodes stop accessing low-value tasks and prioritize urgent tasks. The variance change rate of the global order value gap matrix drops from 0.22 to below 0.15, and urgent tasks are completed and put into storage within the specified delivery cycle. This logic of intervening in the underlying policy convergence through business constraint data establishes a deterministic connection between the management scheduling intent and the action sequence of surface nodes, enabling the system to spontaneously shift from physical efficiency optimization to management goal alignment.

[0032] Example 2: In a discrete manufacturing simulation platform with 50 processing nodes and handling 200 concurrent tasks, the experimental data comes from the production task management system flow data. Gaussian white noise with a signal-to-noise ratio of 20dB is actively superimposed at the input to simulate electromagnetic interference during sensor acquisition. The sampling period is set to balance the real-time data acquisition and system computational load. When the arrival frequency of the task flow exceeds 50 items per minute, the sampling frequency is adjusted to the upper limit of 10Hz, and otherwise it is maintained at the base frequency of 1Hz to maintain the timeliness of system decision-making. To verify the practical effectiveness of a workshop scheduling system and method based on multi-agent reinforcement learning, the present invention sample group and control group are set up. Control group 1 adopts a static reward function scheduling method based on equipment idle rate, and control group 2 is a partially missing control group with the global business evaluation module removed.

[0033] The experiment verified the scheduling logic for 500 concurrent tasks with differentiated default penalty bases. Initially, the congestion length ratio of the task queues on each resource node ranged from 0.35 to 0.42. The sample group of this invention monitored a data variance variation rate of 0.18 for the global order value gap matrix, exceeding the preset threshold of 0.15. The feedback correction module dynamically adjusted the adaptive factor based on this variation rate. The value was reduced from 0.88 to 0.81 to reduce the local performance reward signal. The weighting proportion in the final credit allocation signal; at this point, the strategy evaluation updater increases the commercial default penalty feature value. The calculation weights are used to generate node scheduling instructions by the scheduling agent module based on the corrected parameter gradient. This drives resource nodes that tend to handle simple tasks to process complex orders with high default penalty bases. This achieves a 5.2% drop in local physical performance indicators in exchange for suppressing global commercial overdue risk. Observational data shows that, under a 12.5% ​​emergency order insertion disturbance, the on-time delivery rate of the sample group in this invention remained at 94.6%, while the on-time delivery rate of control group one dropped to 72.3%. Control group two, lacking asymmetric credit allocation signal adjustment, exhibited strategy oscillations, with a delivery rate of 78.1%. The experimental results demonstrate that the characteristic value of commercial default penalty... With local performance reward signals The linear weighted coupling generates synergistic effects, resolving the technical contradiction between optimizing equipment utilization and aligning with system-level delivery goals, and targeting dynamic adaptive factors. Stress tests are performed on the boundary values ​​of the range when When the value is within the limited range of 0.8 to 0.9, the system's comprehensive decision evaluation parameters The system exhibits a monotonic adjustment trend as the intensity of business constraints increases, demonstrating that it can achieve convergence towards business objectives while ensuring equipment utilization; once... Beyond the performance inflection point of 0.92, the system's sensitivity to business constraints saturates, leading to an increase in the number of overdue high-value orders; while In the out-of-range control group with values ​​below 0.75, the local efficiency of resource nodes deteriorated, and the single-machine utilization rate dropped from 88.5% to below 61.2%. Due to excessive suppression of physical efficiency, the overall production capacity supply was insufficient, which in turn led to the risk of secondary default.

[0034] Example 3: This example combines Figures 1 to 2 This paper describes a workshop scheduling system and method based on multi-agent reinforcement learning, such as... Figure 1As shown, step S1, the global business evaluation module, obtains the sequence of tasks to be processed and the expected delivery of production orders. It calculates the deviation of the delivery date of the sequence of tasks to be processed from the expected delivery date of production orders to generate a global order value gap matrix. Step S2, each scheduling agent module, obtains the availability status of the corresponding resource nodes and the characteristics of the tasks to be processed. It determines the scheduling action through the policy network and generates a local performance reward signal based on the degree of improvement of the resource node processing efficiency by the scheduling action. Then, step S3, the feedback correction module, extracts the task identifier pointed to by the scheduling action and matches the business default penalty feature value associated with the task identifier from the global order value gap matrix. Finally, step S4, the feedback correction module, performs linear weighted summation. The logic performs a weighted summation of the local performance reward signal and the commercial default penalty feature value to generate the final credit allocation signal used to adjust the weights of the strategy network. The dynamic adaptive factor used to balance physical efficiency and commercial constraints has a value range of 0.8 to 0.9. Then, it proceeds to step S5, the feedback correction module monitors the data variance change rate of the global order value gap matrix, and when the data variance change rate exceeds the preset threshold of 0.15, it lowers the value of the dynamic adaptive factor to reduce the weight ratio of the local performance reward signal in the final credit allocation signal. Finally, in step S6, each scheduling agent module adjusts the parameter gradient of the strategy network according to the final credit allocation signal and generates the node scheduling instructions after commercial constraint correction.

[0035] like Figure 2 As shown, the specific generation mechanism of scheduling instructions unfolds through a path structure with judgment conditions. The scheduling instruction generation decision path triggers the global business assessment module's state extraction, advancing to the assessment node to determine whether the data variance change rate exceeds a preset threshold of 0.15. When the judgment feature shows that it crosses the preset threshold limit, the process is directed to the left processing sequence, executing the global value gradient full calibration, lowering the dynamic adaptive factor value, and increasing the weight of the business default penalty feature value. When the judgment feature shows that it is within the steady-state fluctuation envelope, the process is directed to the right processing sequence, executing the asynchronous driving steady-state window period, calling the local storage historical credit allocation coefficient, and maintaining the local efficiency reward signal to optimize the weight. The above two parallel branch sequences converge downwards to output to generate the final credit allocation signal, mapping the business management intent to the policy network objective function. After that, the process enters the next judgment node to execute the logic simulation component pre-play: whether the virtual contribution is positive. If the output is yes, it is confirmed as a valid scheduling decision, generating the node scheduling instruction after business constraint correction. If the output is no, it suppresses the action distribution strategy oscillation to avoid resource node mutual exclusion and deadlock.

[0036] Example 4: In a production management scenario involving 1000 sets of historical order delivery deviation characteristic sequences, the system performs parameter calibration procedures to determine the preset threshold for the asynchronous event-driven mechanism. In an offline environment, the global business evaluation module sets each resource node to a 55% load condition and continuously monitors the characteristic variance change rate of the global order value gap matrix over 72 hours. The collected change rate data is fitted with a normal distribution, and its distribution mean is calculated. with standard deviation , and select As the critical value for determining whether a business management indicator crosses the steady-state window, the critical value determined under this baseline load condition is 0.15. When the policy network executes decisions, its state space specifically includes a feature vector composed of the current idling time of the corresponding resource node and the remaining number of processes for the task to be processed. Its action space is defined as the probability vector for selecting the corresponding target task serial number. The policy network adopts a multilayer perceptron architecture at its bottom layer, which consists of an input layer that receives the state space features, two hidden layers that use the ReLU activation function, and an output layer that uses the Soft max function to output the probability of deterministic scheduling actions, so that the available state of the equipment and the task features are transformed into specific scheduling instructions through forward propagation.

[0037] When the global business evaluation module identifies that the rate of change of the current feature variance exceeds 0.15, the policy evaluation updater calculates the local physical basis reward value. Characteristic of penalties for commercial default The linear dynamic compensation relationship is determined, and the comprehensive decision evaluation parameters are established. , specifically The calculation formula is as follows: ,in, For comprehensive decision-making evaluation parameters, As a dynamic adaptive factor, As an initial local physical performance evaluation index, To address the causal misalignment issue of local agent responses lagging behind global business penalties in asynchronous update architectures, a global shared memory area with timestamp verification is deployed within the system to represent the global value change gradient. The global business evaluation module calculates the current... Then, it is immediately overwritten into the shared memory area. When each scheduling agent module starts each independent asynchronous parameter update cycle, it forcibly triggers a read interrupt for the shared memory area. After extracting the latest comprehensive decision evaluation parameters, it executes the subsequent gradient evolution action, thereby establishing a deterministic causal timing handshake closed loop between the asynchronous computing framework and the real-time intervention penalty. The policy evaluation updater then uses the obtained comprehensive decision evaluation parameters... The original local performance reward function is replaced by the objective function imported into the policy network. During the backpropagation of parameter gradients, the scheduling agent module calculates the policy gradient based on the reconstructed objective function. Where θ is the parameter to be optimized in the policy network, and when the dynamic adaptive factor... When the value is reduced stepwise from the initial 0.9 to 0.2, the characteristic value of commercial default penalty... With the increase of control weights on the gradient direction, the neural network weights converge toward the solution space that minimizes the global delivery deviation. When the scheduling system faces a 20% task weight change disturbance, the variance of the loss function of its policy network decreases from 0.45 to 0.08, and the alignment accuracy between the scheduling instruction sequence and the management intention increases by 18.3%.

[0038] Example 5: In a discrete manufacturing deployment scenario with 60 heterogeneous processing units handling concurrent production tasks, the system establishes a local performance reward signal through pre-deployment calibration. The initial value mapping relationship is established, and the pre-calibration method includes a hardware noise reduction procedure. During the initial operation phase after power-on of each resource node, sensors continuously acquire motor operating current and high-frequency vibration data of the spindle at a sampling frequency higher than 50Hz. The system uses a first-order hysteresis filtering algorithm to filter out mechanical abrupt noise and high-frequency electromagnetic interference signals that exceed the preset amplitude threshold. The smoothed signal envelope is extracted as the baseline of the equipment's no-load environment. Using 102 sets of unconstrained production instructions under standard steady-state conditions, the average task completion delay of each resource node is collected as the initial performance benchmark. At the same time, the global business evaluation module extracts order fulfillment records from the business database for the past 12.4 months, and uses the max-min normalization method to convert the remaining delivery time period of each order into a dimensionless delivery urgency weight. This weight is written into the initial feature space of the global order value gap matrix to provide a physical dimension source for management intervention penalty items within the calculation step. Based on the above-collected initial performance benchmark, each scheduling agent module generates local performance reward signals. At that time, an explicit mathematical extraction and transformation rule is adopted: the system extracts the reciprocal of the single-piece processing time of the current controlled resource node for the corresponding processing task, and performs a linear weighted summation with the ratio of the actual fault-free running time of the resource node within the current monitoring sliding window. In this way, the discrete single-machine production time and equipment idle rate and other basic physical operation data are fused and mapped into a scalar performance index with values ​​distributed between 0 and 1.

[0039] When the scheduling system faces fluctuations in external management constraints caused by cross-regional production collaboration, the system adopts an environment-adaptive fine-tuning approach to recalibrate the entropy limits in the asynchronous event-driven mechanism. The global business evaluation module retrieves the business priority distribution characteristics from the current task sequence and calculates the normalized variance fluctuation envelope of these characteristics within a sliding window of 3600 seconds. Specifically, the entropy limit... The calculation formula is as follows: ,in, This is the entropy limit. The mean of variance fluctuation. The standard deviation of variance fluctuation is defined as the characteristic variance rate of change remaining stable at the entropy limit for 3.0 sampling periods. In-neighborhood and dynamically adaptive factor When the first derivative is less than 0.01, the asymmetric credit allocation signal is determined to have entered a steady state.

[0040] Example 6: In a discrete scheduling deployment scenario involving high-dimensional task attributes and mutually exclusive resource nodes, when multiple scheduling agent modules simultaneously target the same bottleneck task, the lack of a means to separate the contribution of independent actions to the global value leads to decision conflicts and drives resource nodes into a deadlock state. This causes the statistical entropy value of the neural network output action distribution to remain above 0.80 for 10 consecutive sampling periods. The system corrects the decision weights through an asymmetric credit adjustment calibration method, and the marginal contribution calculation unit in the feedback correction module retrieves the global state feature vector at the current moment. and the set of joint actions of each scheduling agent module Calculate the joint action value function using the internally stored action value network. The marginal contribution calculation unit determines the first action by performing sampling calculations within the candidate action space. The scheduling agent module fixes the actions of other agent modules. Net contribution at time, asymmetric credit adjustment signal The calculation formula is as follows: ,in, This is an asymmetric credit adjustment signal. For the value function of joint actions, For the first The current policy distribution of each scheduling agent module. This refers to the observed state characteristics of local resource nodes. For action sample index, To exclude the first The action set outside the scheduling agent module, and the internally stored action value network, serves as a centralized evaluator in conjunction with the policy network. Its network topology includes a feature concatenation layer and a three-layer feedforward neural network. During runtime, this network uses a global state feature vector containing the load rates of all resource nodes and a joint action vector generated by all agent modules as determined input features. After hierarchical transmission, it outputs a scalar value representing the expected global long-term return. Furthermore, its internal network weight parameters are independently updated based on the temporal difference error generated in each interaction and the Bellman equation. The feedback correction module calculates the asymmetric credit adjustment signal. As a weight correction factor superimposed on the gradient update step size, it drives the parameter evolution direction of the neural network to approach the global target. This signal is used to quantify the degree of influence of specific actions on the convergence of the global order value gap. Resource nodes spontaneously give up the ineffective preemption of mutually exclusive resources. The scheduling instruction sequence, which was originally in an oscillating state, becomes stable within 50 training cycles, and the task allocation success rate increases from 65.4% to over 91.8%.

[0041] In discrete scheduling scenarios where the concurrent task flow density exceeds a preset throughput threshold, the logic simulation component within each scheduling agent module performs a decision-making pre-simulation by reading the task matching vector of the resource node. The logic simulation component uses a state transition probability calculation model to virtually evaluate the completion probability of the task to be executed, and determines the virtual contribution of the pre-simulation result to the global business objective. When virtual contribution When the value is positive, the scheduling agent module issues the corresponding node scheduling instruction to the physical execution mechanism. The aforementioned state transition probability calculation model is a long short-term memory network model pre-trained under supervised supervision based on the workshop's historical 6-month scheduling log records. Its input receives a joint tensor composed of the equipment operating status feature identifier of the current resource node and the parameter vector of the action to be executed. After forward propagation calculation by internal temporal logic, its output directly maps and outputs the confidence scalar probability value of the relevant task under the action to be executed within the expected delivery time. This serves as the quantitative basis for virtual evaluation. When the production management center adjusts the delivery strategy according to the business operation status, the system receives the intervention level signal through the data bus. The feedback correction module is based on the intervention level signal. Implement step-by-step adjustment, and use dynamic adaptive factors The step adjustment amount is set to By reducing local performance reward signals The weighting coefficients are used to suppress the optimization tendency of resource nodes and ensure that the action sequence of the execution mechanism is controlled by the scheduling instructions of the management center.

[0042] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A workshop scheduling system based on multi-agent reinforcement learning, characterized in that, Includes a task database, a global business assessment module, a distributed scheduling and control module, and a feedback and correction module. The global business assessment module is used to obtain the sequence of tasks to be processed and the expected delivery of production orders, calculate the degree of deviation of the delivery date of the sequence of tasks to be processed relative to the expected delivery date of production orders, and generate a global order value gap matrix. The distributed scheduling control module includes multiple scheduling agent modules. Each scheduling agent module is used to obtain the availability status and characteristics of the tasks to be processed of the corresponding resource nodes, determine the scheduling actions through the policy network, and generate local performance reward signals based on the degree of improvement of the resource node processing efficiency by the scheduling actions. The feedback correction module is used to extract the task identifier pointed to by the scheduling action and match the business default penalty feature value associated with the task identifier from the global order value gap matrix; The feedback correction module is used to perform a weighted summation of the local performance reward signal and the commercial default penalty feature value through linear weighted logic to generate the final credit allocation signal for adjusting the weights of the strategy network. The dynamic adaptive factor used to balance processing performance and commercial constraints has a value range of 0.8 to 0.

9. The feedback correction module is used to monitor the rate of change of the data variance of the global order value gap matrix, and when the rate of change of the data variance exceeds a preset threshold of 0.15, it lowers the value of the dynamic adaptive factor to reduce the weight ratio of the local performance reward signal in the final credit allocation signal. Each scheduling agent module is used to adjust the parameter gradient of the policy network based on the final credit allocation signal and generate node scheduling instructions corrected by business constraints.

2. The workshop scheduling system based on multi-agent reinforcement learning according to claim 1, characterized in that, When generating the global order value gap matrix, the global business assessment module obtains the remaining delivery time period and the preset default penalty base for each pending order in the task database, and uses the product of the remaining delivery time period and the preset default penalty base as the basis for determining the value gap weight of each task node.

3. A workshop scheduling system based on multi-agent reinforcement learning according to claim 1, characterized in that, The system adopts an asynchronous parameter update architecture; the global business evaluation module is used to monitor the distribution variance of the global order value gap matrix; when the rate of change of the distribution variance exceeds the preset task deviation fluctuation threshold, the global business evaluation module sends a recalculation instruction to the feedback correction module to recalibrate the credit allocation weight of each scheduling agent module; during the steady-state window period when no recalculation instruction is received, each scheduling agent module calls the historical credit allocation coefficient in the local memory to correct the scheduling action.

4. A workshop scheduling system based on multi-agent reinforcement learning according to claim 1, characterized in that, The feedback correction module also includes a marginal contribution calculation unit. The marginal contribution calculation unit is used to calculate the marginal contribution of each scheduling action to the convergence of the global order value gap matrix based on the resource node task matching vector reported by each scheduling agent module, and convert the marginal contribution into an asymmetric credit adjustment signal to suppress logical conflicts between scheduling agent modules.

5. A workshop scheduling system based on multi-agent reinforcement learning according to claim 1, characterized in that, The scheduling agent module monitors the congestion length ratio of the task queue to be processed on the corresponding resource node. When the first derivative of the congestion length ratio is greater than zero, it synchronously reduces the value of the dynamic adaptive factor to improve the scheduling priority of the scheduling agent module in handling tasks with associated high default penalty bases.

6. A workshop scheduling system based on multi-agent reinforcement learning according to claim 1, characterized in that, The global business evaluation module updates the weight distribution in the global order value gap matrix based on the order priority change information in the task database. The feedback correction module sets the dynamic adaptive factor to a preset lower limit value in the first control cycle after the order priority change, forcing the search space of the policy network to converge to the new business constraint boundary.

7. A workshop scheduling system based on multi-agent reinforcement learning according to claim 1, characterized in that, The feedback correction module is used to normalize the extracted business default penalty feature value so that the business default penalty feature value and the local performance reward signal are on the same order of magnitude. The feedback correction module obtains the expected action benefit value of each scheduling agent module and uses the normalized business default penalty feature value to perform nonlinear modulation on the expected action benefit value.

8. A workshop scheduling system based on multi-agent reinforcement learning according to claim 1, characterized in that, Each scheduling agent module includes a logic simulation component, which is used to pre-simulate the correlation effect between the characteristics of the task to be processed and the scheduling action, and calculate the virtual contribution of the simulation result to the global business objective; and only when the virtual contribution is positive, the scheduling agent module will confirm the corresponding scheduling action as a valid scheduling decision.

9. A workshop scheduling system based on multi-agent reinforcement learning according to claim 1, characterized in that, The system connects to the production management center via a data bus and receives intervention level signals from the production management center; The feedback correction module, based on the intervention level signal, adjusts the weight of the dynamic adaptive factor in a step-by-step manner to constrain the scheduling logic of each resource node.

10. A workshop scheduling method based on multi-agent reinforcement learning, implemented by the workshop scheduling system based on multi-agent reinforcement learning as described in claim 1, characterized in that, Includes the following steps: Step S1: The global business assessment module obtains the sequence of tasks to be processed and the expected delivery of production orders, calculates the degree of deviation of the delivery date of the sequence of tasks to be processed relative to the expected delivery date of production orders, and generates a global order value gap matrix. In step S2, each scheduling agent module obtains the availability status and characteristics of the tasks to be processed of the corresponding resource nodes, determines the scheduling action through the policy network, and generates a local performance reward signal based on the degree of improvement of the resource node processing efficiency by the scheduling action. Step S3: The feedback correction module extracts the task identifier pointed to by the scheduling action and matches the commercial default penalty feature value associated with the task identifier from the global order value gap matrix; In step S4, the feedback correction module performs a weighted summation of the local efficiency reward signal and the commercial default penalty feature value using linear weighted logic to generate the final credit allocation signal used to adjust the weights of the strategy network; wherein, the value range of the dynamic adaptive factor used to balance physical efficiency and commercial constraints is 0.8 to 0.

9. Step S5: The feedback correction module monitors the data variance change rate of the global order value gap matrix, and when the data variance change rate exceeds the preset threshold of 0.15, it lowers the value of the dynamic adaptive factor to reduce the weight ratio of the local performance reward signal in the final credit allocation signal. In step S6, each scheduling agent module adjusts the parameter gradient of the policy network according to the final credit allocation signal and generates node scheduling instructions corrected by business constraints.

Images