Multi-agent cooperative scheduling method based on federal reinforcement learning and digital twinning

Abstract

The embodiment of the invention provides a multi-agent collaborative scheduling method based on federal reinforcement learning and digital twinning, which belongs to the technical field of computing, and specifically comprises the following steps: importing a test task list, environmental parameters and an agent capability matrix; constructing a mathematical model of a scheduling decision; based on a digital twinborn model of a physical entity, rehearsing a scheduling process and outputting a pre-optimization strategy set containing a conflict avoidance strategy; independently executing a deep reinforcement learning algorithm locally, generating a private experience pool, and updating local model parameters according to the private experience pool; the aggregation server calculates an aggregation weight based on the performance index of each agent, and adopts a weighted average strategy to aggregate local model parameters of each agent; dynamically allocating tasks to corresponding agents and monitoring task execution states; real-time resource competition among the intelligent agents is solved through an evolutionary game mechanism; and performing scheduling ending evaluation and iteration. Through the scheme of the invention, the scheduling efficiency, security and robustness are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computing, and in particular to a multi-agent collaborative scheduling method based on federated reinforcement learning and digital twinning. BACKGROUND

[0002] At present, with the development of federated learning, reinforcement learning and digital twinning technology, it has become a technical breakthrough direction to build a decentralized, adaptive and iterative collaborative scheduling system. However, the existing solutions have not achieved deep coupling of the three, and it is difficult to balance the privacy protection, scheduling efficiency and environmental robustness.

[0003] Therefore, there is an urgent need for a multi-agent collaborative scheduling method based on federated reinforcement learning and digital twinning with high scheduling efficiency, security and robustness. SUMMARY

[0004] Therefore, the embodiments of the present application provide a multi-agent collaborative scheduling method based on federated reinforcement learning and digital twinning, which at least partially solves the problem of poor scheduling efficiency, security and robustness in the prior art.

[0005] The embodiments of the present application provide a multi-agent collaborative scheduling method based on federated reinforcement learning and digital twinning, comprising: Step 1, system initialization: import the test task list, environmental parameters and agent capability matrix, and complete the basic configuration of the system; Step 2, reinforcement learning modeling: based on the test task target and constraints, define the state space, action space and reward function, and build a mathematical model of scheduling decision; Step 3, digital twinning pre-simulation: based on the digital twinning model of the physical entity, pre-play the scheduling process and output a pre-optimized strategy set containing conflict avoidance strategies; Step 4, local training: each agent performs a deep reinforcement learning algorithm independently based on the pre-optimized strategy set and local real-time data, generates a private experience pool and updates the local model parameters accordingly; Step 5, federated parameter aggregation: the aggregation server calculates the aggregation weight based on the performance indicators of each agent, aggregates the local model parameters of each agent using the weighted average strategy, generates global optimal model parameters and distributes them to each agent; Step 6, real-time task allocation and monitoring: based on the global optimal model parameters and real-time data stream, dynamically allocate tasks to the corresponding agents and monitor the task execution status; Step 7, dynamic game decision: when a resource conflict is monitored, solve the real-time resource competition between agents through an evolutionary game mechanism, and output an evolutionary stable strategy as a conflict resolution solution; Step 8, scheduling ending evaluation and iteration: after the end of the scheduling period, a scheduling evaluation report is generated, the system configuration is updated based on the evaluation results, and the next round of closed-loop iteration is triggered.

[0006] According to a specific implementation manner of an embodiment of the present application, the step 1 specifically comprises: Step 1.1, importing a test task list, wherein the test task list comprises a plurality of test tasks, each test task containing priority, time window and resource requirement attribute; Step 1.2, importing environment parameters, wherein the environment parameters comprise initial resource state, agent space distribution and environment disturbance coefficient; Step 1.3, constructing an agent capability matrix, wherein the agent capability matrix is used to represent the capability values of each agent in the dimensions of execution efficiency, resource carrying capacity and reliability.

[0007] According to a specific implementation manner of an embodiment of the present application, the step 2 specifically comprises: Step 2.1, defining state space to include time environment state, agent real-time capability matrix and remaining task set; Step 2.2, defining action space as the matching relationship between tasks and agents and resource allocation strategy; Step 2.3, using a composite reward function to quantify scheduling performance, wherein the composite reward function is: ; wherein, 、 、 is a normalized weight coefficient, used to adjust the relative importance of the three core scheduling indicators in the overall performance evaluation and satisfy , is the task completion utility at time t, is the resource consumption cost at time t, is the task conflict penalty at time t.

[0008] According to a specific implementation manner of an embodiment of the present application, the step 3 specifically comprises: Step 3.1, constructing a digital twin model, wherein the digital twin model comprises agent digital mapping, environment dynamic model and task evolution model, used to reproduce the entity test environment and simulate the test process and abnormality; Step 3.2, based on task prediction parameters , simulation is performed through the digital twin model to generate a task allocation pre-strategy, a resource reservation scheme and a conflict avoidance rule as a pre-optimization strategy set, wherein the task prediction parameters The task prediction set including future T time steps and the environment disturbance prediction are expressed by the digital twin model simulation as follows: ; wherein, is a twin simulation function, represents a complete digital twin model constructed based on a physical entity; Step 3.3, simulate multiple extreme scenarios, perform robustness evaluation on the generated pre-strategy, and eliminate the strategies in the pre-optimization strategy set whose robustness indicators are lower than the preset threshold.

[0009] According to a specific implementation manner of an embodiment of the present application, the step 4 specifically comprises: Step 4.1, each agent constructs a private experience pool by performing actions and recording interaction data, and updates local model parameters by using a proximal policy optimization or a deep deterministic policy gradient algorithm; Step 4.2, after updating, the model performance is checked by using a local validation set, and the model parameters are retained when the accuracy rate meets the preset condition.

[0010] According to a specific implementation manner of an embodiment of the present application, the step 5 specifically comprises: Step 5.1, the local model parameters of each agent are transmitted to an aggregation server in an encrypted manner, and an aggregation weight is calculated based on the historical performance indicators of the agents: ; wherein, is the weight of the i th agent, is the task completion rate, is the model accuracy rate, and satisfies ; Step 5.2, the local parameters are aggregated by using a weighted average to obtain global optimal model parameters: ; wherein, is the local model parameter of the i th agent; Step 5.3, the global optimal model parameters are distributed to each agent.

[0011] According to a specific implementation manner of an embodiment of the present application, the step 6 specifically comprises: Step 6.1, real-time data streams are received by an Internet of Things sensor and an artificial recording system to update the system state; Step 6.2, according to the updated system state and the global optimal model parameters, a real-time task-agent matching scheme is generated based on a global strategy network: ; wherein, is a global policy function, represents the action at time t, represents the system state at time t, including the state of all current tasks, the resources and capabilities of the agent, and environmental disturbances; Step 6.3, collect execution data through the Internet of Things interface to generate an execution log, and trigger an early warning when an anomaly is detected.

[0012] According to a specific implementation of an embodiment of the present application, the step 7 specifically includes: Step 7.1, defining the agent participating in resource competition as a game participant and its strategy space; Step 7.2, calculating the individual income and group average income of each participant under different strategies based on task utility and resource cost; Step 7.3, updating the strategy distribution of the agent through the replicator dynamic equation until the evolutionary stable strategy is reached; Step 7.4, issuing the evolutionary stable strategy as a conflict resolution solution to the relevant agent.

[0013] According to a specific implementation of an embodiment of the present application, the step 8 specifically includes: Step 8.1, outputting a scheduling evaluation report R after one scheduling period ends, wherein the core indicators in the evaluation report include task completion rate , resource utilization , conflict occurrence rate , and average scheduling delay ; Step 8.2, calculating the iterative correction amount based on the deviation of the core indicators and the target values: ; wherein Target is the target value of the indicator, Indicator is the actual value of the core indicator, is the correction coefficient; Step 8.3, updating the system configuration according to the iterative correction amount, and storing the key data of this scheduling into the knowledge base, wherein the key data includes the evaluation report, the conflict resolution solution, and the optimization parameters, and the expression for updating the system configuration is: ; wherein, is a configuration update function, is the initial parameter of the next round of closed loop; Step 8.4, after the update is completed, the system returns to step 1 to start the next round of closed loop iteration.

[0014] The multi-agent cooperative scheduling scheme based on federated reinforcement learning and digital twins in this invention embodiment includes: Step 1, system initialization: importing the test task list, environmental parameters, and agent capability matrix to complete the basic system configuration; Step 2, reinforcement learning modeling: defining the state space, action space, and reward function based on the test task objectives and constraints, and constructing a mathematical model for scheduling decisions; Step 3, digital twin pre-simulation: based on the digital twin model of the physical entity, pre-simulating the scheduling process and outputting a pre-optimized policy set including conflict avoidance strategies; Step 4, local training: each agent independently executes a deep reinforcement learning algorithm locally based on the pre-optimized policy set and local real-time data, generating a private experience pool and updating the local model parameters accordingly. Step 5, Federation Parameter Aggregation: The aggregation server calculates the aggregation weight based on the performance indicators of each agent, and uses a weighted average strategy to aggregate the local model parameters of each agent, generating globally optimal model parameters and distributing them to each agent; Step 6, Real-time Task Allocation and Monitoring: Based on the globally optimal model parameters and real-time data stream, tasks are dynamically allocated to the corresponding agents and the task execution status is monitored; Step 7, Dynamic Game Theory Decision: When resource conflicts are detected, the real-time resource competition between agents is resolved through an evolutionary game mechanism, and an evolutionary stable strategy is output as a conflict resolution solution; Step 8, Scheduling Final Evaluation and Iteration: After the scheduling cycle ends, a scheduling evaluation report is generated, the system configuration is updated based on the evaluation results, and the next round of closed-loop iteration is triggered.

[0015] The beneficial effects of the embodiments of the present invention are as follows: 1. Federated-Twin Collaborative Architecture: This architecture achieves deep coupling between federated learning and digital twins for the first time. It reduces the trial-and-error cost of federated training through twin pre-simulation and ensures distributed optimization of the twin model through federated learning, balancing privacy and scheduling accuracy. Unlike single federated learning frameworks, this architecture adds a twin pre-simulation-verification closed loop, which improves the convergence speed of the global model. 2. Multi-dimensional reward and game mechanism: Design a composite reward function that integrates task utility, resource consumption and conflict penalty, and introduce evolutionary game theory to resolve real-time conflicts, so as to achieve a balance between individual optimization and group optimality; 3. Closed-loop iteration throughout the entire process: Construct a closed-loop system of "pre-simulation - local training - global aggregation - real-time execution - evaluation iteration" to continuously correct model parameters and environment configuration through actual running data, thereby improving long-term adaptability; 4. Dynamic weight aggregation strategy: The federated aggregation weights are dynamically adjusted based on indicators such as agent task completion rate and data quality to avoid the drag on global performance by low-quality local models. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a multi-agent cooperative scheduling method based on federated reinforcement learning and digital twins provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the specific implementation process of a multi-agent cooperative scheduling method based on federated reinforcement learning and digital twins, provided by an embodiment of the present invention. Figure 3 This is a schematic diagram illustrating the coupling of reinforcement learning and twin simulation, provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of a dynamic game and federated aggregation process provided in an embodiment of the present invention. Detailed Implementation

[0018] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0019] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this invention, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0021] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. The illustrations only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0022] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0023] This invention provides a multi-agent cooperative scheduling method based on federated reinforcement learning and digital twins, which can be applied to the optimization of test processes.

[0024] See Figure 1 This is a flowchart illustrating a multi-agent cooperative scheduling method based on federated reinforcement learning and digital twins, provided by an embodiment of the present invention. Figure 1 and Figure 2 As shown, the method mainly includes the following steps: Step 1, System Initialization: Import the test task list, environment parameters, and agent capability matrix to complete the basic system configuration; This implementation plan is based on the reliability testing of 100 large-scale devices, and revolves around the core objectives of "completing the task as quickly as possible and minimizing the overall probability of missed detections." The execution process strictly follows these constraints: the test is undertaken by a single team, with one shift per day and a working time not exceeding 12 hours; the test process requires the individual testing of subsystems A, B, and C to be completed first, and the E comprehensive test can only be carried out after all three have passed; if a problem is found in a certain process, it must be retested, and if it fails twice consecutively, the device is removed from the test; after the device has completed the test, it takes 0.5 hours to transport it out of and into the test hall, and the transportation process does not affect the testing of other devices. The testing hall is equipped with two test benches capable of parallel testing. The team is divided into four specialized groups: A, B, C (subsystem testing), and E (integrated testing). Each group has a dedicated workstation and can conduct tests independently. The initial setup and commissioning time for the equipment is 30 minutes for group A, 20 minutes for group B, 20 minutes for group C, and 40 minutes for group E. Normal testing time is 2.5 hours for group A, 2 hours for group B, 2.5 hours for group C, and 3.0 hours for group E. Abnormal situations include Y1 equipment failure, Y2 subsystem problems, Y3 operator errors, and Y4 integrated testing problems. Testing begins when "the testing equipment is in place and all equipment has been debugged." If the process is interrupted, it must be restarted. Based on the above constraints, the detailed implementation steps are as follows: In specific implementation, step 1 (S1): System initialization The system receives test task instructions and environment configuration, and completes the import and initialization of three core data items: task list, environment parameters, and agent capability matrix, providing basic data support for subsequent scheduling. 1. Import Task List: Import a collection of tasks to be executed. Each of the tasks Includes priority Time window Resource requirements Attributes such as equipment testing timing constraints are included. For example, import a set of tasks to be executed for 100 large devices, with each device corresponding to one independent test task. The priority of the first device task (number: Dev-001) is set to 0.8 (priority range 0-1, the higher the value, the higher the priority, matching the requirement of "prioritizing the testing of key batch devices"), and the time window is set to "8:00 on the first day of test start to 20:00 on the third day after start" (reserving a buffer time for retesting). The resource requirements are specified as "occupying 1 test bench, 1 dedicated workstation for A / B / C / E each, and 2 testers (1 responsible for testing A / B subsystems, and 1 responsible for testing C / E subsystems)". Subsequent device tasks are imported sequentially according to the principle of "priority gradient allocation (0.5-0.8), time window staggered by 1-2 hours, and resource requirements matching the correspondence between test benches and workstations" to form a complete task list.

[0025] 2. Environment parameter configuration: Import environment parameter vector ,in This is the initial resource state. For the spatial distribution of intelligent agents, This represents the environmental interference coefficient. The initial resource status clearly defines the initial state of the test benches and workstations, as well as the physical parameters of the equipment. For example, test benches 1 and 2 are both idle, and the equipment at dedicated workstations A, B, C, and E is all operating normally. The spatial distribution of intelligent agents is defined according to the physical layout of the test hall. For example, an intelligent agent bound to workstation A is located in area 1 of the east zone of the hall, an intelligent agent bound to workstation B is located in area 2 of the east zone of the hall, an intelligent agent bound to test bench 1 is located in area 1 of the central zone of the hall, and an intelligent agent bound to test bench 2 is located in area 2 of the central zone of the hall (to avoid wasting time transporting equipment due to cross-area scheduling). The environmental interference coefficient is set to 0.05 (based on the actual scenario of the test hall, corresponding to slight interference such as "workshop temperature fluctuation ±2℃, voltage fluctuation ±3%").

[0026] 3. Agent capability modeling: Constructing the initial agent capability matrix Where N is the number of agents and K is the capability dimension (such as execution efficiency, resource capacity, reliability). This represents the capability value of the i-th agent in the k-th dimension. For example, the execution efficiency of the agent bound to workstation A (corresponding to test group A) is set to 0.92 (because the normal test time for group A is 2.5 hours, and the testers are highly skilled, the actual test time is 8% shorter than the standard time), the resource capacity is set to 8 units / day (within a 12-hour shift per day, after deducting debugging and equipment cooling time, a maximum of 8 devices of subsystem A can be tested), and the reliability is set to 0.98 (no fault records in the last 30 tests). The remaining agents bound to test stations and workstations B / C are assigned values ​​sequentially according to the principle of "capability matching group test characteristics" to form a complete agent capability matrix.

[0027] After initialization, the data is verified and stored in the system knowledge base as the basic input for subsequent modeling and scheduling.

[0028] Step 2, Reinforcement Learning Modeling: Based on the test task objectives and constraints, define the state space, action space, and reward function, and construct a mathematical model for scheduling decisions; In specific implementation, step 2 (S2): The specific process of reinforcement learning modeling is as follows: By combining the test task objectives and constraints, a reinforcement learning model is constructed, defining the state space, action space, and reward function to provide algorithmic support for scheduling decisions. 1. Definition of State Space: The state space S includes the environment state, the agent state, and the task state, i.e. ,in Let t be the environmental state. For the real-time capability matrix of the intelligent agent, This represents the set of remaining tasks. t = 8:00 AM on the first day of the test. Test benches 1 and 2 are both idle, equipment at stations A, B, C, and E are all operating normally, and the environmental interference factor is 0.05. The matrix is ​​6×3. The agent bound to workstation A has an execution efficiency of 0.92, a resource capacity of 8 units / day, and a reliability of 0.98; the agent bound to workstation B has an execution efficiency of 0.88, a resource capacity of 10 units / day, and a reliability of 0.95; the agent bound to workstation C has an execution efficiency of 0.90, a resource capacity of 7 units / day, and a reliability of 0.97; the agent bound to workstation E has an execution efficiency of 0.85, a resource capacity of 6 units / day, and a reliability of 0.96; the agents bound to test stations 1 and 2 both have an execution efficiency of 0.95, a resource capacity of 1 unit / time period each, and a reliability of 0.99. The set consists of 100 devices to be tested, with each device labeled with priority, planned test period and resource requirements.

[0029] 2. Action Space Definition: Action space A represents the task-agent matching and resource allocation strategy, i.e. This indicates that the task will be completed. Assigned to intelligent agents and allocate resources Actions were generated at 8:30 AM on the first day of the test. ,in For the testing of the first device, To bind the intelligent agent at workstation C, Includes access to Testbench 1, dedicated equipment for Workstation C, and one tester responsible for testing the C subsystem; synchronous action generation. ,in For the testing of the second device, To bind the intelligent agent at workstation B, Includes access to Test Station 2, dedicated equipment for Workstation B, and one tester responsible for testing Subsystem B, ensuring no resource overlap in actions.

[0030] 3. Reward Function Design: A composite reward function is used to quantify scheduling performance, expressed as follows: ; in , , And satisfy , Let t be the utility of task completion. Let be the resource consumption cost at time t. Penalty for task conflict at time t. This occurs at 10:00 AM on the first day of the test. according to" Subsystem A testing is complete (30% completion rate), priority 0.8” calculated to be 0.9. Based on the calculation of "0.5 hours of idle time for test bench 1 and 0 hours of idle time for workstation A", the result is 0.1. Since there are no resource conflicts, the result is 0. Substituting this into the equation, we get... The modeling results are output to the digital twin pre-simulation module and the local training module. Step 3, Digital Twin Pre-simulation: Based on the physical entity digital twin model, the scheduling process is pre-simulated and a set of pre-optimized strategies including conflict avoidance strategies is output; In specific implementation, such as Figure 3 As shown, step 3 (S3): the digital twin pre-simulation process includes: Digital twin model M based on physical entities and task prediction parameters Conduct scheduling process rehearsals and strategy optimization: 1. Twin Model Construction: The twin model M includes agent digital mapping, environmental dynamics model, and task evolution model. Agent digital mapping restores the physical parameters of six agents, including test equipment power and test accuracy thresholds. It also integrates an equipment loss model, which operates according to the cumulative loss pattern over test duration. For example, after test bench 1 operates for 120 hours, the failure probability increases from 3% to 5%, matching the failure parameters of device Y1. The environmental dynamics model simulates the real-time fluctuations of temperature (18-25℃) and voltage (380V±10%) in the test hall. These fluctuations are correlated with changes in equipment performance; for example, when the voltage is below 360V, the testing efficiency of station A decreases by 10%. The task evolution model simulates the device test state transitions according to the "subsystem test - retest - comprehensive test" process. The retest stage is triggered when a problem in subsystem Y2 or an error by the tester in Y3 is detected, and the retest duration is calculated as 1.2 times the original test duration, achieving accurate reproduction of the actual scheduling scenario.

[0031] 2. Pre-optimization strategy generation: Input task prediction parameters (Including the task prediction set and environmental disturbance prediction for the next T time steps), a pre-optimized strategy set is generated through twin simulation: in, For twin simulation functions, This includes task allocation pre-strategy, resource reservation scheme, and conflict avoidance rules. For this example, the input task prediction parameters are... ,in This is a set of 20 devices to be tested over the next 8 hours (T=8). For predicting environmental disturbances over the next 8 hours (including two voltage fluctuation warnings), a twin simulation function is used. Generate a set of pre-optimized strategies .in A pre-strategy for task allocation that prioritizes assigning high-priority devices to available workstations. A resource reservation plan for "reserving one spare workstation during periods of voltage fluctuation" is proposed. Conflict avoidance rules for "scheduling related workstations on the same test bench by 1 hour".

[0032] 3. Robustness Assessment: Simulate K extreme scenarios (such as sudden resource depletion or sudden increase in tasks) and calculate the robustness index of the pre-strategy. Remove those below the threshold The strategy is as follows. This example uses K=3 extreme scenarios: Scenario 1 is "Test bench 1 experiences a sudden Y1 failure (working for 10 hours)", Scenario 2 is "5 new emergency device testing tasks added within 1 hour", and Scenario 3 is "Simultaneous failure of workstations B and C". The robustness index of each pre-strategy is calculated. (Values ​​range from 0 to 1). In Scenario 1 (The 2nd schedulable test station can undertake tasks). In Scenario 1 (No task delays after the backup workstation was activated), removed. of The pre-simulation results are output to the local training module to guide the generation of the experience pool. Step 4, Local Training: Each agent independently executes a deep reinforcement learning algorithm locally based on the pre-optimized policy set and local real-time data to generate a private experience pool and update the local model parameters accordingly; In practice, step 4 (S4): The specific process of local training is as follows: Each agent independently executes the DRL algorithm (such as PPO, DDPG) locally, and conducts training based on pre-simulation results and local data: 1. Experience pool construction: The agent is based on a pre-policy Combine local real-time data, execute actions and record interaction data to generate a private experience pool. ; in Let i be the local observation state of the i-th agent in the k-th instance. In order to perform the action, For instant rewards, The state for the next moment. The agent (i=1) bound to workstation A is generated. The 10th data point is... , Includes "Occupied workstation A", "Test progress 50%" To "continue execution" Subsystem A test” , Includes "Occupied workstation A", Testing progress 60%.

[0033] 2. Local Model Update: The DRL algorithm is used to update the local model parameters. To update, taking PPO as an example, parameter optimization is achieved by minimizing the pruning objective function: Among them, The ratio of the new strategy to the old strategy. For the dominant function, is the pruning factor. The agent bound to workstation B (i=2) updates the model parameters through 1000 rounds of training. In the 500th round, the objective function value drops to 0.03 (a decrease of 85% from the initial value), and the policy stability meets the requirements.

[0034] 3. Model Validation: The model performance was evaluated using a local validation set (containing test data from 20 devices). The validation metric was "task assignment accuracy." Only when the accuracy reached a certain level... The current model parameters are retained. The agent bound to workstation E (i=4) is verified. Preserve parameters; bind the agent at workstation C (i=3) initially. After supplementing with 100 Y2 abnormal scenario data points and retraining, the final result was... And retain the parameters.

[0035] After local training is complete, each agent will transfer the model parameters. (Excluding raw data) Uploaded to the Federal Parameter Aggregation Module.

[0036] Step 5, Federation parameter aggregation: The aggregation server calculates the aggregation weight based on the performance indicators of each agent, aggregates the local model parameters of each agent using a weighted average strategy, generates the globally optimal model parameters, and distributes them to each agent. In specific implementation, step 5 (S5): the specific process of federated parameter aggregation is as follows: The federated aggregation server receives the local model parameters from each agent and generates a global model using a weighted averaging strategy. 1. Weight Calculation: Based on the agent's historical performance metrics (such as task completion rate) Model accuracy Calculate the aggregate weights: in, Let be the weight of the i-th agent, satisfying .

[0037] 2. Global parameter generation: Local parameters are aggregated using a weighted average. ; in, These are the local model parameters for the i-th agent. These are the globally optimal model parameters; 3. Parameter distribution: Distribute the data to each agent, such as the six agents in this example, to update the local model.

[0038] The aggregation process ensures parameter security and avoids model privacy leakage through encrypted transmission (such as isomorphic encryption). Step 6, Real-time Task Allocation and Monitoring: Based on the globally optimal model parameters and real-time data stream, tasks are dynamically allocated to the corresponding agents and the task execution status is monitored; In practice, step 6 (S6) involves real-time task allocation and monitoring. Based on global model With real-time data streaming, dynamic task allocation and execution monitoring are achieved. 1. Real-time status awareness: Receives real-time data streams through IoT sensors and a manual recording system. This includes environmental status updates (test bench 1 operating time 12 hours, workshop temperature 22℃) and task status updates ( Completed testing of subsystem B and entered the testing queue for subsystem C; agent status update (the execution efficiency of the agent bound to workstation C dropped to 0.88); system status updated every 5 minutes. .

[0039] 2. Task allocation decision: Real-time allocation actions are generated based on a global policy network. ; in, This is the global policy function, which outputs the task-agent matching scheme at the current time step. t = 14:00 the day after the test starts. Contains " "Pending testing of subsystem A, workstation A is idle" - generate That is, allocation The intelligent agent bound to workstation A is provided with the following resources: access to test bench 2, dedicated equipment for workstation A, and one tester. The intelligent agent will start immediately upon receiving the action. Subsystem A test.

[0040] 3. Execution Monitoring: Collect execution data (test progress, equipment temperature, operator operation records) from each intelligent agent via IoT interface, and generate execution logs. At 15:30 the day after the test started, t= was detected. The test of subsystem A was delayed by 30 minutes (due to a temporary change of testers), triggering an anomaly warning. The anomaly type was recorded as "delay in tester handover," and the warning information was simultaneously transmitted to the dynamic game decision-making module and the closing evaluation module.

[0041] Monitoring data is synchronously output to the dynamic game decision-making module and the closing evaluation module. Step 7, Dynamic Game Decision-Making: When resource conflicts are detected, the real-time resource competition between agents is resolved through an evolutionary game mechanism, and an evolutionary stable strategy is output as a conflict resolution solution; In specific implementation, such as Figure 4 As shown, step 7 (S7): the dynamic game decision-making process is as follows: When the monitoring module detects a resource conflict (such as multiple agents competing for the same resource), it initiates an evolutionary game mechanism to resolve the conflict: 1. Game Theory Model Construction: Define N=2 agents as game participants, i.e., bind them together. intelligent agents ( ) and binding intelligent agents ( The two companies are competing for the right to use workstation B; strategy space ,in Assigning "Workstation B" " Assigning "Workstation B" " Assign "Workstation B at 16:00" 18:00 allocated " Let be the strategy distribution of the i-th participant.

[0042] 2. Benefit Calculation: Calculate individual benefits based on task utility and resource costs. Compared with the average group income .use hour, of ( Priority 0.8, no delay) of ( (2-hour delay) ;use hour, of (No delay) of (Delayed by 2 hours but no conflict) .

[0043] 3. Strategy Evolution: By replicating dynamic equations ; Update agent policy distribution. This represents the diffusion rate of strategy i. This represents the proportion of agents in the population that adopt strategy i. This represents the expected reward of the agent using strategy i. This represents the average expected return for the entire group. Strategy, , At this point, the policy distribution reaches the evolutionarily stable policy (ESS).

[0044] 4. Conflict resolution: Match the corresponding ESS... Strategy as a conflict resolution method is issued to and , Workstation B will be activated at 16:00. Workstation B will be activated at 18:00 to avoid resource competition.

[0045] The game results are synchronously stored in the system knowledge base for subsequent strategy optimization. Step 8, Scheduling Closure Evaluation and Iteration: After the scheduling cycle ends, a scheduling evaluation report is generated, the system configuration is updated based on the evaluation results, and the next round of closed-loop iteration is triggered.

[0046] In specific implementation, step 8 (S8): the specific process of scheduling finalization evaluation and iteration is as follows: At the end of a scheduling cycle, the system performs a comprehensive evaluation and updates its configuration: 1. Evaluation Report Generation: Outputs a scheduling evaluation report R, including the core metric: task completion rate. Resource utilization rate Conflict incidence rate Average scheduling delay ; 2. Calculation of Iterative Correction Amount: Based on the deviation between the evaluation index and the target value, the iterative correction amount is calculated. : ; Where Target is the target value of the indicator, and Indicator is the actual value. The correction factor is 0.1. The target resource utilization rate is 90%, calculated as follows: The average scheduling delay target value is 20 minutes, calculated as follows: minute.

[0047] 3. Configuration Update: Updates the initial system configuration parameters. ; in, To configure the update function, These are the initial parameters for the next closed loop. In this example, the resource capacity of the intelligent agents bound to workstations A and B will be increased by 0.5 units / day each, and the scheduling time window interval will be shortened by 0.5 minutes. 4. Knowledge base update: Key data such as assessment reports, conflict solutions, and optimization parameters are stored in a long-term knowledge base to enable the reuse of experience.

[0048] After the update is complete, the system returns to step 1 and starts the next round of the "initialization-scheduling-optimization" closed loop.

[0049] This embodiment provides a multi-agent cooperative scheduling method based on federated reinforcement learning and digital twins. It reduces the trial-and-error cost of federated training through twin pre-simulation and ensures distributed optimization of the twin model through federated learning, balancing privacy and scheduling accuracy. A composite reward function integrating task utility, resource consumption, and conflict penalties is designed, and evolutionary game theory is introduced to resolve real-time conflicts, achieving a balance between individual optimization and group optimality. A closed-loop system of "pre-simulation - local training - global aggregation - real-time execution - evaluation iteration" is constructed, continuously correcting model parameters and environment configurations through actual operating data to improve long-term adaptability. The federated aggregation weights are dynamically adjusted based on agent task completion rate, data quality, and other indicators to avoid low-quality local models dragging down global performance.

[0050] The method of the present invention will be further described below with reference to a specific embodiment: 1. Experimental subjects and scenarios Using a large-scale device containing parallel subsystems A, B, and C, and subsystems D (connection system) and E (integrated testing) as the test object, the full-process testing scenario of two batches of 100 devices each was simulated. The test environment strictly replicated the constraints of the competition: the test hall contained 2 parallel test benches and 4 dedicated workstations (A, B, C, E); the time for transporting devices in and out was 0.5 hours each; the equipment debugging time (A: 30 min, B: 20 ​​min, C: 20 min, E: 40 min) and the normal testing time (A: 2.5 h, B: 2 h, C: 2.5 h, E: 3.0 h) were all configured according to the competition requirements.

[0051] 2. Abnormal Scenario Parameters Y1 (Equipment Failure): The cumulative failure probability of A / B / C / E within 120h is 3% / 4% / 2% / 3%, which increases to 5% / 7% / 6% / 5% from 120-240h, and mandatory replacement is required after ≥240h; Y2 (Subsystem Problem): The probability of the original problem A / B / C is 2.5% / 3% / 2%; Y3 (Testing Error): The error probability of A / B / C / E is 3% / 4% / 2% / 2%, with misjudgment (Y31) and missed judgment (Y32) each accounting for 50%; Y4 (Comprehensive Test Problem): The probability of a problem in subsystem D is 0.1%.

[0052] 3. Experimental Grouping and Comparison Model Three sets of comparative experiments were set up, with each set undergoing 1000 Monte Carlo simulations to ensure statistical stability: Control group 1 (traditional FCFS scheme): adopts the "first-come, first-served" scheduling strategy, with a fixed 12-hour shift for each team, restarting after a process interruption, and no dynamic optimization mechanism; Control group 2 (single reinforcement learning scheme): centralized scheduling model based on DDPG algorithm, without digital twin pre-simulation and federated learning privacy protection; Experimental group (the scheme of this invention): adopts a "federated reinforcement learning + digital twin + multi-agent collaboration" architecture, the core modules of which include: Digital Twin (DT) module: Real-time mapping of test benches, workstations, and device status, and simulation of abnormal scenarios; Federated Reinforcement Learning (FL) module: 6 local agents in total, 4 workstations + 2 test benches, dynamically aggregate weights to generate global policy; Multi-agent collaboration module: The workstation agent is responsible for local scheduling, and the test bench agent is responsible for global resource balancing.

[0053] 4. Evaluation Indicators The core evaluation indicators include: average number of days to complete the task (T); average number of devices that pass the test (S); total false negative probability (P_L): number of devices that are false negative / number of devices that pass the test; total false positive probability (P_W): number of false positives / total number of tests; and the ratio of effective working time of the professional testing team (YXB).

[0054] The experimental results are shown in Table 1: Table 1

[0055] It is evident that the twin simulation identifies bottlenecks in workstation E, and the federated intelligent agent dynamically reorders the process, shortening time by 8.7 days, significantly reducing the risk of missed detections, and achieving a more balanced workload across workstations.

[0056] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof.

[0057] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-agent cooperative scheduling method based on federated reinforcement learning and digital twins, characterized in that, include: Step 1, System Initialization: Import the test task list, environment parameters, and agent capability matrix to complete the basic system configuration; Step 2, Reinforcement Learning Modeling: Based on the test task objectives and constraints, define the state space, action space, and reward function, and construct a mathematical model for scheduling decisions; Step 3, Digital Twin Pre-simulation: Based on the digital twin model of the physical entity, the scheduling process is simulated and a set of pre-optimized strategies including conflict avoidance strategies is output; Step 4, Local Training: Each agent independently executes a deep reinforcement learning algorithm locally based on the pre-optimized policy set and local real-time data, generates a private experience pool, and updates the local model parameters accordingly. Step 5, Federation parameter aggregation: The aggregation server calculates the aggregation weight based on the performance indicators of each agent, aggregates the local model parameters of each agent using a weighted average strategy, generates the globally optimal model parameters, and distributes them to each agent. Step 6, Real-time task allocation and monitoring: Based on the globally optimal model parameters and real-time data stream, dynamically allocate tasks to the corresponding agents and monitor the task execution status; Step 7, Dynamic Game Decision-Making: When resource conflicts are detected, the real-time resource competition between agents is resolved through an evolutionary game mechanism, and an evolutionary stable strategy is output as a conflict resolution solution. Step 8, Scheduling Closure Evaluation and Iteration: After the scheduling cycle ends, a scheduling evaluation report is generated, the system configuration is updated based on the evaluation results, and the next round of closed-loop iteration is triggered.

2. The method according to claim 1, characterized in that, Step 1 specifically includes: Step 1.1: Import the test task list, wherein the test task list includes multiple test tasks, and each test task includes priority, time window and resource requirement attributes; Step 1.2, import environmental parameters, wherein the environmental parameters include initial resource state, agent spatial distribution and environmental interference coefficient; Step 1.3: Construct the agent capability matrix, wherein the agent capability matrix is ​​used to characterize the capability values ​​of each agent in terms of execution efficiency, resource carrying capacity and reliability.

3. The method according to claim 1, characterized in that, Step 2 specifically includes: Step 2.1, define the state space as including the environment state at any given time, the agent's real-time capability matrix, and the set of remaining tasks; Step 2.2, define the action space as the matching relationship between tasks and agents and the resource allocation strategy; Step 2.3: Quantify scheduling performance using a composite reward function, wherein the composite reward function is: ； in, , , The normalized weighting coefficients are used to adjust the relative importance of the three core scheduling metrics in the overall performance evaluation and satisfy the following conditions: , Let t be the utility of task completion. Let be the resource consumption cost at time t. Penalty for task conflict at time t.

4. The method according to claim 1, characterized in that, Step 3 specifically includes: Step 3.1: Construct a digital twin model, wherein the digital twin model includes an agent digital mapping, an environmental dynamic model, and a task evolution model, used to replicate the actual test environment and simulate the test process and anomalies; Step 3.2, based on task prediction parameters Simulation is performed using a digital twin model to generate a pre-optimization strategy set, including task allocation pre-strategies, resource reservation schemes, and conflict avoidance rules. The task prediction parameters... Including the task prediction set and environmental disturbance prediction for the next T time steps, the expression for simulation using a digital twin model is: ； in, For twin simulation functions, This represents a complete digital twin model built upon physical entities; Step 3.3: Simulate various extreme scenarios, evaluate the robustness of the generated pre-policies, and remove policies from the pre-optimized policy set whose robustness index is lower than a preset threshold.

5. The method according to claim 1, characterized in that, Step 4 specifically includes: Step 4.1: Each agent builds a private experience pool by executing actions and recording interaction data, and updates the local model parameters using proximal policy optimization or deep deterministic policy gradient algorithm. Step 4.2: After the update, the model performance is verified using a local validation set. The model parameters are retained when the accuracy meets the preset conditions.

6. The method according to claim 1, characterized in that, Step 5 specifically includes: Step 5.1: Encrypt and transmit the local model parameters of each agent to the aggregation server, and calculate the aggregation weight based on the historical performance metrics of the agents. ； in, Let be the weight of the i-th agent. For task completion rate, For model accuracy, satisfy ; Step 5.2: Use a weighted average to aggregate the local parameters to obtain the globally optimal model parameters. ； in, These are the local model parameters for the i-th agent; Step 5.3: Distribute the globally optimal model parameters to each agent.

7. The method according to claim 6, characterized in that, Step 6 specifically includes: Step 6.1: Receive real-time data streams through IoT sensors and a manual recording system to update the system status; Step 6.2: Based on the updated system state and globally optimal model parameters, generate a real-time task-agent matching scheme using the global policy network. ； in, This is the global policy function. Indicates the action at time t. This represents the system state at time t, including the current state of all tasks, the resources and capabilities of the agents, and environmental disturbances. Step 6.3: Collect execution data through the IoT interface to generate execution logs, and trigger an alert when an anomaly is detected.

8. The method according to claim 1, characterized in that, Step 7 specifically includes: Step 7.1: Define the intelligent agents participating in resource competition as game participants and their strategy space; Step 7.2: Based on task utility and resource cost, calculate the individual and average group benefits of each participant under different strategies. Step 7.3: Update the agent's policy distribution by replicating the dynamic equations until an evolutionarily stable policy is reached; Step 7.4: Issue the evolutionary stabilization strategy as a conflict resolution to the relevant agents.

9. The method according to claim 1, characterized in that, Step 8 specifically includes: Step 8.1: After a scheduling cycle ends, output a scheduling evaluation report R, where the core indicators in the evaluation report include the task completion rate. Resource utilization rate Conflict incidence rate and average scheduling delay ; Step 8.2: Calculate the iterative correction amount based on the deviation between the core indicator and the target value. : ； Where Target is the target value of the indicator, and Indicator is the actual value of the core indicator. This is a correction factor; Step 8.3: Update the system configuration based on the iterative correction amount, and simultaneously store the key data of this scheduling in the knowledge base. The key data includes the evaluation report, conflict resolution methods, and optimization parameters. The expression for updating the system configuration is: ； in, To configure the update function, These are the initial parameters for the next closed loop; Step 8.4: After the update is completed, the system returns to step 1 and starts the next round of closed-loop iteration.

Images