Multi-agent cooperative scheduling method based on federated reinforcement learning and digital twinning

Abstract

The embodiment of the application provides a kind of multi-agent collaborative scheduling method based on federal reinforcement learning and digital twinning, belong to computing technology field, specifically include: import test task list, environmental parameter and agent capability matrix;The mathematical model of scheduling decision is constructed;Based on the digital twin model of physical entity, the scheduling process is previewed and the pre-optimization strategy set containing conflict avoidance strategy is output;Local independent execution deep reinforcement learning algorithm generates private experience pool and updates local model parameters accordingly;The aggregation server calculates the aggregation weight based on the performance indicators of each agent, and aggregates the local model parameters of each agent using the weighted average strategy;Dynamically allocate tasks to corresponding agents and monitor task execution status;Real-time resource competition between agents is solved through evolutionary game mechanism;Scheduling end evaluation and iteration.The scheme of the application improves the scheduling efficiency, safety and robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computing, and in particular to a multi-agent collaborative scheduling method based on federated reinforcement learning and digital twinning. BACKGROUND

[0002] At present, with the development of federated learning, reinforcement learning and digital twinning technology, it has become a technical breakthrough direction to build a decentralized, adaptive and iterative collaborative scheduling system. However, the existing solutions have not achieved deep coupling of the three, and it is difficult to balance the privacy protection, scheduling efficiency and environmental robustness.

[0003] Therefore, there is an urgent need for a multi-agent collaborative scheduling method based on federated reinforcement learning and digital twinning with high scheduling efficiency, security and robustness. SUMMARY

[0004] Therefore, the embodiments of the present application provide a multi-agent collaborative scheduling method based on federated reinforcement learning and digital twinning, which at least partially solves the problem of poor scheduling efficiency, security and robustness in the prior art.

[0005] The embodiments of the present application provide a multi-agent collaborative scheduling method based on federated reinforcement learning and digital twinning, comprising:

[0006] Step 1, system initialization: import the test task list, environment parameters and agent capability matrix, and complete the basic configuration of the system;

[0007] Step 2, reinforcement learning modeling: based on the test task target and constraints, define the state space, action space and reward function, and build a mathematical model of scheduling decision;

[0008] Step 3, digital twinning pre-simulation: based on the digital twinning model of the physical entity, pre-play the scheduling process and output a pre-optimized strategy set containing conflict avoidance strategies;

[0009] Step 4, local training: each agent performs a deep reinforcement learning algorithm based on the pre-optimized strategy set and local real-time data independently in the local, generates a private experience pool and updates the local model parameters accordingly;

[0010] Step 5, federated parameter aggregation: the aggregation server calculates the aggregation weight based on the performance indicators of each agent, aggregates the local model parameters of each agent using the weighted average strategy, generates the global optimal model parameters and distributes them to each agent;

[0011] Step 6, real-time task allocation and monitoring: based on the global optimal model parameters and real-time data stream, dynamically allocate tasks to the corresponding agents and monitor the task execution status;

[0012] Step 7, dynamic game decision: when the resource conflict is monitored, the real-time resource competition among the agents is solved through the evolutionary game mechanism, and an evolutionary stable strategy is output as a conflict solution;

[0013] Step 8, scheduling end evaluation and iteration: after the end of the scheduling period, a scheduling evaluation report is generated, the system configuration is updated based on the evaluation result, and the next round of closed-loop iteration is triggered.

[0014] According to a specific implementation manner of an embodiment of the present application, the step 1 specifically comprises:

[0015] Step 1.1, importing a test task list, wherein the test task list comprises a plurality of test tasks, each test task containing priority, time window and resource requirement attribute;

[0016] Step 1.2, importing environment parameters, wherein the environment parameters comprise initial resource state, agent space distribution and environment disturbance coefficient;

[0017] Step 1.3, constructing an agent capability matrix, wherein the agent capability matrix is used to represent the capability values of each agent in the execution efficiency, resource carrying capacity and reliability dimensions.

[0018] According to a specific implementation manner of an embodiment of the present application, the step 2 specifically comprises:

[0019] Step 2.1, defining a state space comprising current environment state, agent real-time capability matrix and remaining task set;

[0020] Step 2.2, defining an action space as the matching relationship between tasks and agents and resource allocation strategy;

[0021] Step 2.3, using a composite reward function to quantify scheduling performance, wherein the composite reward function is

[0022] wherein, 、 、 is a normalized weight coefficient, used to adjust the relative importance of the three core scheduling indexes in the total performance evaluation and satisfy , is the task completion utility at t moment, is the resource consumption cost at t moment, is the task conflict penalty at t moment.

[0023] According to a specific implementation manner of an embodiment of the present application, the step 3 specifically comprises:

[0024] Step 3.1, a digital twin model is constructed, wherein the digital twin model comprises an agent digital mapping, an environment dynamic model and a task evolution model, and is used to reproduce an entity test environment and simulate a test process and an exception;

[0025] Step 3.2, based on a task prediction parameter , a task allocation pre-strategy, a resource reservation scheme and a conflict avoidance rule are generated as a pre-optimization strategy set through simulation by the digital twin model, wherein the task prediction parameter includes a task prediction set and an environment disturbance prediction for future T time steps, and an expression for the simulation by the digital twin model is

[0026] wherein, is a twin simulation function, indicates a complete digital twin model constructed based on a physical entity;

[0027] Step 3.3, a plurality of extreme scenarios are simulated, the generated pre-strategy is evaluated for robustness, and a strategy with a robustness index lower than a preset threshold is removed from the pre-optimization strategy set.

[0028] According to a specific implementation manner of an embodiment of the present application, the step 4 specifically comprises:

[0029] Step 4.1, each agent constructs a private experience pool by performing an action and recording interaction data, and updates local model parameters by using a proximal policy optimization or a deep deterministic policy gradient algorithm;

[0030] Step 4.2, after the update, a local verification set is used to check the performance of the model, and the model parameters are retained when the accuracy rate meets a preset condition.

[0031] According to a specific implementation manner of an embodiment of the present application, the step 5 specifically comprises:

[0032] Step 5.1, local model parameters of each agent are transmitted to an aggregation server in an encrypted manner, and an aggregation weight is calculated based on a historical performance index of the agent:

[0033]

[0034] wherein, is a weight of the i th agent, is a task completion rate, is a model accuracy rate, and satisfies ;

[0035] Step 5.2, the local parameters are aggregated by using a weighted average to obtain global optimal model parameters:

[0036]

[0037] wherein, is the local model parameter of the i-th agent;

[0038] Step 5.3, distributing the global optimal model parameter to each agent.

[0039] According to a specific implementation manner of an embodiment of the present application, the step 6 specifically comprises:

[0040] Step 6.1, receiving real-time data streams through the Internet of Things sensor and the artificial record system to update the system state;

[0041] Step 6.2, generating a real-time task-agent matching scheme based on the global policy network according to the updated system state and the global optimal model parameter:

[0042]

[0043] wherein, is the global policy function, represents the action at time t, represents the system state at time t, including the state of all tasks, the resources and capabilities of the agent, and the environmental disturbance;

[0044] Step 6.3, collecting execution data through the Internet of Things interface to generate an execution log, and triggering an early warning when an exception is detected.

[0045] According to a specific implementation manner of an embodiment of the present application, the step 7 specifically comprises:

[0046] Step 7.1, defining the agents participating in the resource competition as game participants and their strategy space;

[0047] Step 7.2, calculating the individual income and the group average income of each participant under different strategies based on the task utility and the resource cost;

[0048] Step 7.3, updating the strategy distribution of the agent through the replicator dynamics equation until the evolutionary stable strategy is reached;

[0049] Step 7.4, issuing the evolutionary stable strategy as a conflict resolution scheme to the relevant agent.

[0050] According to a specific implementation manner of an embodiment of the present application, the step 8 specifically comprises:

[0051] Step 8.1, outputting a scheduling evaluation report R when a scheduling period ends, wherein the core indicators in the evaluation report include the task completion rate , the resource utilization , and the conflict occurrence rate and average scheduling delay ;

[0052] Step 8.2: Calculate the iterative correction amount based on the deviation between the core indicator and the target value. :

[0053]

[0054] Where Target is the target value of the indicator, and Indicator is the actual value of the core indicator. This is a correction factor;

[0055] Step 8.3: Update the system configuration based on the iterative correction amount, and simultaneously store the key data of this scheduling in the knowledge base. The key data includes the evaluation report, conflict resolution methods, and optimization parameters. The expression for updating the system configuration is:

[0056]

[0057] in, To configure the update function, These are the initial parameters for the next closed loop;

[0058] Step 8.4: After the update is completed, the system returns to step 1 and starts the next round of closed-loop iteration.

[0059] The multi-agent collaborative scheduling scheme based on federated reinforcement learning and digital twinning in the embodiment of the application comprises: step 1, system initialization: importing a test task list, environment parameters and an agent capability matrix, and completing system basic configuration; step 2, reinforcement learning modeling: defining a state space, an action space and a reward function based on a test task target and a constraint, and constructing a mathematical model of scheduling decision; step 3, digital twinning pre-simulation: pre-playing a scheduling process based on a digital twinning model of a physical entity and outputting a pre-optimization strategy set containing conflict avoidance strategies; step 4, local training: each agent independently executes a deep reinforcement learning algorithm based on the pre-optimization strategy set and local real-time data, generates a private experience pool and updates local model parameters according to the private experience pool; step 5, federated parameter aggregation: an aggregation server calculates aggregation weights based on performance indicators of each agent, aggregates local model parameters of each agent by using a weighted average strategy, generates global optimal model parameters and distributes the global optimal model parameters to each agent; step 6, real-time task allocation and monitoring: based on the global optimal model parameters and real-time data streams, tasks are dynamically allocated to corresponding agents and task execution states are monitored; step 7, dynamic game decision: when a resource conflict is monitored, real-time resource competition among agents is solved through an evolutionary game mechanism, and an evolutionary stable strategy is output as a conflict solution; and step 8, scheduling end evaluation and iteration: after the end of a scheduling period, a scheduling evaluation report is generated, system configuration is updated based on the evaluation result, and a next round of closed-loop iteration is triggered.

[0060] The embodiment of the application has the following beneficial effects:

[0061] 1. Federated-twin collaborative architecture: the deep coupling of federated learning and digital twinning is realized for the first time, the trial-and-error cost of federated training is reduced through twinning pre-simulation, the distributed optimization of the twinning model is ensured through federated learning, and privacy security and scheduling accuracy are considered; different from only a single federated learning framework, the architecture adds a twinning pre-play-verification closed loop, so that the global model convergence speed is improved;

[0062] 2. Multi-dimensional reward and game mechanism: a composite reward function integrating task utility, resource consumption and conflict penalty is designed, and an evolutionary game is introduced to solve real-time conflicts, so that the balance between individual optimization and group optimization is realized;

[0063] 3. Full-process closed-loop iteration: a closed-loop system of "pre-simulation-local training-global aggregation-real-time execution-evaluation iteration" is constructed, model parameters and environment configurations are continuously corrected through actual running data, and long-term adaptation capability is improved;

[0064] 4. Dynamic weight aggregation strategy: the federated aggregation weight is dynamically adjusted based on agent task completion rate, data quality and other indicators, so as to avoid the drag of low-quality local models on global performance. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings described below only show some of the embodiments of the present application, and the ordinary skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0066] Figure 1 A flowchart of a multi-agent cooperative scheduling method based on federated reinforcement learning and digital twinning provided by the embodiment of the present application is shown in the figure.

[0067] Figure 2 A specific implementation flowchart of a multi-agent cooperative scheduling method based on federated reinforcement learning and digital twinning provided by the embodiment of the present application is shown in the figure.

[0068] Figure 3 A coupling diagram of reinforcement learning and twin simulation provided by the embodiment of the present application is shown in the figure.

[0069] Figure 4 A dynamic game and federated aggregation flowchart provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0070] The embodiments of the present application will be described in detail below with reference to the drawings.

[0071] The embodiments of the present application will be described in detail below with reference to the drawings.

[0072] It is important to note that the various aspects described herein are exemplary in nature and are not intended to limit the scope, applicability, or configuration of the application in any way. Rather, the aspects can be configured in various ways to achieve similar results. Various aspects described herein can be implemented in software and / or firmware and / or hardware and / or other ways. For example, one or more aspects can be implemented using a processor of a device or using one or more devices associated with a device, such as one or more integrated circuits (ICs) and / or one or more processors on an integrated chip with one or more associated buses. Further, it should be appreciated that components of the application can be rearranged and / or complemented and that certain aspects of the application can be employed without others. In other words, the disclosure is not limited to the exact construction described herein.

[0073] It is also important to note that the present application can be implemented in various ways and that the application should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concept of the application to those skilled in the art.

[0074] In addition, the description above is intended to be illustrative, and not restrictive. Many other embodiments will be apparent to those of skill in the art upon reviewing the above description. The scope of the application should, therefore, be determined not with reference to the above description, but instead with reference to the appended claims, along with their full scope of equivalents.

[0075] The embodiment of the application provides a multi-agent collaborative scheduling method based on federated reinforcement learning and digital twinning, which can be applied to a test flow optimization process.

[0076] Referring to Figure 1 A flowchart of a multi-agent collaborative scheduling method based on federated reinforcement learning and digital twinning provided by the embodiment of the application is shown. As shown in Figure 1 and Figure 2 The method mainly comprises the following steps:

[0077] Step 1, system initialization: import test task list, environment parameters and agent capability matrix, and complete system basic configuration;

[0078] This implementation plan is based on the reliability testing of 100 large-scale devices, and revolves around the core objectives of "completing the task as quickly as possible and minimizing the overall probability of missed detections." The execution process strictly follows these constraints: the test is undertaken by a single team, with one shift per day and a working time not exceeding 12 hours; the test process requires the individual testing of subsystems A, B, and C to be completed first, and the E comprehensive test can only be carried out after all three have passed; if a problem is found in a certain process, it must be retested, and if it fails twice consecutively, the device is removed from the test; after the device has completed the test, it takes 0.5 hours to transport it out of and into the test hall, and the transportation process does not affect the testing of other devices. The testing hall is equipped with two test benches capable of parallel testing. The team is divided into four specialized groups: A, B, C (subsystem testing), and E (integrated testing). Each group has a dedicated workstation and can conduct tests independently. The initial setup and commissioning time for the equipment is 30 minutes for group A, 20 minutes for group B, 20 minutes for group C, and 40 minutes for group E. Normal testing time is 2.5 hours for group A, 2 hours for group B, 2.5 hours for group C, and 3.0 hours for group E. Abnormal situations include Y1 equipment failure, Y2 subsystem problems, Y3 operator errors, and Y4 integrated testing problems. Testing begins when "the testing equipment is in place and all equipment has been debugged." If the process is interrupted, it must be restarted. Based on the above constraints, the detailed implementation steps are as follows:

[0079] In specific implementation, step 1 (S1): System initialization

[0080] The system receives test task instructions and environment configuration, and completes the import and initialization of three core data items: task list, environment parameters, and agent capability matrix, providing basic data support for subsequent scheduling.

[0081] 1. Import Task List: Import a collection of tasks to be executed. Each of the tasks Includes priority Time window Resource requirements Attributes such as equipment testing timing constraints are included. For example, import a set of tasks to be executed for 100 large devices, with each device corresponding to one independent test task. The priority of the first device task (number: Dev-001) is set to 0.8 (priority range 0-1, the higher the value, the higher the priority, matching the requirement of "prioritizing the testing of key batch devices"), and the time window is set to "8:00 on the first day of test start to 20:00 on the third day after start" (reserving a buffer time for retesting). The resource requirements are specified as "occupying 1 test bench, 1 dedicated workstation for A / B / C / E each, and 2 testers (1 responsible for testing A / B subsystems, and 1 responsible for testing C / E subsystems)". Subsequent device tasks are imported sequentially according to the principle of "priority gradient allocation (0.5-0.8), time window staggered by 1-2 hours, and resource requirements matching the correspondence between test benches and workstations" to form a complete task list.

[0082] 2. Environment parameter configuration: import environment parameter vector wherein is the initial resource state, is the agent spatial distribution, is the environmental interference coefficient. The initial resource state specifies the initial state of the test bench and the station and the equipment test physical parameters, such as test bench 1, 2 are in idle state, A, B, C, E special station equipment are normal operation; the agent spatial distribution is defined according to the physical layout of the test hall, such as the agent bound to A station is located in the east zone No. 1 area of the hall, the agent bound to B station is located in the east zone No. 2 area of the hall, the agent bound to test bench 1 is located in the central No. 1 area of the hall, and the agent bound to test bench 2 is located in the central No. 2 area of the hall (to avoid the waste of device handling time caused by cross-area scheduling); the environmental interference coefficient is set to 0.05 (combined with the actual scene of the test hall, corresponding to "workshop temperature fluctuation ± 2℃, voltage fluctuation ± 3%" slight interference).

[0083] 3. Agent capability modeling: build initial agent capability matrix wherein N is the number of agents, K is the capability dimension (such as execution efficiency, resource carrying capacity, reliability), represents the capability value of the ith agent in the kth dimension. For example, the execution efficiency of the agent bound to A station (corresponding to A test group) is set to 0.92 (because the A group normal test time is 2.5 hours, and the operation proficiency of the test hand is high, the actual test time is shortened by 8% compared with the standard time), the resource carrying capacity is set to 8 sets / day (in a daily 12-hour shift, excluding debugging and equipment cooling time, a maximum of 8 devices can be tested for A subsystem), and the reliability is set to 0.98 (no fault record in the last 30 tests), and the rest of the agents bound to test benches, B / C stations are assigned values in turn according to the "capability matching group test characteristics" principle to form a complete agent capability matrix.

[0084] The initialized data is stored in the system knowledge base after verification as the basis for subsequent modeling and scheduling.

[0085] Step 2, reinforcement learning modeling: based on the test task target and constraint, define the state space, action space and reward function, and build a mathematical model of scheduling decision;

[0086] In specific implementation, step 2 (S2): the specific process of reinforcement learning modeling is as follows:

[0087] Combined with the test task target and constraint, the reinforcement learning model is built, the state space, action space and reward function are defined, and the algorithm support is provided for scheduling decision:

[0088] 1. State space definition: the state space S includes the environment state, the agent state and the task state, i.e. wherein is the environment state at time t, is the agent real-time capability matrix, is the remaining task set. t = 8:00 on the first day of test launch, contains both test benches 1 and 2 in idle state, A, B, C, E stations equipment are all normal operation, environmental interference coefficient 0.05; is a 6x3 dimensional matrix, binding the agent of A station execution efficiency 0.92, resource carrying capacity 8 sets / day, reliability 0.98, binding the agent of B station execution efficiency 0.88, resource carrying capacity 10 sets / day, reliability 0.95; binding the agent of C station execution efficiency 0.90, resource carrying capacity 7 sets / day, reliability 0.97; binding the agent of E station execution efficiency 0.85, resource carrying capacity 6 sets / day, reliability 0.96; the execution efficiency of the agent binding test benches 1 and 2 is 0.95, the resource carrying capacity is 1 set per period, and the reliability is 0.99; is the set of 100 devices to be tested, each device is labeled with priority, planned test period and resource demand.

[0089] 2. Action space definition: the action space A is the task-agent matching and resource allocation strategy, that is, , which means assigning task to agent and allocating resource . At 8:30 on the first day of test launch, action is generated, where is the first device test task, is the agent binding C station, contains the right to use test bench 1, C station dedicated equipment, and 1 tester responsible for C subsystem testing; simultaneously, action is generated, where is the second device test task, is the agent binding B station, contains the right to use test bench 2, B station dedicated equipment, and 1 tester responsible for B subsystem testing, ensuring that the actions have no resource overlap.

[0090] 3. Reward function design: a composite reward function is used to quantify the scheduling performance, and the expression is:

[0091]

[0092] where , , and satisfy , is the task completion utility at time t, Resource consumption cost at t, Task conflict penalty at t. Test start day 10:00, "Test bench 1 idle time 0.5 hours, A station idle time 0 hours" calculated 0.1, Completed A subsystem test (completion rate 30%), priority 0.8" calculated 0.9, "Test bench 1 idle time 0.5 hours, A station idle time 0 hours" calculated 0.1, 0 due to no resource conflict, substitute . The modeling result is output to the digital twin pre-simulation module and the local training module.

[0093] Step 3, digital twin pre-simulation: based on the digital twin model of the physical entity, pre-play the scheduling process and output the pre-optimized strategy set containing conflict avoidance strategies;

[0094] In specific implementation, as shown in Figure 3 Step 3 (S3): the process of digital twin pre-simulation includes:

[0095] Based on the digital twin model M of the physical entity and the task prediction parameters , the scheduling process is pre-played and the strategy is optimized:

[0096] 1. Twin model construction: the twin model M includes agent digital mapping, environment dynamic model and task evolution model. The agent digital mapping restores the physical parameters of the six agents, including test equipment power, test accuracy threshold, and integrates the equipment wear model, which runs according to the cumulative wear law of test time, for example, the failure probability of test bench 1 increases from 3% to 5% after 120 hours of work, which can match the Y1 equipment failure parameters; the environment dynamic model simulates the real-time fluctuation of test hall temperature (18-25℃) and voltage (380V±10%), and such fluctuation will be associated with the change of equipment performance, for example, when the voltage is lower than 360V, the test efficiency of A station will decrease by 10%; the task evolution model simulates the device test state transition according to the "subsystem test - retest - comprehensive test" process, among which the retest link is triggered when Y2 subsystem problem or Y3 tester error is detected, and the retest time is calculated as 1.2 times of the original test time, which realizes the accurate reproduction of the actual scheduling scene.

[0097] 2. Pre-optimized strategy generation: input task prediction parameters (including task prediction set and environment disturbance prediction for future T time steps), generate pre-optimized strategy set through twin simulation: wherein, is the twin simulation function, comprising task allocation pre-strategy, resource reservation scheme and conflict avoidance rule. For this instance, input task prediction parameters , wherein is the set of 20 devices to be tested in the next 8 hours (T=8), is the environmental disturbance prediction for the next 8 hours (including 2 voltage fluctuation warnings), through the twin simulation function generate a pre-optimization strategy set , wherein is the task allocation pre-strategy of "allocating high-priority devices to idle stations first", is the resource reservation scheme of "reserving 1 standby station for voltage fluctuation period", is the conflict avoidance rule of "peak-shaving 1 hour scheduling for associated stations on the same test bench".

[0098] 3. Robustness evaluation: simulate K extreme scenarios (such as sudden resource reduction, sudden increase in tasks), calculate the robustness index of the pre-strategy , eliminate strategies below the threshold . This instance K=3 extreme scenarios, scenario 1 is "test bench 1 suddenly Y1 failure (working for 10 hours)", scenario 2 is "adding 5 emergency device test tasks in 1 hour", scenario 3 is "B, C stations fail at the same time", calculate the robustness index of each pre-strategy (value 0-1). In scenario 1 (schedulable test bench 2 takes on the task), in scenario 1 (no task delay after standby station is enabled), eliminate . The pre-simulation result is output to the local training module for guiding the experience pool generation.

[0099] Step 4, local training: each agent independently executes a deep reinforcement learning algorithm based on the pre-optimization strategy set and local real-time data, generates a private experience pool and updates the local model parameters accordingly;

[0100] In implementation, step 4 (S4): the specific process of local training is as follows:

[0101] Each agent independently executes a DRL algorithm (such as PPO, DDPG) based on the pre-simulation result and local data for training:

[0102] 1. Experience pool construction: the agent executes actions based on the pre-strategy and local real-time data, records interaction data, and generates a private experience pool

[0103] ,​

[0104] wherein is the k-th local observation state of the i-th agent, is the action performed, is the immediate reward, is the next state. The agent (i = 1) bound to the A station generates wherein the 10th data is , contains "A station occupancy, test progress 50%", is "continue to perform A subsystem test", , contains "A station occupancy, test progress 60%".

[0105] 2. Local model update: the local model parameters are updated using the DRL algorithm. Taking PPO as an example, the parameter optimization is realized by minimizing the clipping objective function: wherein, wherein is the ratio of new and old policies, is the advantage function, is the clipping coefficient. The agent (i = 2) bound to the B station updates the model parameters through 1000 rounds of training. At the 500th round, the objective function value drops to 0.03 (85% lower than the initial value), and the policy stability meets the requirements. 3. Model verification: the model performance is evaluated using the local verification set (containing 20 device test data). The verification index is "task allocation accuracy". Only when the accuracy is

[0106] , the current model parameters are retained. The agent (i = 4) bound to the E station verifies , and the parameters are retained; the agent (i = 3) bound to the C station initially , re-trains after supplementing 100 Y2 abnormal scene data, and finally and retains the parameters. After local training is completed, each agent uploads the model parameters

[0107] (not containing original data) to the federal parameter aggregation module.

[0108] Step 5, federal parameter aggregation: the aggregation server calculates the aggregation weight based on the performance indicators of each agent, aggregates the local model parameters of each agent using the weighted average strategy, generates the globally optimal model parameters, and distributes them to each agent;

[0109] In specific implementation, the specific process of step 5 (S5): federal parameter aggregation is as follows: ​

[0110] The federal aggregation server receives the local model parameters of each agent, and generates a global model using a weighted average strategy:

[0111] 1. Weight calculation: Calculate the aggregation weight based on the historical performance indicators of the agent (such as task completion rate , model accuracy ): where, is the weight of the ith agent, satisfying .

[0112] 2. Global parameter generation: Aggregate local parameters using weighted average:

[0113]

[0114] where, is the local model parameter of the ith agent, is the global optimal model parameter;

[0115] 3. Parameter distribution: Distribute to each agent, such as the 6 agents in this example, for updating the local model.

[0116] The aggregation process ensures parameter security through encrypted transmission (such as homomorphic encryption) to avoid model privacy leakage.

[0117] Step 6, real-time task allocation and monitoring: Based on the global optimal model parameters and real-time data flow, dynamically allocate tasks to corresponding agents and monitor task execution status;

[0118] In specific implementation, step 6 (S6): Real-time task allocation and monitoring

[0119] Based on the global model and real-time data flow, dynamic task allocation and execution monitoring are achieved:

[0120] 1. Real-time state perception: Through Internet of Things sensors and artificial recording systems, receive real-time data flow , including environment state updates (test bench 1 working time 12 hours, workshop temperature 22℃), task state updates (complete B subsystem test, enter C subsystem test queue), agent state updates (the agent bound to C station reduces the execution efficiency to 0.88), update system state every 5 minutes .

[0121] 2. Task allocation decision: Generate real-time allocation actions based on the global policy network: where, ​For the global strategy function, output the task-agent matching scheme at the current time step. t = 14:00 the next day after the test starts, contains "A subsystem test" , A station is idle", generate , that is, allocate to the agent bound to the A station, and the supporting resources are the use right of test bench 2, A station special equipment, and 1 tester. After the agent receives the action, it immediately starts A subsystem test.

[0122] 3. Execution monitoring: Collect the execution data (test progress, equipment temperature, tester operation record) of each agent through the Internet of Things interface to generate an execution log . t = 15:30 the next day after the test starts, detects that A subsystem test is delayed by 30 minutes (because the tester is temporarily replaced), triggers an abnormal warning, records the abnormal type as "tester handover delay", and synchronously transmits the warning information to the dynamic game decision module and the end evaluation module.

[0123] The monitoring data is synchronously output to the dynamic game decision module and the end evaluation module.

[0124] Step 7, dynamic game decision: when the resource conflict is monitored, solve the real-time resource competition between agents through the evolutionary game mechanism, and output the evolutionary stable strategy as the conflict resolution scheme;

[0125] In specific implementation, as shown in Figure 4 , step 7 (S7): the process of dynamic game decision is as follows:

[0126] When the monitoring module detects a resource conflict (such as multiple agents competing for the same resource), start the evolutionary game mechanism to solve the conflict:

[0127] 1. Game model construction: define N = 2 agents as game participants, that is, the agent bound to ( ) and the agent bound to ( ), both of which compete for the use right of B station; the strategy space , where is "B station allocated to ", is "B station allocated to ", is "B station 16:00 allocated to , 18:00 allocated to ", is the strategy distribution of the i-th participant.

[0128] 2. Profit calculation: individual profit is calculated based on task utility and resource cost and the group average profit . When , of ( priority 0.8, no delay), of ( delay 2 hours), ; when , of (no delay), of (delay 2 hours but no conflict), .

[0129] 3. Strategy evolution: through the replication dynamic equation

[0130]

[0131] update the agent strategy distribution, denotes the diffusion speed of strategy i, denotes the proportion of agents using strategy i in the total, denotes the expected profit of agents using strategy i, denotes the average expected profit of the entire group. For strategy, , at this time the strategy distribution reaches the evolutionary stable strategy (ESS).

[0132] 4. Conflict resolution: the ESS corresponding strategy is used as a conflict resolution solution and is issued to and , enable B station at 16:00, enable B station at 18:00 to avoid resource competition.

[0133] The game results are synchronized into the system knowledge base for subsequent strategy optimization.

[0134] Step 8, scheduling end evaluation and iteration: after the end of the scheduling period, generate a scheduling evaluation report, update the system configuration based on the evaluation results, and trigger the next round of closed-loop iteration.

[0135] In implementation, the specific process of step 8 (S8): scheduling end evaluation and iteration is as follows:

[0136] When a scheduling period ends, the system conducts a comprehensive evaluation and updates the configuration:

[0137] 1. Evaluation report generation: output the scheduling evaluation report R, containing core indicators: task completion rate , resource utilization rate , conflict occurrence rate , and average scheduling delay ;

[0138] 2. Iterative correction amount calculation: based on the deviation of the evaluation indicators and the target value, calculate the iterative correction amount :

[0139]

[0140] where Target is the target value of the indicator, Indicator is the actual value, and the correction coefficient is 0.1. The resource utilization rate target value is 90%, and the calculation result is ; the average scheduling delay target value is 20 minutes, and the calculation result is minutes.

[0141] 3. Configuration update: update the initial configuration parameters of the system:

[0142]

[0143] where is the configuration update function, is the initial parameter of the next round of closed loop. In this example, the resource carrying capacity of the agents bound to A and B stations is increased by 0.5 units / day, and the scheduling time window interval is shortened by 0.5 minutes.

[0144] 4. Knowledge base update: store key data such as evaluation reports, conflict resolution solutions, and optimization parameters into the long-term knowledge base to realize experience reuse.

[0145] After the update is completed, the system returns to step 1 to start the next round of "initialization-scheduling-optimization" closed loop.

[0146] The multi-agent collaborative scheduling method based on federated reinforcement learning and digital twinning provided in this embodiment reduces the trial-and-error cost of federated training through twinning pre-simulation, guarantees the distributed optimization of the twinning model through federated learning, and balances privacy security and scheduling accuracy; a composite reward function is designed to integrate task utility, resource consumption, and conflict penalty, and evolutionary game is introduced to solve real-time conflicts, achieving a balance between individual optimization and group optimization; a closed loop system of "pre-simulation-local training-global aggregation-real-time execution-evaluation iteration" is constructed, and model parameters and environment configurations are continuously corrected based on actual running data to improve long-term adaptation ability; the federated aggregation weight is dynamically adjusted based on indicators such as agent task completion rate and data quality, avoiding the drag of low-quality local models on global performance.

[0147] The method of the present application will be further described below in conjunction with a specific embodiment:

[0148] 1. Experimental subjects and scenarios

[0149] A large device containing A, B, C parallel subsystems and D (coupling system), E (integrated test) subsystems is taken as the test object, and the full-process test scenarios of two batches of 100 devices are simulated. The test environment strictly replicates the constraints of the competition: the test hall contains 2 parallel test benches and 4 special workstations (A, B, C, E), and each device takes 0.5h to be transported in / out, the equipment debugging time (A: 30min, B: 20min, C: 20min, E: 40min) and the normal test time (A: 2.5h, B: 2h, C: 2.5h, E: 3.0h) are all configured according to the competition.

[0150] 2. Abnormal scenario parameters

[0151] Y1 (equipment failure): 3% / 4% / 2% / 3% cumulative failure probability of A / B / C / E within 120h, increased to 5% / 7% / 6% / 5% from 120-240h, and forced replacement for ≥240h;

[0152] Y2 (subsystem problem): A / B / C original problem probability 2.5% / 3% / 2%;

[0153] Y3 (tester error): A / B / C / E error probability 3% / 4% / 2% / 2%, with misjudgment (Y31) and missed judgment (Y32) each accounting for 50%;

[0154] Y4 (integrated test problem): D subsystem problem probability 0.1%.

[0155] 3. Experimental grouping and comparison model

[0156] Three groups of comparison experiments are set, and 1000 times of Monte Carlo simulation are performed for each group to ensure statistical stability:

[0157] Control group 1 (traditional FCFS scheme): using "first-come-first-served" scheduling strategy, single division team 12h fixed shift, process interruption and restart, no dynamic optimization mechanism;

[0158] Control group 2 (single reinforcement learning scheme): centralized scheduling model based on DDPG algorithm, without digital twin pre-simulation and federated learning privacy protection;

[0159] Experimental group (the scheme of the present application): using "federated reinforcement learning + digital twin + multi-agent collaboration" architecture, the core modules include:

[0160] Digital twin (DT) module: real-time mapping of test bench, workstation, and device status, and pre-playing abnormal scenarios;

[0161] Federal reinforcement learning (FL) module: 4 stations + 2 test benches, a total of 6 local agents, dynamic weight aggregation to generate a global strategy;

[0162] Multi-agent collaboration module: station agent is responsible for local scheduling, and test bench agent is responsible for global resource balancing.

[0163] 4. Evaluation index

[0164] The core evaluation indexes include: average number of days for task completion (T); average number of devices passing the test (S); total missed judgment probability (P_L): missed judgment device number / passing device number; total misjudgment probability (P_W): misjudgment times / total test times; professional test group effective working time ratio (YXB).

[0165] The experimental results are shown in Table 1:

[0166] Table 1

[0167]

[0168] As can be seen, the twin preposition positioning E station bottleneck, federal agent dynamic sequencing, T shortens 8.7 days, the risk of missed judgment is significantly reduced, and the station load is more balanced.

[0169] It should be understood that parts of the present application can be realized by hardware, software, firmware or their combination.

[0170] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A multi-agent collaborative scheduling method based on federated reinforcement learning and digital twinning, characterized in that, include: Step 1, System Initialization: Import the test task list, environment parameters, and agent capability matrix to complete the basic system configuration; Step 2, Reinforcement Learning Modeling: Based on the test task objectives and constraints, define the state space, action space, and reward function, and construct a mathematical model for scheduling decisions; Step 2 specifically includes: Step 2.1, define the state space as including the environment state at any given time, the agent's real-time capability matrix, and the set of remaining tasks; Step 2.2, define the action space as the matching relationship between tasks and agents and the resource allocation strategy; Step 2.3: Quantify scheduling performance using a composite reward function, wherein the composite reward function is... in, , , The normalized weighting coefficients are used to adjust the relative importance of the three core scheduling metrics in the overall performance evaluation and satisfy the following conditions: , Let t be the utility of task completion at time t. Let be the resource consumption cost at time t. Penalty for task conflict at time t; Step 3, Digital Twin Pre-simulation: Based on the digital twin model of the physical entity, the scheduling process is simulated and a set of pre-optimized strategies including conflict avoidance strategies is output; Step 3 specifically includes: Step 3.1: Construct a digital twin model, wherein the digital twin model includes an agent digital mapping, an environmental dynamic model, and a task evolution model, used to replicate the real-world test environment and simulate test processes and anomalies; Step 3.2, based on task prediction parameters Simulation is performed using a digital twin model to generate a pre-optimization strategy set, including task allocation pre-strategies, resource reservation schemes, and conflict avoidance rules. The task prediction parameters... Including the task prediction set and environmental disturbance prediction for the next T time steps, the expression for simulation using a digital twin model is: in, For twin simulation functions, This represents a complete digital twin model built upon physical entities; Step 3.3: Simulate various extreme scenarios, evaluate the robustness of the generated pre-policies, and remove policies in the pre-optimized policy set whose robustness index is lower than a preset threshold. Step 4, Local Training: Each agent independently executes a deep reinforcement learning algorithm locally based on the pre-optimized policy set and local real-time data, generates a private experience pool, and updates the local model parameters accordingly. Step 5, Federation parameter aggregation: The aggregation server calculates the aggregation weight based on the performance indicators of each agent, aggregates the local model parameters of each agent using a weighted average strategy, generates the globally optimal model parameters, and distributes them to each agent. Step 5 specifically includes: Step 5.1: Encrypt and transmit the local model parameters of each agent to the aggregation server, and calculate the aggregation weight based on the historical performance metrics of the agents. in, Let be the weight of the i-th agent. For task completion rate, For model accuracy, satisfy ; Step 5.2: Use a weighted average to aggregate the local parameters to obtain the globally optimal model parameters. in, These are the local model parameters for the i-th agent; Step 5.3: Distribute the globally optimal model parameters to each agent; Step 6, Real-time task allocation and monitoring: Based on the globally optimal model parameters and real-time data stream, dynamically allocate tasks to the corresponding agents and monitor the task execution status; Step 7, Dynamic Game Decision-Making: When resource conflicts are detected, the real-time resource competition between agents is resolved through an evolutionary game mechanism, and an evolutionary stable strategy is output as a conflict resolution solution. Step 8, Scheduling Closure Evaluation and Iteration: After the scheduling cycle ends, a scheduling evaluation report is generated, the system configuration is updated based on the evaluation results, and the next round of closed-loop iteration is triggered.

2. The method according to claim 1, characterized in that, Step 1 specifically includes: Step 1.1: Import the test task list, wherein the test task list includes multiple test tasks, and each test task includes priority, time window and resource requirement attributes; Step 1.2, import environmental parameters, wherein the environmental parameters include initial resource state, agent spatial distribution and environmental interference coefficient; Step 1.3: Construct the agent capability matrix, wherein the agent capability matrix is ​​used to characterize the capability values ​​of each agent in terms of execution efficiency, resource carrying capacity and reliability.

3. The method according to claim 1, characterized in that, Step 4 specifically includes: Step 4.1: Each agent builds a private experience pool by executing actions and recording interaction data, and updates the local model parameters using proximal policy optimization or deep deterministic policy gradient algorithm. Step 4.2: After the update, the model performance is verified using a local validation set. The model parameters are retained when the accuracy meets the preset conditions.

4. The method according to claim 1, characterized in that, Step 6 specifically includes: Step 6.1: Receive real-time data streams through IoT sensors and a manual recording system to update the system status; Step 6.2: Based on the updated system state and globally optimal model parameters, generate a real-time task-agent matching scheme using the global policy network. in, This is the global policy function. Indicates the action at time t. This represents the system state at time t, including the current state of all tasks, the resources and capabilities of the agents, and environmental disturbances. Step 6.3: Collect execution data through the IoT interface to generate execution logs, and trigger an alert when an anomaly is detected.

5. The method according to claim 1, characterized in that, Step 7 specifically includes: Step 7.1: Define the intelligent agents participating in resource competition as game participants and their strategy space; Step 7.2: Based on task utility and resource cost, calculate the individual benefits and average group benefits of each participant under different strategies. Step 7.3: Update the agent's policy distribution by replicating the dynamic equations until an evolutionarily stable policy is reached; Step 7.4: Issue the evolutionary stabilization strategy as a conflict resolution to the relevant agents.

6. The method according to claim 1, characterized in that, Step 8 specifically includes: Step 8.1: After a scheduling cycle ends, output a scheduling evaluation report R, where the core indicators in the evaluation report include the task completion rate. Resource utilization rate Conflict incidence rate and average scheduling delay ; Step 8.2: Calculate the iterative correction amount based on the deviation between the core indicator and the target value. : Where Target is the target value of the indicator, and Indicator is the actual value of the core indicator. This is a correction factor; Step 8.3: Update the system configuration based on the iterative correction amount, and simultaneously store the key data of this scheduling in the knowledge base. The key data includes the evaluation report, conflict resolution methods, and optimization parameters. The expression for updating the system configuration is: in, To configure the update function, These are the initial parameters for the next closed loop; Step 8.4: After the update is completed, the system returns to step 1 and starts the next round of closed-loop iteration.

Images