Federated learning participation compensation method and system based on reinforcement learning and digital twinning

Abstract

The application discloses a kind of based on reinforcement learning and federated learning participation compensation method and system of digital twinning, belong to artificial intelligence technical field, including the following steps: S1, the current state of several client devices of client network is evaluated using central server, determine state information;S2, according to state information, construct joint optimization problem, and solve, obtain optimal strategy;S3, according to optimal strategy, carry out federated learning training;S4, after federated learning training, carry out synchronization.The technology described in the application can be widely applied in industrial internet of things, smart city, mobile edge computing and intelligent medical field, especially suitable for the real physical environment that equipment connection is unstable and frequently drops, has strong practical application value and broad commercial prospect.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a federated learning participation compensation method and system based on reinforcement learning and digital twins. Background Technology

[0002] In recent years, federated learning, as an emerging distributed machine learning paradigm, allows model training on a large number of distributed edge devices without uploading their local private data to a central server. This technology offers unprecedented advantages in protecting data privacy and security by computing model updates locally and only uploading encrypted model parameters to a central server for aggregation, demonstrating enormous application potential in fields such as finance, healthcare, and autonomous driving. Traditional federated learning algorithms, represented by FedAvg, typically involve a central server selecting a subset of clients for training, distributing the global model to the selected clients, clients training on their local data and computing model updates, uploading the local model updates to the server, and the server performing a weighted average of the collected model updates to generate a new generation of global models.

[0003] However, when applying federated learning to the real physical world, especially in industrial IoT environments, its performance and stability face severe challenges. The most critical issue is training interruption caused by intermittent client disconnections. In actual deployments, client devices typically connect to the central server via unreliable wireless channels. These channels are susceptible to signal interference, network congestion, device movement, and battery depletion, causing frequent and unpredictable disconnections during training, preventing successful uploading of local model updates. This intermittent disconnection phenomenon is particularly prominent in industrial scenarios, severely impacting the training efficiency and model convergence quality of federated learning.

[0004] Traditional methods for handling system outages primarily employ passive fault-tolerance strategies. These either wait for the out-of-systems device to recover, which leads to training stagnation, or directly discard the contributions of the out-of-systems device, resulting in insufficient data utilization and degraded model performance. These methods are essentially "post-hoc remediation" and cannot fundamentally solve the problem of information loss caused by outages. More importantly, existing methods lack the ability to learn and utilize the historical behavior patterns of out-of-systems devices, as well as a global understanding of the system state and forward-looking planning for future benefits.

[0005] In recent years, digital twin technology has been widely used in the industrial field, enabling state monitoring, behavior prediction, and decision optimization by creating virtual copies of physical entities. However, the application of digital twin technology in federated learning, particularly for predicting and compensating for downtime client model updates, remains in the exploratory stage and has not yet yielded mature technical solutions. Meanwhile, deep reinforcement learning has demonstrated powerful capabilities in complex decision-making problems, but how to apply it to virtual participation in decision-making and compensation strategy optimization within federated learning requires further research and practical verification.

[0006] Therefore, there is an urgent need in this field for a novel technical solution that can learn the historical gradient patterns of the client through digital twin technology, generate high-quality virtual compensation updates when the client goes offline, make intelligent decisions using deep reinforcement learning, dynamically optimize virtual participation strategies and aggregate weight allocation, and achieve synergy between physical training and virtual compensation, transforming from "passive fault tolerance" to "active intelligent compensation," thereby systematically improving the robustness, training efficiency, and final model performance of federated learning in real and complex environments. Summary of the Invention

[0007] To address the problems of training interruption, unstable model convergence, and reduced accuracy caused by intermittent client disconnections in real physical network environments, this invention proposes a federated learning participation compensation method and system based on reinforcement learning and digital twins.

[0008] The technical solution of this invention is: a federated learning participation compensation method based on reinforcement learning and digital twins, comprising the following steps:

[0009] S1. Use the central server to assess the current status of several client devices in the client network and determine the status information;

[0010] S2. Based on the state information, construct a joint optimization problem and solve it to obtain the optimal strategy;

[0011] S3. Based on the optimal strategy, perform federated learning training;

[0012] S4. Synchronize after federated learning training.

[0013] Furthermore, in S1, the state information adopts a multi-dimensional state vector, including the device's online status, reliability, performance capability, connection stability, data quality, historical performance, digital twin prediction confidence, and virtual participation weight.

[0014] Furthermore, in S2, the expression for solving the joint optimization problem is:

[0015] ;

[0016] ;

[0017] ;

[0018] ;

[0019] ;

[0020] ;

[0021] ;

[0022] in, The strategies followed by deep reinforcement learning agents. As a discount factor, For state variables, As an action variable, For the reward function, For training round index, The total number of training rounds, This is the first constraint condition. This is the second constraint. This is the third constraint. This is the fourth constraint. This is the fifth constraint. For the type of decision-making participation, for Dimensional device selection vector, for Dimensional weight allocation vector, For the first The system state of the wheel, For state transition, For mathematical expectation, For performance improvement amount, Training takes time, For virtual participation numbers, For relay costs, For communication costs, To participate in the balanced rewards, For virtual quality rewards, The weighting coefficient for the performance improvement. The weight coefficients represent the training time. The weighting coefficient for the number of virtual participants. This is the weighting coefficient for relay costs. This is the weighting coefficient for communication costs.

[0023] Furthermore, S3 includes the following sub-steps:

[0024] S31. Distribute the global model to all online clients and wait for local training to complete;

[0025] S32. After completing local training, collect local model updates from all successfully responding physical clients and retain complete training records.

[0026] S33. Select a subset for aggregation from the collected local model updates of physical clients;

[0027] S34. Activate the digital twin model corresponding to the virtual device from the set of digital twin models and generate a compensatory model update;

[0028] S35. Perform weighted aggregation calculation on the selected local model update and compensatory model update to generate a new version of the global model.

[0029] Furthermore, in S32, the compensatory model weights are updated by the compensatory model. The expression is:

[0030] ;

[0031] in, For the next gradient, For learning rate, For the first Global model weights for the wheel;

[0032] In S33, a selection mechanism based on quality scoring and participation fairness is adopted. After collecting local model updates from all physical clients, a subset for aggregation is selected by maintaining device participation counters and calculating fairness adjustment factors.

[0033] In S35, the training loss based on the digital twin model is used to calculate the prediction confidence score, which is then used to adjust the weights of virtual devices in the weighted aggregation. The expression is:

[0034]

[0035] in, For the training loss of the digital twin model, To obtain the minimum value, To obtain the maximum value;

[0036] In S35, the new version of the global model The expression is:

[0037] ;

[0038] ;

[0039] ;

[0040] ;

[0041] in, This refers to any client that actually participated in the aggregation in this round. To successfully respond and upload the locally updated physical client, For clients detected as disconnected, For the first The set of all clients actually selected to participate in the aggregation. For physical device weights, For the aggregate weight of virtual devices, For virtual device weights, This is a subset of physical clients that actually participate in the aggregation after being filtered. For physical clients In the Local model updates uploaded in rounds, For the selected subset of disconnected clients that actually generate virtual compensation, For physical client collection, This is a collection of clients that have experienced disconnections.

[0042] Furthermore, in S3, pre-training is performed before federated learning training. Specifically, within a preset round, the central server forces all registered clients to participate in training, including clients that simulate disconnection.

[0043] Furthermore, in S4, after each round of training in federated learning, real gradients and performance data are collected from all trained physical clients and labeled as real data, and it is recorded whether they participate in this round of aggregation; predicted gradients are collected from virtual participating clients and labeled as virtual data; when the training conditions are met, the digital twin model is retrained and the multidimensional state vector is updated.

[0044] Based on the above methods, this invention also proposes a federated learning participation compensation system based on reinforcement learning and digital twins, including a central server and several client devices.

[0045] The central server includes:

[0046] State assessment and prediction module: used to maintain multi-dimensional state vectors and manage digital twin models;

[0047] Deep reinforcement learning decision module: used for weight allocation for optimal policy and weighted aggregation;

[0048] Virtual Participation Compensation Module: Used to maintain an LSTM gradient prediction model for each client, employing a weighted training mechanism;

[0049] Enhanced aggregation module: used for weighted aggregation calculations;

[0050] Communication management module: Used to manage client communication and perform model distribution, collection, and synchronization;

[0051] Status Management and Fairness Guarantee Module: Used to track device status and participation count, and calculate the fairness adjustment factor for the fairness selection mechanism.

[0052] The beneficial effects of this invention are:

[0053] (1) This invention addresses the problem of intermittent client disconnection in industrial IoT scenarios by proposing a collaborative solution that uses digital twin technology for high-quality virtual compensation and combines deep reinforcement learning for intelligent decision optimization. This "proactive intelligent compensation" mechanism represents a fundamental breakthrough compared to the traditional "passive fault tolerance".

[0054] (2) The present invention fully considers the multidimensional features of the client state, as well as the inherent contradictions and trade-offs between model accuracy, training efficiency and system stability. The four-step closed-loop adaptive framework constructed is more intelligent and efficient than the traditional "open-loop" model.

[0055] (3) This invention, through an innovative dynamic weighted hybrid aggregation protocol, not only achieves seamless integration of physical updates and virtual compensation, but also ensures through a stable normalization mechanism that the update step size of the global model remains constant regardless of how the virtual participation ratio changes, thereby maximizing the smoothness and convergence speed of the model training process.

[0056] (4) The system provided by the present invention provides solid and practical technical support for the implementation of the above methods through the modular integration and tight coupling of six major functional modules. In particular, the training and aggregation decoupling mechanism and the continuous updating of digital twins of all devices ensure the long-term adaptive capability of the system.

[0057] (5) The technology described in this invention can be widely applied in fields such as industrial Internet of Things, smart cities, mobile edge computing and smart healthcare. It is particularly suitable for real physical environments where device connections are unstable and disconnections are frequent, and has strong practical application value and broad commercial prospects. Attached Figure Description

[0058] Figure 1 A flowchart illustrating a federated learning participation compensation method based on reinforcement learning and digital twins;

[0059] Figure 2 This is a schematic diagram of the overall system architecture based on deep reinforcement learning and digital twins provided in an embodiment of the present invention;

[0060] Figure 3 This is a closed-loop schematic diagram of a joint optimization decision-making process for a deep reinforcement learning agent provided in an embodiment of the present invention.

[0061] Figure 4 This is a schematic diagram illustrating the principle of LSTM-based digital twin gradient prediction and compensation provided in an embodiment of the present invention.

[0062] Figure 5 This is a flowchart illustrating an adaptive intelligent federated learning method provided in an embodiment of the present invention. Detailed Implementation

[0063] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0064] like Figure 1 As shown, this invention provides a federated learning participation compensation method based on reinforcement learning and digital twins, comprising the following steps:

[0065] S1. Use the central server to assess the current status of several client devices in the client network and determine the status information;

[0066] S2. Based on the state information, construct a joint optimization problem and solve it to obtain the optimal strategy;

[0067] S3. Based on the optimal strategy, perform federated learning training;

[0068] S4. Synchronize after federated learning training.

[0069] The core technical problem of this invention is how to generate high-quality virtual compensation updates through digital twin technology when the client disconnects without violating the principle of privacy protection, and how to achieve optimal synergy between physical training and virtual compensation through deep reinforcement learning intelligent decision-making, thereby transforming the traditional passive fault-tolerant federated learning framework into a robust distributed learning system with active intelligent compensation and continuous adaptive optimization capabilities.

[0070] The core technical concept of this invention lies in deeply coupling deep reinforcement learning with digital twin technology to construct a four-step closed-loop framework of "prediction-optimization-compensation-synchronization". Specifically, digital twin technology learns the historical gradient patterns of the client to generate high-quality virtual compensation updates for offline devices; the deep reinforcement learning agent is responsible for global decision optimization, dynamically determining virtual participation strategies and aggregation weight allocation; the two are seamlessly integrated through an innovative dynamic weighted hybrid aggregation protocol to jointly construct a new generation of global models.

[0071] This method is executed by a central server, and the system architecture explicitly includes:

[0072] like Figure 2 As shown, a central server acts as the system coordination center, responsible for executing, scheduling, and managing all core tasks throughout the federated learning process; [The text abruptly ends here, likely due to an incomplete translation or source material.] A client network consisting of individual client devices capable of independent computing and storage; deployed on a central server and containing... Individual and A collection of digital twin models consisting of independent digital twin models corresponding to each physical client device; deployed on a central server, providing a global insight into the entire client network status and making optimal virtual participation decisions for each round of federated learning; a centralized intelligent decision-making agent.

[0073] The method is executed by the central server based on the above system architecture, such as... Figure 5 As shown, it includes the following well-defined and collaborative steps:

[0074] Prediction Steps: At the start of each federated learning training round, the central server first evaluates the current state of all N client devices in the client network, detecting online and offline devices; for offline devices, it obtains their virtual participation candidate status and prediction confidence from the digital twin model set; this step aims to comprehensively grasp the current state of the system and provide accurate state information for subsequent decisions.

[0075] Optimization steps: Utilizing the centralized intelligent decision-making agent, the state information is comprehensively analyzed and prospectively evaluated, and an optimal joint decision is output accordingly. This decision includes: participation strategy selection (waiting, physical devices only, physical devices plus virtual devices, or relay transmission), virtual device selection (selecting a subset for virtual participation from disconnected devices), and aggregation weight allocation. This step aims to achieve accurate compensation for virtual devices and optimization of aggregation strategies through intelligent joint optimization.

[0076] The compensation step involves performing federated learning training based on the decision results of the optimization step. Specifically, this includes: distributing the global model to all online clients and waiting for them to complete their local training; collecting local model updates from all successfully responding physical clients and retaining complete training records; selecting a subset of physical updates for aggregation based on quality scores and participation fairness; for the virtual participating devices selected in the decision, activating their corresponding digital twin models from the digital twin model set to generate compensatory model updates; performing weighted aggregation calculations on the selected physical model updates and virtual compensation updates to generate a new version of the global model; and retaining updates from all trained devices for subsequent updates to the digital twin models. This step achieves synergy between physical training and virtual compensation, ensuring the integrity of the training data.

[0077] Synchronization steps: Before each training round begins, the central server broadcasts the current global model to all online clients to ensure that clients that have resumed online can synchronize to the latest model state and avoid training bias caused by inconsistent model versions.

[0078] The specific execution process of this synchronization mechanism is as follows: For any client If it is in the first The wheel is offline and in the first... If the system is restored to online status in the next round, then in the [number]th round... Before each round of training begins, the server sends the latest global model to it. Client Upon receiving the data, the client updates its local model. This synchronization operation ensures that the client... In the The local training round is based on the latest global model state, not the outdated model from before the outage. The technical significance of this synchronization mechanism is to avoid training bias caused by model version inconsistencies after device recovery, ensuring that all participating devices always perform local training based on the latest global model. Furthermore, for devices that have been offline for extended periods, this mechanism effectively prevents excessive version differences between their local and global models, thus avoiding problems such as gradient explosion or convergence difficulties during training recovery.

[0079] In this embodiment of the invention, in S1, the state information adopts a multi-dimensional state vector, including the device's online status, reliability, performance capability, connection stability, data quality, historical performance, digital twin prediction confidence, and virtual participation weight.

[0080] The core task of the centralized intelligent decision-making agent is to make the optimal joint participation decision for this round based on a global insight into the state of the entire client network. This decision comprehensively considers the selection of physical clients, compensation for virtual participation, and dynamic allocation of weights.

[0081] The values ​​of each dimension are all normalized to the interval [0,1].

[0082] In this embodiment of the invention, in S2, the virtual participant decision-making problem is constructed and solved as a joint optimization problem under a constrained Markov decision process. This joint optimization problem is precisely formalized as finding an optimal strategy. To solve the optimization problem.

[0083] The expression for solving the joint optimization problem is:

[0084] ;

[0085] ;

[0086] ;

[0087] ;

[0088] ;

[0089] ;

[0090] ;

[0091] in, The strategies followed by deep reinforcement learning agents. As a discount factor, For state variables, As an action variable, For the reward function, For training round index, The total number of training rounds, This is the first constraint condition. This is the second constraint. This is the third constraint. This is the fourth constraint. This is the fifth constraint. For the type of decision-making participation, for Dimensional device selection vector, for Dimensional weight allocation vector, For the first The system state of the wheel, For state transition, For mathematical expectation, For performance improvement amount, Training takes time, For virtual participation numbers, For relay costs, For communication costs, To participate in the balanced rewards, For virtual quality rewards, The weighting coefficient for the performance improvement. The weight coefficients represent the training time. The weighting coefficient for the number of virtual participants. This is the weighting coefficient for relay costs. This is the weighting coefficient for communication costs.

[0092] To solve the aforementioned joint optimization problem, this invention employs a model-free solution method based on deep reinforcement learning. Specifically, the solution process consists of two stages: an offline training stage and an online deployment stage. In the offline training stage, the deep reinforcement learning agent is trained on a large scale in a simulation environment using the SAC algorithm. This algorithm approximates the optimal policy by constructing a dual-network architecture consisting of a policy network and a value network. The policy network is state-based. As input, output three-dimensional motion The probability distribution of the value network; the value network is based on the state. and actions Given input, output a value estimate for the state-action pair. The training process employs an experience playback mechanism, which uses the experience quadruples generated from the agent's interaction with the environment. The data is stored in an experience replay buffer with a capacity of 10,000, and then randomly sampled from it in batches of 64 for training. The value network updates its parameters by minimizing the temporal difference error, with the goal of maximizing the current... Value estimate approximates the target Value, Target Value is determined by instant rewards Add the discount value of the next state The calculations show that the policy network updates its parameters using the policy gradient method, aiming to maximize the expected cumulative reward. To ensure training stability, the system maintains a target value network, whose parameters are periodically synchronized from the main value network via soft updates. Set to 0.005. After learning in the simulation environment, the policy network converges to the optimal policy. This strategy can satisfy the constraints. Under this premise, the system simultaneously optimizes decision-making across three dimensions: policy selection, device selection, and weight allocation. During the online deployment phase, the trained policy network is loaded into the deep reinforcement learning decision module of the central server. Each new federated learning round... At the beginning, the decision module receives the current state. The optimal action is directly output by performing a forward propagation calculation through the policy network. This method enables rapid online decision-making without requiring a complete training process. Its core advantage lies in learning the dynamic patterns of the environment and optimal decision-making strategies through offline training. During online deployment, only a simple forward propagation is needed to obtain the optimal decision, ensuring both decision quality and real-time performance.

[0093] Due to constraints Medium probability transfer function Due to the unknown nature and complexity of the problem, traditional optimization methods requiring precise system models fail. Therefore, this invention employs model-free deep reinforcement learning techniques to solve this optimization problem. A single-round composite reward function... This is the core of the entire optimization process. The formula for the reward function embodies an understanding and skillful balance of multiple conflicting performance objectives in the virtual participation compensation of federated learning. As a performance improvement measure, it is the only positive incentive term, directly related to the fundamental task of improving model performance, driving the agent to explore the participation strategies and device combinations that can bring the maximum accuracy gain; The negative penalty term corresponding to the training time. Incentivize agents to select devices and strategies with fast computation and good communication to shorten the training cycle; The number of virtual participants corresponds to a negative penalty. This forces intelligent agents to seriously consider the cost of virtual participation when making decisions, and to avoid over-reliance on virtual compensation while ignoring the real training of physical devices; The relay cost corresponds to a negative penalty term. Constrain the frequency of use of relay transmission strategies; The corresponding negative penalty term is for communication costs. Encourage intelligent agents to optimize communication efficiency; To ensure balanced rewards, agents are encouraged to guarantee long-term fairness in participation across all devices. Virtual quality rewards incentivize agents to choose virtual participation devices with high confidence levels. Weighting coefficients... , , , and The introduction of this feature allows for flexible adjustments to the system's emphasis on performance, efficiency, virtual participation costs, relay costs, and communication costs based on specific application scenarios. By maximizing the long-term expectation of this composite reward function, deep reinforcement learning agents can dynamically and intelligently balance multiple dimensions such as model accuracy, training efficiency, virtual participation quality, and system stability, thereby maximizing the system's long-term benefits.

[0094] Weighting coefficient , , , and Typical values ​​are 1.0, 0.3, 0.2, 0.1 and 0.1.

[0095] The optimization objective of this problem is to maximize a long-running condition determined by a discount factor. Weighted expected cumulative reward. Discount factor. The introduction of this approach allows the agent to consider not only the immediate reward of the current round but also the impact of the action on future states and potential rewards when evaluating the current action, thus learning a globally optimal rather than a locally optimal policy. The decision variables in this optimization problem are in each round. The execution 3D binary action vector At the same time, this decision must be subject to a series of strict operational constraints, which together define the boundary and intrinsic properties of the problem.

[0096] , , , and All are constraints, the first constraint To constrain the decision-making action structure, the action variables are clearly defined. A composite decision vector consisting of three sub-decisions, where , and These correspond to the participating strategy type, the device selection vector, and the weight allocation vector, respectively, demonstrating the extension of the traditional single-device selection problem into a joint optimization problem; the second constraint... To participate in strategy type constraints, the following is specified: These represent four strategies: waiting, physical device only, physical device plus virtual device, and relay transmission, enabling the agent to flexibly choose the training strategy based on the network state; the third constraint... To select vector constraints for the device, specify... Each element represents the degree to which a client is selected for virtual participation; this continuous design offers greater flexibility compared to traditional binary selection; constraints Assign vector constraints to weights, specifying Each element represents the weight allocation of a client in the aggregation, enabling the agent to dynamically adjust the influence of different clients; constraints To constrain the dynamic evolution of the system, state transitions are specified to follow a probability distribution. This constraint reveals the inherent randomness and dynamism of the federated learning environment, and the probability transition function Encapsulating all uncertainties such as network fluctuations, device load changes, and disconnection recovery, deep reinforcement learning agents can autonomously learn the underlying patterns of dynamic constraints through continuous interaction with the dynamic environment, and ultimately converge to an optimal strategy that can adapt to such uncertainties. ;

[0097] like Figure 3 As shown, in this optimization problem paradigm, the optimization objective is to find a strategy. To maximize the discount factor Weighted long-term expected cumulative reward, where each variable is defined as:

[0098] The state variable is This reflects the first The system state of the wheel contains a multi-state vector of all clients;

[0099] Action variable is , for the first The decision-making process of a wheel consists of three parts: the type of decision-making involved. ; Dimensional device selection vector Each element takes a value of [0,1], which indicates the degree to which the device is selected. Dimensional weight allocation vector Each element takes a value of [0,1] to represent the aggregate weight of the device;

[0100] Discount factor is Its value range is (0,1], and it is used to balance immediate rewards and long-term rewards;

[0101] The reward function is Defined as a multi-objective weighted sum, it drives the entire optimization process. This function is designed by this invention for specific scenarios of federated learning.

[0102] State transition Given the current action and state, this represents the probability distribution of the next state.

[0103] By solving this joint optimization problem, the deep reinforcement learning model can learn an optimal decision-making strategy that simultaneously optimizes the selection of participating policies, devices, and weights while satisfying all constraints. This will maximize the long-term benefits of the system.

[0104] Federated learning training is performed based on the decision results of the optimization steps. The central server distributes the current global model to all online clients. The system then waits for local training to complete. The key innovation here is that all online devices participate in training, rather than just selected devices. This ensures the system collects as much training data as possible, providing a sufficient data foundation for subsequent selective aggregation and digital twin updates. After collecting local model updates from all successfully responding physical clients, the system doesn't simply use all of them for aggregation. Instead, it employs a selection mechanism based on quality scoring and participation fairness. By maintaining a device participation counter and calculating a fairness adjustment factor, a subset of the collected physical updates is selected for aggregation, ensuring long-term participation fairness. This mechanism guarantees priority participation from high-quality devices while preventing low-quality devices from being excluded in the long term, achieving a balance between efficiency and fairness.

[0105] In this embodiment of the invention, S3 includes the following sub-steps:

[0106] S31. Distribute the global model to all online clients and wait for local training to complete;

[0107] S32. After completing local training, collect local model updates from all successfully responding physical clients and retain complete training records.

[0108] S33. Select a subset for aggregation from the collected local model updates of physical clients;

[0109] S34. Activate the digital twin model corresponding to the virtual device from the set of digital twin models and generate a compensatory model update;

[0110] S35. Perform weighted aggregation calculation on the selected local model update and compensatory model update to generate a new version of the global model.

[0111] Federated learning training is performed based on the decision results of the optimization step. Before this step begins, the communication management module first broadcasts the current global model to all online clients. This ensures that clients that have resumed operation can synchronize with the latest model state, avoiding training bias caused by inconsistent model versions. For any client... If it is in the first The wheel is offline and in the first... If a client returns to online status, the server sends it the latest global model. Update its local model upon receipt;

[0112] Then, all online clients perform local training using local data, with the learning rate set to [value missing]. Calculate local model update The central server collects local model updates from all successfully responding physical clients and maintains a complete training record. Instead of simply aggregating all physical updates, the system employs a selection mechanism based on quality scoring and participation fairness. This is achieved by maintaining a device participation counter and calculating a fairness adjustment factor. Select a subset of the collected physical updates for aggregation to ensure long-term participation fairness;

[0113] For the virtual participation device selected in the decision-making process Activate from the digital twin model set with each non-responsive client The only corresponding digital twin model .like Figure 4 As shown, each digital twin model Both are pre-learned LSTM neural networks tailored to their physical counterparts. The network employs a multi-layer architecture, including LSTM layers for capturing temporal gradient patterns, fully connected layers for feature transformation, Dropout layers to prevent overfitting, and an output layer for generating predicted gradients. When training the digital twin model, real gradient data is fully preserved and assigned a weight of 1.0; virtual gradient data is sampled and preserved proportionally and assigned a weight of 0.3. This differentiated weighting strategy ensures that real data dominates model training while utilizing virtual data to enhance generalization ability.

[0114] Activated digital twin model Based on client recent Historical gradient sequence of the wheel Predicting the next gradient Then, gradient updates are applied to generate compensatory model weights. The calculation formula is as follows:

[0115]

[0116] This process will use the latest global model of the current round. As input, by predicting gradients The model is updated, and the output is a high-quality compensatory model update that is highly similar to the lost client. This method can recover the client device information lost due to the disconnection to the greatest extent and reduce the statistical bias of the global model aggregation process.

[0117] In this embodiment of the invention, in S32, the compensatory model weights are updated by the compensatory model. The expression is:

[0118] ;

[0119] in, For the next gradient, For learning rate, For the first Global model weights for the wheel;

[0120] This formula simulates the local model update that should occur if the offline client were online by applying the predicted gradient to the current global model, thus achieving virtual compensation for the offline device.

[0121] In S33, a selection mechanism based on quality scoring and participation fairness is adopted. After collecting local model updates from all physical clients, a subset for aggregation is selected by maintaining device participation counters and calculating fairness adjustment factors.

[0122] In S35, when obtaining the current state of all clients... Then, the communication management module detects online and offline devices. For online devices... The system updates its state vector. The online status is marked as 1, and its reliability, performance capability, connection stability, data quality, and historical performance characteristics are updated based on historical training records; for offline devices... The system sets its state vector The online status is marked as 0, and its virtual participation candidate qualification and prediction confidence are obtained from the set of digital twin models. .

[0123] The training loss based on the digital twin model is used to calculate the prediction confidence score, which is then used to adjust the weights of virtual devices in the weighted aggregation. The expression is:

[0124]

[0125] in, For the training loss of the digital twin model, To obtain the minimum value, To obtain the maximum value;

[0126] The design logic of this formula is: when training loss... The smaller the value, the higher the prediction confidence. The closer the value is to 1, the higher the prediction quality; when the training loss... The larger the value, the closer the prediction confidence is to 0, indicating lower prediction quality. (This is achieved through...) The operation limits the loss value to The interval is then used, and the confidence level is obtained by subtracting the value from 1. Finally, the confidence level is determined by... Ensure the confidence level is non-negative. This confidence level is used to dynamically adjust the weight of virtual participating devices in the aggregation; predictions with high confidence receive greater weight, and predictions with low confidence receive less weight, thereby ensuring the quality of virtual compensation.

[0127] In S35, the hybrid aggregation step is used to generate a new version of the global model. The core aggregation algorithm is explicitly defined as a process of dynamically weighted averaging the physical model update and the virtual compensation update.

[0128] This invention seamlessly and fundamentally integrates physical model updates and virtual compensation updates through an innovative hybrid aggregation protocol. At the core of this protocol is a specially designed dynamic weighted aggregation algorithm to jointly construct a new generation of global models. New version global model The expression is:

[0129] ;

[0130] ;

[0131] ;

[0132] ;

[0133] in, This refers to any client that actually participated in the aggregation in this round. To successfully respond and upload the locally updated physical client, For clients detected as disconnected, For the first The set of all clients actually selected to participate in the aggregation. For physical device weights, For the aggregate weight of virtual devices, For virtual device weights, This is a subset of physical clients that actually participate in the aggregation after being filtered. For physical clients In the Local model updates uploaded in rounds, For the selected subset of disconnected clients that actually generate virtual compensation, For physical client collection, This is a collection of clients that have experienced disconnections.

[0134] The constituent elements and technical implications of this aggregation formula are as follows:

[0135] The core summation part: The formula explicitly sums the model update sets from two different sources, as shown in the first term within the parentheses. It comes from the collection of successfully corresponding physical clients. Real local model update The second item This comes from the collection of disconnected clients. Virtual compensation updates generated by its digital twin This approach achieves synergy between physical training and virtual compensation at the aggregation level.

[0136] Dynamic weighting mechanism: Physical device weights ,in For the dataset size, To rate the quality, Fairness adjustment factor; virtual device weight ,in To estimate the amount of data, To predict confidence levels, As a dynamic virtual weighting factor, This is a time-sensitive decay factor; this dynamic weighting mechanism ensures that high-quality equipment gains greater influence while guaranteeing fairness in participation.

[0137] Normalization term: the denominator of the formula It is a key normalization term, and its summation range is... This includes all clients selected for aggregation in this round, not just those that successfully responded. This ensures that even if the virtual participation ratio changes, the total weight remains stable, thus maintaining the stability of the global model update and avoiding learning rate instability caused by fluctuations in the number of participating devices.

[0138] The digital twin model is used for any non-responsive client Generate its compensatory model update The technical process employs an LSTM-based gradient prediction mechanism and uses a weighted training method to ensure prediction quality, specifically including:

[0139] Digital twin model structure: Each client Corresponding to an independent gradient prediction LSTM network The network employs a multi-layer architecture, including LSTM layers for capturing temporal gradient patterns, fully connected layers for feature transformation, Dropout layers for preventing overfitting, and output layers for generating predictive gradients.

[0140] Weighted training mechanism: When training a digital twin model, real gradient data is fully preserved, while virtual gradient data is weighted and sampled proportionally. This differentiated weighting strategy ensures that real data dominates model training, while virtual data is used to enhance generalization ability.

[0141] Gradient prediction process: client-based recent Historical gradient sequence of the wheel Through LSTM network Predicting the next gradient Then, gradient updates are applied to generate compensatory model weights. ;

[0142] Confidence Calculation and Application: Training Loss Based on Digital Twin Models The prediction confidence level is calculated and used to dynamically adjust the weight of virtual participating devices in the aggregation, with predictions of higher confidence receiving greater weight.

[0143] The physical client aggregation selection adopts a selection mechanism based on quality scores and participation fairness. After collecting model updates from all physical clients, a subset for aggregation is selected by maintaining device participation counters and calculating fairness adjustment factors to ensure long-term participation fairness.

[0144] In this embodiment of the invention, in S3, pre-training is performed before federated learning training. Specifically, within a preset number of rounds, the central server is used to forcibly select all registered clients to participate in training, including clients that simulate disconnection.

[0145] The method also includes a digital twin pre-training step. In the initial preset rounds of federated learning, the central server forces all registered clients to participate in training, including clients that will later simulate disconnection, to ensure that their corresponding digital twin models accumulate enough real gradient data for subsequent prediction training. This pre-training step lays the data foundation for the effective operation of the virtual participation compensation mechanism.

[0146] In this embodiment of the invention, in S4, after each round of training in federated learning, real gradients and performance data of all physical clients that have completed training are collected and marked as real data, and it is recorded whether they participate in this round of aggregation; predicted gradients are collected for virtual participating clients and marked as virtual data; when the training conditions are met, the digital twin model is retrained and the multidimensional state vector is updated.

[0147] The method also includes a digital twin update step. After each training round, real gradients and performance data are collected from all trained physical clients and labeled as real data, while recording whether they participated in the current round of aggregation; predicted gradients are collected from virtual participating clients and labeled as virtual data; digital twin model retraining is triggered when training conditions are met; and the state vector is updated to reflect the latest device state. This mechanism ensures that the digital twin models of all trained devices continuously improve, even if the device is not selected to participate in aggregation.

[0148] The training of deep reinforcement learning agents supports a configurable three-stage learning mechanism. When enabled, the basic policy is trained starting from a low-failure-rate environment, gradually transitioning to a medium-difficulty environment to optimize joint decision-making, and finally achieving a robust policy in a high-failure-rate environment. The progressive increase in difficulty avoids training convergence difficulties. When disabled, training is carried out directly in an environment with a fixed difficulty.

[0149] For all trained physical clients, collect real gradients and performance data and label them as real data with a weight of 1.0, while recording whether they participate in this round of aggregation; for virtual participating clients, collect predicted gradients and label them as virtual data with a weight of 0.3, and update their prediction confidence. When the historical data of the digital twin model meets the training conditions, model retraining is triggered. The retraining process adopts a weighted training strategy, fully retaining real data and sampling and retaining virtual data with a weight of 0.3, using the Adam optimizer with a learning rate of 0.001, and training until convergence or reaching the maximum number of iterations;

[0150] The central server will update the corresponding state vectors only for the selected clients in this round, based on their actual performance. This is to generate more accurate state information for the next round of decision-making. This includes dynamic features such as historical performance metrics, prediction confidence, online status, and participation statistics. This state update step plays a crucial feedback role, enabling the deep reinforcement learning decision model to continuously learn and fine-tune its online policy in a dynamically changing environment.

[0151] To ensure the digital twin model possesses sufficient predictive power, this invention also includes a digital twin pre-training step. In the initial preset rounds of federated learning, typically 10-15 rounds, the central server forcibly selects all registered clients to participate in training, including clients that will subsequently simulate disconnections. During this stage, the clients... Calculate its gradient after completing local training. The gradient is labeled as real data and stored in the digital twin model. The training set. At the end of the pre-training phase, the digital twin model... Training is performed based on accumulated real gradient sequences to learn the client. The gradient evolution pattern. This pre-training step lays the data foundation for the effective operation of the virtual participation compensation mechanism, ensuring that the digital twin model can generate high-quality virtual compensation updates.

[0152] To improve the training efficiency of deep reinforcement learning agents, this invention supports a configurable three-stage learning mechanism. When enabled, the agent learns according to a progressively increasing difficulty strategy during a total training period of 800,000 time steps. The first stage (0-150,000 time steps) trains basic policies in a low-failure-rate environment, learning basic device selection and virtual participation decisions. The second stage (150,000-300,000 time steps) gradually transitions to optimizing joint decision-making in a medium-difficulty environment, strengthening the collaborative optimization capabilities in the three-dimensional action space. The third stage (300,000-800,000 time steps) implements robust policies in a high-failure-rate environment, ensuring the agent maintains performance even under extreme disconnection scenarios. Progressive difficulty increases prevent training convergence difficulties. When disabled, training is performed directly in an environment with a fixed difficulty level.

[0153] Based on the above methods, this invention also proposes a federated learning participation compensation system based on reinforcement learning and digital twins, including a central server and several client devices.

[0154] The central server includes:

[0155] State assessment and prediction module: used to maintain multi-dimensional state vectors and manage digital twin models;

[0156] Deep reinforcement learning decision module: used for weight allocation for optimal policy and weighted aggregation;

[0157] Virtual Participation Compensation Module: Used to maintain an LSTM gradient prediction model for each client, employing a weighted training mechanism;

[0158] Enhanced aggregation module: used for weighted aggregation calculations;

[0159] Communication Management Module: Used to manage client communication and perform model distribution, collection, and synchronization;

[0160] Status Management and Fairness Guarantee Module: Used to track device status and participation count, and calculate the fairness adjustment factor for the fairness selection mechanism.

[0161] The system comprises a central server and multiple client devices. The central server is specially configured to integrate the following tightly coupled and collaborative modules to execute the methods described above: A state assessment and prediction module maintains a multi-dimensional state vector. It manages a set of digital twin models, providing virtual participation candidate eligibility assessments for each client. The deep reinforcement learning decision-making module executes joint optimization decisions, solving the joint optimization problem based on multi-dimensional state vectors and outputting participation strategies. Equipment selection and weight allocation It also supports configurable course learning mechanisms. The virtual participation compensation module is available for each client. Maintaining the LSTM gradient prediction model A weighted training mechanism is used to generate high-quality compensatory model updates for non-responsive clients. The enhanced aggregation module performs dynamic weighted hybrid aggregation, taking into account quality comprehensively. Fairness adjustment factor Confidence level Maturity and timeliness To ensure the stability of the aggregation process and the global model. The quality of the model is ensured. The communication management module manages client communication, performs model distribution, collection, and synchronization. Specifically, this module performs client synchronization waiting before training begins, ensuring all registered clients are ready before starting training; and broadcasts the global model to all online clients before each training round. This enables model synchronization of the recovery equipment. The state management and fairness assurance module tracks equipment status and participation counts, and calculates the fairness adjustment factor. This ensures long-term fairness in participation. These modules interact with each other through well-defined interfaces, forming a complete "prediction-optimization-compensation-synchronization" closed-loop system, thereby transforming a standard federated learning server into a system with advanced intelligence and robustness.

[0162] The deep reinforcement learning decision module solves the joint optimization problem based on multi-dimensional state vectors and supports a configurable course learning mechanism; the virtual participation compensation module adopts LSTM gradient prediction and weighted training mechanism; the reinforcement aggregation module performs dynamic weight aggregation; and the communication management module performs client synchronization waiting and model broadcasting.

[0163] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A federated learning participation compensation method based on reinforcement learning and digital twins, characterized in that, Includes the following steps: S1. Use the central server to assess the current status of several client devices in the client network and determine the status information; S2. Based on the state information, construct a joint optimization problem and solve it to obtain the optimal strategy; S3. Based on the optimal strategy, perform federated learning training; S4. Synchronize after federated learning training.

2. The federated learning participation compensation method based on reinforcement learning and digital twins according to claim 1, characterized in that, In S1, the state information adopts a multi-dimensional state vector, including the device's online status, reliability, performance capability, connection stability, data quality, historical performance, digital twin prediction confidence, and virtual participation weight.

3. The federated learning participation compensation method based on reinforcement learning and digital twins according to claim 1, characterized in that, In S2, the expression for solving the joint optimization problem is: ； ； ； ； ； ； ； in, The strategies followed by deep reinforcement learning agents. As a discount factor, For state variables, As an action variable, For the reward function, For training round index, The total number of training rounds, This is the first constraint condition. This is the second constraint. This is the third constraint. This is the fourth constraint. This is the fifth constraint. For the type of decision-making participation, for Dimensional device selection vector, for Dimensional weight allocation vector, For the first The system state of the wheel, For state transition, For mathematical expectation, For performance improvement amount, Training takes time, For virtual participation numbers, For relay costs, For communication costs, To participate in the balanced rewards, For virtual quality rewards, The weighting coefficient for the performance improvement. The weight coefficients represent the training time. The weighting coefficient for the number of virtual participants. This is the weighting coefficient for relay costs. This is the weighting coefficient for communication costs.

4. The federated learning participation compensation method based on reinforcement learning and digital twins according to claim 1, characterized in that, S3 includes the following sub-steps: S31. Distribute the global model to all online clients and wait for local training to complete; S32. After completing local training, collect local model updates from all successfully responding physical clients and retain complete training records. S33. Select a subset for aggregation from the collected local model updates of physical clients; S34. Activate the digital twin model corresponding to the virtual device from the set of digital twin models and generate a compensatory model update; S35. Perform weighted aggregation calculation on the selected local model update and compensatory model update to generate a new version of the global model.

5. The federated learning participation compensation method based on reinforcement learning and digital twins according to claim 4, characterized in that, In step S32, the compensatory model weights are updated by the compensatory model. The expression is: ； in, For the next gradient, For learning rate, For the first Global model weights for the wheel; In S33, a selection mechanism based on quality scoring and participation fairness is adopted. After collecting local model updates from all physical clients, a subset for aggregation is selected by maintaining device participation counters and calculating fairness adjustment factors. In step S35, the prediction confidence is calculated based on the training loss of the digital twin model. This confidence is used to adjust the weights of virtual devices in the weighted aggregation. The expression is: ； in, For the training loss of the digital twin model, To obtain the minimum value, To obtain the maximum value; In S35, the new version of the global model The expression is: ； ； ； ； in, This refers to any client that actually participated in the aggregation in this round. To successfully respond and upload the locally updated physical client, For clients detected as disconnected, For the first The set of all clients actually selected to participate in the aggregation. For physical device weights, For the aggregate weight of virtual devices, For virtual device weights, This is a subset of physical clients that actually participate in the aggregation after being filtered. For physical clients In the Local model updates uploaded in rounds, For the selected subset of disconnected clients that actually generate virtual compensation, For physical client collection, This is a collection of clients that have experienced disconnections.

6. The federated learning participation compensation method based on reinforcement learning and digital twins according to claim 1, characterized in that, In S3, pre-training is performed before federated learning training. Specifically, within a preset number of rounds, the central server forces all registered clients to participate in training, including clients that simulate disconnection.

7. The federated learning participation compensation method based on reinforcement learning and digital twins according to claim 1, characterized in that, In S4, after each round of training in federated learning, real gradients and performance data of all trained physical clients are collected, labeled as real data, and it is recorded whether they participate in this round of aggregation. Collect predicted gradients from virtual participating clients and label them as virtual data; trigger retraining of the digital twin model when training conditions are met and update the multidimensional state vector.

8. A federated learning participation compensation system based on reinforcement learning and digital twins, characterized in that, It includes a central server and several client devices; The central server includes: State assessment and prediction module: used to maintain multi-dimensional state vectors and manage digital twin models; Deep reinforcement learning decision module: used for weight allocation for optimal policy and weighted aggregation; Virtual Participation Compensation Module: Used to maintain an LSTM gradient prediction model for each client, employing a weighted training mechanism; Enhanced aggregation module: used for weighted aggregation calculations; Communication Management Module: Used to manage client communication and perform model distribution, collection, and synchronization; Status Management and Fairness Guarantee Module: Used to track device status and participation count, and calculate the fairness adjustment factor for the fairness selection mechanism.

Images