A distributed federated reinforcement learning method based on digital twins

Abstract

The present invention relates to a distributed federated reinforcement learning method based on digital twins, which belongs to the field of machine learning. First, physical devices in the real world are mapped to a twin environment, and then the global parameters of the twin of each device are initialized. Then, the global model parameters are used to run a deep reinforcement learning algorithm in the twin. After that, the trained local deep reinforcement learning model is subjected to credibility weighting processing based on the twin mapping deviation and the training deviation. The global model aggregation node is selected by measuring the credibility of the locally trained deep reinforcement learning model, and then the global model is aggregated on the selected nodes. Finally, the global model parameters are sent to each local twin, and the next round of training is started until the global model converges. The present invention can improve the training quality of the model while reducing redundant communications between the device end and the central server, reducing communication overhead and improving the training speed and accuracy of the global model.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning and relates to a distributed federated reinforcement learning method based on digital twins. Background Art

[0002] With the proliferation of smartphones, tablets, and other mobile devices, more and more data is being generated and stored locally on these devices. Traditional machine learning and deep learning model technologies require large-scale and diverse data to train complex models. Because uploading large amounts of local data poses the risk of data privacy breaches, most users are reluctant to share information to build shared models. Therefore, traditional centralized machine learning training methods face limitations in data collection and processing. Federated learning, by integrating data distributed across devices, allows model training on broader and more diverse datasets, thereby improving model performance and generalization.

[0003] Traditional federated learning algorithms send trained model parameters to a central server during model aggregation. The central server then simply averages the received parameters and sends the averaged parameters as the global model to all participants. This aggregation method cannot accurately measure the contribution of the nodes participating in the training to the global model and is heavily dependent on the central server. Therefore, a method is urgently needed to solve the problems that traditional federated learning has in global model aggregation. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a distributed federated reinforcement learning method based on digital twins, which effectively reduces the dependence on the central server in federated learning and reflects the contribution of each training node to the global model during the model aggregation process.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A distributed federated reinforcement learning method based on digital twins. This method first maps physical devices in the real world into a twin environment, and then initializes the global parameters W for each device's twin. g , and then use the global model parameters to run the deep reinforcement learning algorithm in the twin body. After that, the trained local deep reinforcement learning model is weighted based on the credibility of the twin mapping deviation and training deviation. The global model aggregation node is selected by measuring the credibility of the locally trained deep reinforcement learning model. The global model is then aggregated on the selected nodes. Finally, the global model parameters are sent to each local twin body to start the next round of training until the global model converges.

[0007] Preferably, the method specifically comprises the following steps:

[0008] S1: Map the physical device in the real world to the twin to obtain the high-fidelity operating status of the physical device in the real world;

[0009] S2: In the twin environment, initialize the global parameters W locally for each twin participating in the federated learning training g ;

[0010] S3: In the twin environment, the twin itself is based on the global parameter W g Conduct local deep reinforcement learning model training and perform credibility weighting on the trained local deep reinforcement learning model based on twin mapping deviation and training deviation;

[0011] S4: In the twin environment, by measuring the credibility calculated in S3, the node for global model aggregation is selected, and the locally trained deep reinforcement learning model parameters are forwarded to the selected aggregation node for credibility-based global model aggregation.

[0012] S5: Determine whether the current global model has converged. If not, repeat the above operation. Otherwise, synchronize the trained global model to the device in the physical world.

[0013] Preferably, in step S1, various sensors are used to collect the operating status and data of physical equipment in the real world, and the data are converted into digital models through physical modeling and simulation technology.

[0014] Preferably, in step S2, the same model parameters are initialized for each twin participating in the federated training.

[0015] Preferably, in step S3, the initialized model parameters are used to perform local deep reinforcement learning model training, and the credibility of the locally trained model parameters is calculated.

[0016] Preferably, in step S3, the credibility of the model is calculated by calculating the deviation between the locally trained model parameters and the global model parameters and the mapping deviation between the physical device and its twin at the current moment. The model training deviation can be expressed as: The mapping deviation of the digital twin can be expressed as: Δf i (k) / f i (k), the credibility of the training model can be expressed as:

[0017]

[0018] Preferably, in step S4, the model credibility of each training node is measured, and the training node with the highest model credibility is selected as the aggregation node, and global model aggregation based on credibility is performed.

[0019] Preferably, in step S4, the global model is aggregated using a model aggregation formula based on credibility, and the aggregation formula can be expressed as:

[0020]

[0021] Where N is the number of devices participating in the federated learning training, c i (k) is the credibility of the model trained by device i during the kth iteration, are the local model parameters trained by device i during the kth iteration.

[0022] Preferably, in step S5, the convergence of the aggregated global model is judged. If the current global model has not yet converged, the global model of this round of aggregation is synchronized from the aggregation node to each participant for the next round of training; otherwise, the twin training model is synchronized to the physical device.

[0023] The beneficial effects of this invention are that, in scenarios with massive IoT devices, distributed machine learning tasks can be completed faster and better, significantly improving the convergence speed and accuracy of federated models while avoiding the risk of privacy leakage between devices caused by centralized data training. Furthermore, because global model parameters are aggregated using a node-based credibility-weighted model aggregation approach, reliance on central servers and redundant communication between devices can be reduced, while also improving model training quality.

[0024] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0026] Figure 1 This is a flow chart of the distributed federated reinforcement learning method based on digital twins described in the present invention;

[0027] Figure 2 Schematic diagram of the distributed federated reinforcement learning method based on digital twins. DETAILED DESCRIPTION

[0028] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0029] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0030] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0031] See also Figures 1 and 2 , Figure 1 A flow chart of a distributed federated reinforcement learning method based on digital twins for collaborative training of multiple autonomous vehicles provided by the present invention. Figure 2 This is a schematic diagram of a distributed federated reinforcement learning method based on digital twins. In this method, real-world autonomous vehicles are first mapped into a twin environment. A central server initializes global model parameters on the twins of the participating federated learning vehicles and then distributes them to each device node. Each device uses a local dataset and initialized model parameters to train the task, and then uses stochastic gradient descent to calculate the gradient parameters and complete the update. Specifically:

[0032] Assume that there are N autonomous vehicles in the real world. Each autonomous vehicle forms a twin through mapping. The twin runs the Deep Deterministic Policy Gradient (DDPG) algorithm locally. In the kth round of local iteration, the global model parameter trained by the vehicle group is w g (k-1), the model parameters of each twin based on DDPG training are The vehicle group obtains the global model w by weighted aggregation of the local training model of each twin g (k), the specific steps are as follows:

[0033] Step 1) Map the real-world autonomous vehicle into the digital twin to obtain the high-fidelity operating status of the autonomous vehicle. The digital twin of the autonomous vehicle with limited computing resources can be represented as:

[0034] DT i (k)=(m i (w k ),f i (k),s i (k),z i (k),o i (k))

[0035] where w k is the current training parameter of the vehicle, m i (k) represents the model training state of the vehicle, f i (k) represents the computing power of the vehicle, s i (k) represents the running state of the vehicle, z i (k) represents the vehicle's mission time, o i (k) indicates whether the vehicle has completed the task. Since it is difficult to accurately obtain the computing power of the vehicle when performing twin mapping on the vehicle, there is a deviation between the real value and the mapped value of the CPU frequency. In order to calibrate the mapping deviation, we introduce the CPU frequency deviation Δf i (k) is used to measure the error in the twin mapping calculation ability. After correcting the mapping deviation, the twin model of vehicle i can be expressed as

[0036] DT i (k)=(m i (k),f i (k)+Δf i (k),s i (k),z i (k),o i (k))

[0037] Step 2) Initialize the same global model parameters for each autonomous vehicle twin Assume that all vehicle twin nodes are currently in the kth round of iteration, and the total number of iterations is M.

[0038] Step 3) For each autonomous driving vehicle node, it runs the DDPG algorithm locally. Let the model parameters trained by vehicle i in the kth round be: The parameters of the global model at this time are: w g (k), in order to measure the credibility of the vehicle node training model, we need to consider the local model training quality and the quality of the twin mapping. The training quality of the local model can be expressed by the current model parameters w g (k) and the current global parameter w g (k) is calculated by the deviation between the two. The calculation formula is:

[0039]

[0040] The twin mapping quality can be mapped to the accuracy Δf using computing resources i (k) / f i (k) is used to measure the quality of vehicle node training, so the credibility model is performed on the quality of vehicle node training. The credibility of the node can be expressed as:

[0041]

[0042] Step 4) In the twin environment, the vehicle group selects the aggregation node of the global model by measuring the credibility. The rule for selecting the aggregation node can be expressed as:

[0043]

[0044] Step 5) After selecting the aggregation node, each twin uploads its training parameters to the aggregation node, and then performs global model aggregation based on credibility. The aggregation formula is as follows:

[0045]

[0046] Step 6) For the aggregated global model w g (k) Convergence judgment is performed. If the model has not converged, the global model w g (k) is sent to the twin nodes participating in the federated training, and then the k+1 round of training is performed. Otherwise, the trained model is synchronized to the physical vehicle in the real world.

[0047] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A distributed federated reinforcement learning method based on digital twins, characterized by: The method comprises the following steps: S1: Map the physical devices in the real world to the twins to obtain the high-fidelity operating status of the physical devices in the real world; obtain the model training credibility c of each training node by calculating the deviation between the local deep reinforcement learning model parameters and the current global model parameters and the CPU computing power mapping deviation between the current physical entity and its twin. i (k), the node’s credibility formula is expressed as: in Refers to the degree of deviation of the training model during the kth round of iteration, Δf i (k) refers to the difference between the CPU computing frequency mapped by physical device i and the actual computing frequency during the kth iteration, f i (k) refers to the calculation frequency of the real physical device i during the kth iteration, Δf i (k) / f i (k) refers to the mapping deviation; S2: In the twin environment, initialize the global parameters W locally for each twin participating in the federated learning training g ; S3: In the twin environment, the twin itself is based on the global parameter W g Conduct local deep reinforcement learning model training and perform credibility weighting on the trained local deep reinforcement learning model based on twin mapping deviation and model training deviation; calculate the local deep reinforcement learning model parameters of each vehicle twin trained in the kth round With the current global model parameter w g The deviation between (k-1) is used to calculate the model update credibility. The model deviation formula is expressed as: in is the training model parameter of the twin of the i-th physical device during the k-th round of machine learning iteration, w g (k-1) is the global model parameter used by the physical device twin during the kth iteration; S4: In the twin environment, by measuring the credibility calculated in S3, dynamically select the node for global model aggregation, and forward the locally trained deep reinforcement learning model parameters to the selected aggregation node to perform credibility-based global model aggregation; measure the credibility of the training model of each twin c i (k), select the aggregation node, and the node selection formula is expressed as: Where N is the number of devices participating in the federated learning training, c i is the model training credibility of the i-th device, c N is the model training credibility of the Nth device; S5: Determine whether the current global model has converged. If not, repeat steps S3 to S4. Otherwise, synchronize the trained global model to the device in the physical world. Each twin participating in the federated training uploads its own training parameters to the aggregation node and performs global model parameter aggregation based on credibility. The global model parameter aggregation formula based on credibility is expressed as:

2. The distributed federated reinforcement learning method based on digital twins according to claim 1, characterized in that: In S1, sensors of physical devices in the real world, including lidar and laser ranging sensors, and modeling technologies, including machine learning technology, big data and cloud computing technology, are used to map the operating status, behavior and characteristics of physical devices into the twin.

3. The distributed federated reinforcement learning method based on digital twins according to claim 2, characterized in that: In S2, in the twin environment, the same parameter initialization operation is performed on each twin participating in the federated learning training.

4. The distributed federated reinforcement learning method based on digital twins according to claim 1, characterized in that: In S5, it is determined whether the global model has converged. If not, the aggregated global model is sent to each training node to continue the next round of training. Otherwise, the trained model is synchronized to the physical entity.

Images