Model aggregation method and related equipment for reinforcement learning based on federated learning

Abstract

The embodiment of the present application discloses a model aggregation method for reinforcement learning based on federated learning and related equipment, which is used to aggregate the local model obtained after reinforcement learning of each robot under the framework of federated learning to improve training efficiency. The method of the embodiment of the present application includes: in the Mth aggregation cycle, receiving the target local model and training contribution sent by each robot among multiple robots participating in federated learning; wherein, the target local model is obtained by the robot training the aggregation model sent by the aggregation server in the Mth aggregation cycle, when M is 1, the aggregation model is the preset initial aggregation model, and when M is greater than 1, the aggregation model is the target aggregation model obtained in the M-1th aggregation cycle; the training contribution is used to represent the resources consumed by the robot to train the target local model; based on the aggregation of the training contribution of each robot and the target local models sent by multiple robots, the target aggregation model of the Mth aggregation cycle is obtained.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of machine learning, and in particular to a model aggregation method and related equipment for reinforcement learning based on federated learning. Background Art

[0002] In the face of complex environments, bad weather, and unknown maps, using intelligent robots to replace humans to go to high-risk areas to complete tasks such as reconnaissance and exploration, emergency search and rescue, etc. is an important direction for the development of the robotics field. Among them, the robot's autonomous navigation technology is the basis for the robot to successfully complete complex tasks. Compared with traditional control methods, reinforcement learning can train autonomous navigation models, thereby providing a new solution for improving the robot's autonomous navigation capabilities.

[0003] Existing deep reinforcement learning algorithms are mainly applied to a single robot. Model training requires a large number of experience samples, and the collection of experience samples and the use of experience samples for model training are completely completed by a single robot, resulting in extremely low model training efficiency. Summary of the invention

[0004] The embodiments of the present application provide a model aggregation method for reinforcement learning based on federated learning and related equipment, which are used for a deep reinforcement learning algorithm based on multi-robot reinforcement learning to improve the training efficiency of deep reinforcement learning.

[0005] A first aspect of an embodiment of the present application provides a model aggregation method for reinforcement learning based on federated learning, which is applied to an aggregation server and includes:

[0006] In the Mth aggregation cycle, a target local model and a training contribution sent by each robot among the multiple robots participating in the federated learning are received; wherein the target local model and the training contribution are sent by the robot in response to the timed aggregation request; the target local model is obtained by the robot training the aggregation model sent by the aggregation server in the Mth aggregation cycle, when M is 1, the aggregation model is a preset initial aggregation model, and when M is greater than 1, the aggregation model is the target aggregation model obtained in the M-1th aggregation cycle; the training contribution is used to represent the resources consumed by the robot in training the target local model;

[0007] Based on the training contribution aggregation of each robot and the target local models sent by the multiple robots, a target aggregation model of the Mth aggregation cycle is obtained.

[0008] In a specific implementation, before receiving the target local model and training contribution sent by each robot among the multiple robots participating in the federated learning, the method includes:

[0009] If the duration from the moment when the aggregation model is sent to each robot in the Mth aggregation cycle is greater than or equal to the preset duration, a timed aggregation request is sent to each robot so that each robot sends the target local model trained in the Mth aggregation cycle after receiving the timed aggregation request.

[0010] In a specific implementation, after sending the timing aggregation request to each of the robots, the method further includes:

[0011] Receiving a local training status of the Mth aggregation cycle sent by each of the robots, wherein the local training status includes a training amplitude of the aggregation model trained by the robot in the Mth aggregation cycle;

[0012] Based on the local training status of the Mth aggregation cycle sent by each of the robots, determining whether the M+1th aggregation cycle meets the preset frequency modulation condition;

[0013] If the M+1th aggregation period meets the preset frequency modulation condition, the preset duration is adjusted.

[0014] In a specific implementation, aggregating the target local models sent by the multiple robots based on the training contribution of each robot to obtain the target aggregate model of the Mth aggregation period includes:

[0015] Based on the formula Calculate the model parameters of the target aggregation model for the Mth aggregation cycle, where θ f is the model parameter of the target aggregation model in the Mth aggregation period, θ n is the model parameter of the target local model of each robot in the Mth aggregation period, v n The training contribution of each robot to train the corresponding target local model for the Mth aggregation period;

[0016] The model parameters θ of the target aggregation model f A target aggregation model for the Mth aggregation cycle is determined.

[0017] A second aspect of the embodiment of the present application provides a model aggregation method for reinforcement learning based on federated learning, which is applied to a robot, including:

[0018] In the Mth aggregation cycle, the aggregation model sent by the aggregation server is trained; when M is 1, the aggregation model is a preset initial aggregation model, and when M is greater than 1, the aggregation model is a target aggregation model obtained in the M-1th aggregation cycle;

[0019] In response to the timed aggregation request of the Mth aggregation cycle, determine a target local model obtained by training the aggregation model and determine a training contribution of the robot for training the target local model; the training contribution is used to represent the resources consumed by the robot for training the target local model;

[0020] The target local model and the training contribution are sent to an aggregation server, so that the aggregation server determines a target aggregation model for an Mth aggregation cycle according to the target local model and the training contribution.

[0021] In a specific implementation, the method further includes:

[0022] In response to a request to obtain the training status of the Mth aggregation cycle, the local training status of the Mth aggregation cycle is sent to the aggregation server, so that the aggregation server determines whether the M+1th aggregation cycle meets the preset frequency modulation conditions based on the local training status; the local training status includes the training amplitude of the aggregation model trained by the robot in the Mth aggregation cycle.

[0023] A third aspect of an embodiment of the present application provides an aggregation server, including:

[0024] A receiving unit, configured to receive, in the Mth aggregation cycle, a target local model and a training contribution sent by each robot among a plurality of robots participating in federated learning; wherein the target local model and the training contribution are sent by the robot in response to a timed aggregation request; the target local model is obtained by the robot training the aggregation model sent by the aggregation server in the Mth aggregation cycle, when M is 1, the aggregation model is a preset initial aggregation model, and when M is greater than 1, the aggregation model is a target aggregation model obtained in the M-1th aggregation cycle; the training contribution is used to represent the resources consumed by the robot in training the target local model;

[0025] A computing unit is used to obtain a target aggregation model of the Mth aggregation cycle based on the training contribution aggregation of each robot and the target local models sent by the multiple robots.

[0026] In a specific implementation, the aggregation server further includes: a sending unit;

[0027] The sending unit is used to send a timed aggregation request to each robot if the duration from the moment when the aggregation model is sent to each robot in the Mth aggregation cycle is greater than or equal to a preset duration, so that each robot sends the target local model trained in the Mth aggregation cycle after receiving the timed aggregation request.

[0028] In a specific implementation, the receiving unit is further used to receive the local training status of the Mth aggregation cycle sent by each of the robots, wherein the local training status includes the training amplitude of the aggregation model trained by the robot in the Mth aggregation cycle;

[0029] Based on the local training status of the Mth aggregation cycle sent by each of the robots, determining whether the M+1th aggregation cycle meets the preset frequency modulation condition;

[0030] If the M+1th aggregation period meets the preset frequency modulation condition, the preset duration is adjusted.

[0031] In a specific implementation, the calculation unit is specifically used based on the formula Calculate the model parameters of the target aggregation model for the Mth aggregation cycle, where θ f is the model parameter of the target aggregation model in the Mth aggregation period, θ n is the model parameter of the target local model of each robot in the Mth aggregation period, v n The training contribution of each robot to train the corresponding target local model for the Mth aggregation period;

[0032] The model parameters θ of the target aggregation model f A target aggregation model for the Mth aggregation cycle is determined.

[0033] A fourth aspect of the embodiments of the present application provides a robot, comprising:

[0034] A training unit, used to train the aggregation model sent by the aggregation server in the Mth aggregation cycle; when M is 1, the aggregation model is a preset initial aggregation model, and when M is greater than 1, the aggregation model is a target aggregation model obtained in the M-1th aggregation cycle;

[0035] A determination unit, configured to respond to the timing aggregation request of the Mth aggregation period, determine a target local model obtained by training the aggregation model and determine a training contribution of the robot for training the target local model; the training contribution is used to represent resources consumed by the robot for training the target local model;

[0036] A sending unit is used to send a target local model and a training contribution to an aggregation server, so that the aggregation server determines a target aggregation model for an Mth aggregation cycle according to the target local model and the training contribution.

[0037] In a specific implementation, the sending unit is also used to respond to a request to obtain the training status of the Mth aggregation cycle, and send the local training status of the Mth aggregation cycle to the aggregation server, so that the aggregation server determines whether the M+1th aggregation cycle meets the preset frequency modulation condition based on the local training status; the local training status includes the training amplitude of the aggregation model trained by the robot in the Mth aggregation cycle.

[0038] A fifth aspect of an embodiment of the present application provides a data processing device, including:

[0039] CPU, memory and input / output interface;

[0040] The memory is a short-term storage memory or a persistent storage memory;

[0041] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method described in the first aspect or the second aspect.

[0042] A sixth aspect of the embodiments of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect or the second aspect.

[0043] A seventh aspect of an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the method described in the first aspect or the second aspect.

[0044] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: a model aggregation method based on multi-robot reinforcement learning is provided, and multiple robots are sampled and trained simultaneously, thereby improving the training efficiency of the reinforcement learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flow chart of a model aggregation method for reinforcement learning based on federated learning disclosed in an embodiment of the present application;

[0046] Figure 2 Another schematic diagram of the flow of the model aggregation method of reinforcement learning based on federated learning disclosed in the embodiment of the present application;

[0047] Figure 3 A schematic diagram of a flow chart of navigation control of a robot disclosed in an embodiment of the present application;

[0048] Figure 4 An example diagram of a model aggregation method for reinforcement learning based on federated learning disclosed in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of the structure of the aggregation server disclosed in the embodiment of the present application;

[0050] Figure 6 A schematic diagram of the structure of the robot disclosed in the embodiment of the present application;

[0051] Figure 7 A schematic diagram of the structure of a data processing device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] The embodiments of the present application provide a model aggregation method for reinforcement learning based on federated learning and related equipment, which are used for a deep reinforcement learning algorithm based on multi-robot reinforcement learning to improve the training efficiency of deep reinforcement learning.

[0054] See also Figure 1 , an asynchronous federated reinforcement learning model training method for multi-robot collaborative navigation provided in an embodiment of the present application is as follows:

[0055] First, the aggregation server sends the aggregation model to each robot participating in the model training (if it is the first aggregation cycle, the aggregation model is the preset initial aggregation model; if it is another aggregation cycle after the first aggregation cycle, the aggregation model is the target aggregation model of the corresponding previous aggregation cycle), then each robot trains the aggregation model according to its own local data set, and then each robot sends the target local model obtained after training the aggregation model to the aggregation server regularly or irregularly, and finally the aggregation server aggregates the target local model sent by each robot to obtain the target aggregation model, and an aggregation cycle is completed. Before the start of the Nth aggregation cycle, the aggregation server also needs to determine whether the target aggregation model obtained in the N-1th aggregation cycle converges. If not, the target aggregation model obtained in the N-1th aggregation cycle is sent to each robot participating in the training to start the training of the Nth aggregation cycle, that is, the aggregation model used for training by the robot in the next aggregation cycle is the target aggregation model obtained in the previous aggregation cycle, where N is greater than or equal to 2. This cycle is repeated until the target aggregation model of a certain aggregation cycle converges, and the training ends, that is, the cycle ends.

[0056] In some specific embodiments, each robot has a different, independent training environment. Each robot collects samples in its own training environment as a local data set and performs local model training and updates based on a reinforcement learning algorithm. At the same time, the aggregation server collects the target local models of each robot within a given preset time, and performs weighted aggregation according to the different local iteration times of each robot to obtain a global target aggregation model. Then, the aggregation server distributes the target aggregation model to each robot and notifies each robot to continue local training based on the target aggregation model. Finally, each robot loads the target aggregation model and continues a new round of local training. Until convergence, the cycle ends.

[0057] In this embodiment, each robot performs model training in the local environment in parallel, and only communicates with the aggregation server and transmits the local model instead of the local data set. The aggregation server aggregates the target local models obtained by training the aggregation model in heterogeneous environments and their own heterogeneous properties, and obtains the target aggregation model. At the same time, through multi-robot federated reinforcement learning, each robot obtains the training knowledge of other robots without sharing local data, optimizes the local model, and avoids falling into the local optimum. While ensuring the privacy of the local training data sets of all parties, the burden of communication is reduced, and finally a more robust aggregation model is trained.

[0058] See also Figure 2 , the embodiment of the present application provides a model aggregation method for reinforcement learning, including:

[0059] 201. The aggregation server sends the aggregation model to the robot.

[0060] At the beginning of the Mth aggregation cycle, the aggregation server sends the aggregation model to the robots participating in the training, so that the robots train the aggregation model according to the local data set.

[0061] When M is 1, the aggregation model is a preset initial aggregation model, and when M is greater than 1, the aggregation model is a target aggregation model obtained in the M-1th aggregation cycle. It can be known that the preset initial aggregation model can be any model determined according to the robot local data set and the model training purpose, and this embodiment does not limit it.

[0062] 202. The robot trains the aggregation model.

[0063] When the robot receives the aggregation model sent by the aggregation server in step 201, it can train the aggregation model according to the samples in the local data set.

[0064] See also Figure 3,In some specific embodiments, the robot can be locally trained in the following ,way.

[0065] Specifically, the robot moves in its own training area to collect samples and train the aggregation model. The data that the collected samples need to contain are pre-set according to the different tasks and robot hardware, that is, the differences in robot task design and structural design make the data that the samples need to collect different. It can be known that the data contained in the sample includes but is not limited to the position of the robot itself, the position of the target point (the position information is determined by GPS), the robot's sonar or radar data, and visual image data. This embodiment does not specifically limit the data contained in the sample.

[0066] In addition, the collected samples are the data in the local data set. The robot will collect training samples (s, a, r, s', done) and put them into the local data set (it can also be a playback cache).

[0067] Specifically, the samples may be collected at any time before, after or during robot training, and this embodiment does not limit this. For example, the robot may first move randomly to collect samples, and when a certain number of samples is reached, the collected samples are used for training.

[0068] In some specific embodiments, the data included in the sample may be the current state s of the robot, the execution action a determined by the robot based on the strategy (ε greedy strategy), the reward value r obtained by the robot after executing the action, the next state s' of the robot after executing the action, and whether to end the training of the current cycle and remake the robot done. Among them, the random value ε of the ε greedy strategy will gradually decrease as the training progresses. In addition, remaking the robot is equivalent to remaking actions in reinforcement learning simulation, which is related to the task setting of reinforcement learning. For example, in the navigation and obstacle avoidance task, if the robot encounters an obstacle, falls or is damaged during the task, the robot needs to return to the starting point and repair the original posture, so that the robot can start a new round of training again.

[0069] When the robot starts training, it will retrieve some data from the local data set for gradient descent optimization. Specifically, it can be implemented through the ddpg algorithm, which is divided into two networks, the actor network and the critic network. Among them, the actor network determines the individual's action and outputs behavior a. The critic network is used to evaluate the current action and output Q(s,a), where Q(s,a) is the score of the robot performing action a in state s (rationality judgment). At the same time, in order to train the network more effectively, the target network is introduced. Before training, it is necessary to initialize the robot's deep network parameters Q, μ, Q', μ' and model training hyperparameters, where Q represents the critic network of ddpg; μ represents the actor network of ddpg; Q' represents the target critic network of ddpg; μ' represents the target actor network of ddpg, Q', μ' are the corresponding target networks, and the model training hyperparameters include learning rate η, discount factor γ, greedy strategy factor ε, total number of iterations T, current number of iterations t, etc. In addition, before training, it is necessary to set rewards based on the distance information between the robot and the target point (task target point). If the robot reaches the target point, a large positive reward (R+) is given; if the robot is too close to an obstacle or falls, a large negative reward (R-) is given; other states are based on the distance from the target point. Small negative rewards are given (the closer the robot is, the less negative rewards are given). The mathematical expression of reward r can be set according to the needs. In a specific embodiment, reward r = -e γd , where γ is a constant parameter, d is the distance between the robot and the target point, and this embodiment does not specifically limit the mathematical expression of the reward r. It can be seen that the mathematical expression of the reward r can also be simply set to use the opposite number of the distance from the target point, that is, the farther away from the target point, the greater the negative reward. Furthermore, when the environment is relatively complex and the robot's obstacle avoidance requirements are high, it can be considered to use the distance between the robot and the obstacle as a variable in the mathematical expression of the reward r, or the angle between the robot's orientation and the line connecting the robot and the target point can be included in the mathematical expression of the reward r, hoping that the robot will move toward the target point as much as possible.

[0070] When the robot performs a task (sample collection process), it selects the next action of the robot according to the ε greedy strategy, and ε will decrease as the training progresses; wherein, the ε greedy strategy means that under the ε probability, the reinforcement learning agent will randomly perform any legal action in the next step, and under the remaining (1-ε) probability, the reinforcement learning agent's own algorithm determines the next action to be performed, which is determined by the actor network in this embodiment.

[0071] After collecting a certain number of samples, the robot adopts the batch gradient descent optimization strategy to train the aggregation model and updates the parameters of the actor network using policy gradient descent: Optimizing the loss function Update Q and use the smaller gradient to update the target network: Δθ Q′ =τθ Q +(1-τ)θ Q’ and θ μ′ =τθ μ +(1-τ)θ μ’ Here, the target network is updated to make the target network closer to the current target network. μ is the gradient optimization parameter of the actor network μ, N is the amount of data for each training, used for averaging, is the gradient of Q with respect to a, is about θ μ The gradient of Q′ is the gradient optimization parameter of the target critic network Q', τ is the parameter when updating the target network, and the secondary parameter. It can be seen that the gradient optimization parameter Δθ of the critic network Q Q Use the loss function to back propagate; the gradient optimization parameter Δθ of the target actor network μ' μ’ With Δθ Q′ The same update method is used.

[0072] Each time a task is completed, that is, a local training cycle ends, the robot will save the trained local model.

[0073] 203. The robot responds to the timed aggregation request and obtains the target local model and the training contribution of the training target local model.

[0074] When the robot receives the scheduled aggregation request of the Mth aggregation cycle, the robot completes the current step of training the local model to obtain the target local model, that is, after completing the current training, stops training the local model and no longer performs a new step of training.

[0075] After the robot completes the current training, the local model obtained is the target local model. The robot also needs to obtain the resources consumed by itself to train the aggregation model of the Mth aggregation cycle received in step 201 to obtain the target local model. It can be known that the training contribution can be the computing resources required for training or the data resources required for training, such as: at least one of the computing resources of the robot occupied by the training and the number of samples used. This embodiment does not specifically limit the training contribution and the number of training contributions, that is, the training contribution reported by the robot can simultaneously include the computing resources of the robot occupied by the training and the number of samples used by the robot in this round of training.

[0076] In some specific implementations, the timed aggregation request may be initiated by the robot according to a preset duration, or may be sent to the robot by the aggregation server according to a preset duration. This embodiment does not specifically limit the initiation method of the timed aggregation request. When the time from the moment when the aggregation server sends the aggregation model to the robot in the Mth aggregation cycle in step 201 is greater than or equal to the preset duration, the timed aggregation request may be initiated.

[0077] Furthermore, after sending a timed aggregation request to the robot in the Mth aggregation cycle, the following steps may be performed: receiving the local training status of the Mth aggregation cycle sent by each robot, the local training status including the training amplitude of the aggregation model trained by the robot in the Mth aggregation cycle; judging whether the M+1th aggregation cycle meets the preset frequency modulation condition based on the local training status of the Mth aggregation cycle sent by each robot; if the M+1th aggregation cycle meets the preset frequency modulation condition, adjusting the preset duration. That is, when the local training status of the Mth aggregation cycle meets the preset frequency modulation condition, the preset duration may be adjusted to adjust the initiation time of the timed aggregation request of the M+1th aggregation cycle and the aggregation cycles after the M+1th aggregation cycle. Among them, the training amplitude refers to the amplitude of parameter changes of the local model during the process of the robot training the local model in the Mth aggregation cycle. It can be known that the local training status of the Mth aggregation cycle can also be the number of rounds of local training in the Mth aggregation cycle, the average reward of local training in the Mth aggregation cycle, the success rate of task execution in the sample of local training of the robot in the Mth aggregation cycle, or the efficiency of local training of the robot in the Mth aggregation cycle (i.e., the number of trainings per unit time), which is not limited in this embodiment. Further, the local training status of the Mth aggregation cycle can be at least one of the training amplitude of the Mth aggregation cycle, the number of rounds of local training in the Mth aggregation cycle, the average reward of local training in the Mth aggregation cycle, the success rate of task execution in the sample of local training of the robot in the Mth aggregation cycle, or the efficiency of local training of the robot in the Mth aggregation cycle, which is not limited in this embodiment.

[0078] Specifically, the preset frequency modulation condition may be that when the local training condition is less than or equal to a preset lower limit, the preset duration is shortened, and when the local training condition is greater than or equal to a preset upper limit, the preset duration is increased. This embodiment does not make specific limitations.

[0079] 204. The robot sends the target local model and the training contribution of training the target local model to the aggregation server.

[0080] When the robot completes the calculation of the target local model of the Mth aggregation cycle and the training contribution of the training target local model in step 203, the target local model of the Mth aggregation cycle and the training contribution of the training target local model can be sent to the aggregation server, so that the aggregation server determines the target aggregation model of the Mth aggregation cycle based on the target local model of the Mth aggregation cycle and the training contribution of the training target local model.

[0081] 205. The aggregation server obtains a target aggregation model based on the target local model and the training contribution of the trained target local model.

[0082] It can be known that the aforementioned steps 201 to 202 only describe the interaction method between the aggregation server and a robot. In fact, all robots participating in the aggregation model training use the aforementioned method to interact with the aggregation server and jointly complete the training of the aggregation model.

[0083] The aggregation server may determine the target aggregation model for the Mth aggregation period based on the target local model sent by each robot in the Mth aggregation period and the training contribution of the corresponding training target local model.

[0084] Specifically, we can use the formula Calculate the model parameters of the target aggregation model for the Mth aggregation cycle, where θ f is the model parameter of the target aggregation model in the Mth aggregation period, θ n is the model parameter of the target local model of each robot in the Mth aggregation period, v n The training contribution of each robot to train the corresponding target local model for the Mth aggregation cycle; according to the model parameters θ of the target aggregation model f The target aggregation model for the Mth aggregation cycle can be determined.

[0085] Furthermore, if multiple training contributions are included in the calculation of training contribution, different influencing factors can be added to each training contribution according to the task requirements and actual conditions. Each training contribution is multiplied by the corresponding influencing factor, and finally the results of multiplying each training contribution by the corresponding influencing factor are added together to obtain the final training contribution of each robot (v n ) can be used to pass the formula Compute the model parameters of the target aggregation model.

[0086] See also Figure 4Furthermore, in some specific implementations, different navigation tasks can be designed for each robot to reflect the heterogeneity of the environment system. The aggregation server sends the aggregation model to each robot participating in the training at the beginning of each aggregation cycle. Each robot trains the aggregation model according to the local data set in the corresponding aggregation cycle (the local training rounds of each robot in an aggregation cycle may be the same or different). Finally, after responding to the timed aggregation request, the local target model of the corresponding aggregation cycle and the training contribution of the training corresponding local target model are determined and sent to the aggregation server. The aggregation server obtains the target aggregation model of the corresponding aggregation cycle according to the local target model and the training contribution of the training corresponding local target model. If the target aggregation model has not converged, the next aggregation cycle training is carried out until the aggregation model converges.

[0087] In this embodiment, the proposed asynchronous aggregation algorithm utilizes the heterogeneity of different robots in different training environments and their own movement speeds and model processing capabilities, which can effectively reduce the time cost of the federated learning process and improve the convergence speed of the global model. Specifically, first, compared with the single robot navigation task strategy based on reinforcement learning, the embodiment of the present application uses an online multi-robot federated learning training method to allow each robot to learn the model knowledge of other robots at the same time, and the obtained model generalization ability is stronger than that of a single robot, which greatly increases the scalability and robustness of the entire system. Second, during the training process, the robot only needs to transfer the model with the aggregation server instead of the local original perception and control data, especially in scenarios with high real-time requirements and massive perception data such as high-precision navigation, which greatly reduces the communication cost overhead and improves the training efficiency. Third, the multi-robot federated learning asynchronous model aggregation algorithm proposed in the embodiment of the present application proposes an asynchronous weighted model aggregation algorithm according to different robot training steps for the heterogeneity of the training environment and the differences in the movement speeds and processing capabilities of different robots, which improves the training efficiency and time cost of the model compared with the traditional synchronous algorithm.

[0088] See also Figure 5 , an embodiment of the present application provides an aggregation server, including:

[0089] The receiving unit 501 is used to receive, in the Mth aggregation cycle, the target local model and the training contribution sent by each robot among the multiple robots participating in the federated learning; wherein the target local model and the training contribution are sent by the robot in response to the timed aggregation request; the target local model is obtained by the robot training the aggregation model sent by the aggregation server in the Mth aggregation cycle, when M is 1, the aggregation model is the preset initial aggregation model, and when M is greater than 1, the aggregation model is the target aggregation model obtained in the M-1th aggregation cycle; the training contribution is used to represent the resources consumed by the robot to train the target local model;

[0090] The computing unit 502 is used to obtain a target aggregation model of the Mth aggregation cycle based on the training contribution aggregation of each robot and the target local models sent by multiple robots.

[0091] In a specific implementation, the aggregation server further includes: a sending unit;

[0092] A sending unit is used to send a timed aggregation request to each robot if the time from the moment when the aggregation model is sent to each robot in the Mth aggregation cycle is greater than or equal to a preset time, so that each robot sends the target local model trained in the Mth aggregation cycle after receiving the timed aggregation request.

[0093] In a specific implementation, the receiving unit 501 is further used to receive the local training status of the Mth aggregation cycle sent by each robot, where the local training status includes the training amplitude of the aggregation model trained by the robot in the Mth aggregation cycle;

[0094] The aggregation server also includes: a judgment unit and an adjustment unit;

[0095] A judgment unit, used to judge whether the M+1th aggregation period meets the preset frequency modulation condition based on the local training status of the Mth aggregation period sent by each robot;

[0096] The adjustment unit is used to adjust the preset duration if the M+1th aggregation period meets the preset frequency modulation condition.

[0097] In a specific implementation, the calculation unit 502 is specifically configured to calculate the Calculate the model parameters of the target aggregation model, where θ f are the model parameters of the target aggregation model, θ n is the model parameter of the target local model for each robot, v n Training contribution of the corresponding target local model for each robot;

[0098] The model parameters θ of the target aggregation model f Determine the target aggregation model for the Mth aggregation cycle.

[0099] See also Figure 6 , an embodiment of the present application provides a robot, comprising:

[0100] The training unit 601 is used to train the aggregation model sent by the aggregation server in the Mth aggregation cycle; when M is 1, the aggregation model is a preset initial aggregation model, and when M is greater than 1, the aggregation model is a target aggregation model obtained in the M-1th aggregation cycle;

[0101] A determination unit 602 is used to respond to the timing aggregation request of the Mth aggregation period, determine the target local model obtained by training the aggregation model, and determine the training contribution of the robot training the target local model; the training contribution is used to represent the resources consumed by the robot training the target local model;

[0102] The sending unit 603 is used to send the target local model and the training contribution to the aggregation server, so that the aggregation server determines the target aggregation model of the Mth aggregation period according to the target local model and the training contribution.

[0103] In a specific implementation, the sending unit 603 is also used to respond to a request to obtain the training status of the Mth aggregation cycle, and send the local training status of the Mth aggregation cycle to the aggregation server, so that the aggregation server determines whether the M+1th aggregation cycle meets the preset frequency modulation conditions based on the local training status; the local training status includes the training amplitude of the robot training the aggregation model in the Mth aggregation cycle.

[0104] Figure 7 It is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. The data processing device 700 may include one or more central processing units (CPU) 701 and a memory 705. The memory 705 stores one or more application programs or data.

[0105] The memory 705 may be a volatile storage or a persistent storage. The program stored in the memory 705 may include one or more modules, each of which may include a series of instruction operations in the data processing device. Furthermore, the central processing unit 701 may be configured to communicate with the memory 705 and execute a series of instruction operations in the memory 705 on the data processing device 700.

[0106] The data processing device 700 may also include one or more power supplies 702, one or more wired or wireless network interfaces 703, one or more input and output interfaces 704, and / or one or more operating systems, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0107] The CPU 701 can execute the aforementioned Figures 1 to 6 The operations performed by the aggregation server or robot in the illustrated embodiment will not be described in detail here.

[0108] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0109] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0110] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0111] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0112] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A model aggregation method for reinforcement learning based on federated learning, characterized in that: An aggregation server applied to a multi-robot collaborative navigation scenario, the method comprising: In the Mth aggregation cycle, a target local model and a training contribution sent by each robot among the multiple robots participating in the federated learning are received; wherein the target local model and the training contribution are sent by the robot in response to the timed aggregation request; the target local model is obtained by the robot training the aggregation model sent by the aggregation server in the Mth aggregation cycle, when M is 1, the aggregation model is a preset initial aggregation model, and when M is greater than 1, the aggregation model is the target aggregation model obtained in the M-1th aggregation cycle; the training contribution is used to represent the resources consumed by the robot in training the target local model; Obtaining a target aggregation model of an Mth aggregation period based on the training contribution aggregation of each robot and the target local models sent by the multiple robots; The step of aggregating the target local models sent by the multiple robots based on the training contribution of each robot to obtain the target aggregate model of the Mth aggregation period includes: Based on the formula Calculate the model parameters of the target aggregation model for the Mth aggregation period, where is the model parameter of the target aggregation model in the Mth aggregation period, are the model parameters of the target local model of each robot in the Mth aggregation period, The training contribution of the nth robot to the corresponding target local model in the Mth aggregation period, Includes the number of samples used for training the nth robot in the Mth aggregation period; Aggregate model parameters based on target Determining a target aggregation model for the Mth aggregation cycle; After sending a timing aggregation request to each of the robots, the method further includes: Receiving a local training status of the Mth aggregation cycle sent by each of the robots, wherein the local training status includes a training amplitude of the aggregation model trained by the robot in the Mth aggregation cycle; Based on the local training status of the Mth aggregation cycle sent by each of the robots, determining whether the M+1th aggregation cycle meets the preset frequency modulation condition; If the M+1th aggregation period meets the preset frequency modulation condition, the preset duration is adjusted, and the scheduled aggregation request is initiated regularly according to the preset duration.

2. The method according to claim 1, characterized in that Before receiving the target local model and training contribution sent by each robot among the multiple robots participating in the federated learning, the method includes: If the duration from the moment when the aggregation model is sent to each robot in the Mth aggregation cycle is greater than or equal to the preset duration, a timed aggregation request is sent to each robot so that each robot sends the target local model trained in the Mth aggregation cycle after receiving the timed aggregation request.

3. A model aggregation method for reinforcement learning based on federated learning, characterized in that: A robot applied to a multi-robot collaborative navigation scenario, the method comprising: In the Mth aggregation cycle, the aggregation model sent by the aggregation server is trained; when M is 1, the aggregation model is a preset initial aggregation model, and when M is greater than 1, the aggregation model is a target aggregation model obtained in the M-1th aggregation cycle; In response to the timed aggregation request of the Mth aggregation cycle, determine a target local model obtained by training the aggregation model and determine a training contribution of the robot for training the target local model; the training contribution is used to represent the resources consumed by the robot for training the target local model; Send the target local model and training contribution to the aggregation server so that the aggregation server can Calculate the model parameters of the target aggregation model for the Mth aggregation cycle, and use the model parameters of the target aggregation model Determine the target aggregation model of the Mth aggregation cycle; wherein, is the model parameter of the target aggregation model in the Mth aggregation period, are the model parameters of the target local model of each robot in the Mth aggregation period, The training contribution of each robot to train the corresponding target local model for the Mth aggregation period, Includes the number of samples used for training the nth robot in the Mth aggregation period; The method further comprises: In response to a request to obtain the training status of the Mth aggregation cycle, the local training status of the Mth aggregation cycle is sent to the aggregation server, so that the aggregation server determines whether the M+1th aggregation cycle meets the preset frequency modulation condition based on the local training status, and adjusts the preset duration when the M+1th aggregation cycle meets the preset frequency modulation condition, and the timed aggregation request is initiated according to the preset duration; the local training status includes the training amplitude of the aggregation model trained by the robot in the Mth aggregation cycle.

4. An aggregation server, characterized in that: Applied to multi-robot collaborative navigation scenarios, including: A receiving unit, configured to receive, in the Mth aggregation cycle, a target local model and a training contribution sent by each robot among a plurality of robots participating in federated learning; wherein the target local model and the training contribution are sent by the robot in response to a timed aggregation request; the target local model is obtained by the robot training the aggregation model sent by the aggregation server in the Mth aggregation cycle, when M is 1, the aggregation model is a preset initial aggregation model, and when M is greater than 1, the aggregation model is a target aggregation model obtained in the M-1th aggregation cycle; the training contribution is used to represent the resources consumed by the robot in training the target local model; A computing unit, configured to obtain a target aggregation model of an Mth aggregation period based on the training contribution aggregation of each robot and the target local models sent by the multiple robots; The calculation unit is specifically used based on the formula Calculate the model parameters of the target aggregation model for the Mth aggregation period, where is the model parameter of the target aggregation model in the Mth aggregation period, are the model parameters of the target local model of each robot in the Mth aggregation period, The training contribution of the nth robot to the corresponding target local model in the Mth aggregation period, Includes the number of samples used for training the nth robot in the Mth aggregation period; The receiving unit is also used to receive the local training status of the Mth aggregation cycle sent by each of the robots, and the local training status includes the training amplitude of the aggregation model trained by the robot in the Mth aggregation cycle; based on the local training status of the Mth aggregation cycle sent by each of the robots, determine whether the M+1th aggregation cycle meets the preset frequency modulation condition; if the M+1th aggregation cycle meets the preset frequency modulation condition, adjust the preset duration, and the timed aggregation request is initiated according to the preset duration.

5. A robot, characterized in that: Applied to multi-robot collaborative navigation scenarios, including: A training unit, used to train the aggregation model sent by the aggregation server in the Mth aggregation cycle; when M is 1, the aggregation model is a preset initial aggregation model, and when M is greater than 1, the aggregation model is a target aggregation model obtained in the M-1th aggregation cycle; A determination unit, configured to respond to the timing aggregation request of the Mth aggregation period, determine a target local model obtained by training the aggregation model and determine a training contribution of the robot for training the target local model; the training contribution is used to represent resources consumed by the robot for training the target local model; A sending unit is used to send the target local model and the training contribution to the aggregation server so that the aggregation server can Calculate the model parameters of the target aggregation model for the Mth aggregation cycle, and use the model parameters of the target aggregation model Determine the target aggregation model of the Mth aggregation cycle; wherein, is the model parameter of the target aggregation model in the Mth aggregation period, are the model parameters of the target local model of each robot in the Mth aggregation period, The training contribution of each robot to train the corresponding target local model for the Mth aggregation period, Includes the number of samples used for training the nth robot in the Mth aggregation period; The sending unit is also used to respond to a request for obtaining the training status of the Mth aggregation cycle, and send the local training status of the Mth aggregation cycle to the aggregation server, so that the aggregation server determines whether the M+1th aggregation cycle meets the preset frequency modulation condition based on the local training status, and adjusts the preset duration when the M+1th aggregation cycle meets the preset frequency modulation condition, and the timed aggregation request is initiated at a scheduled time according to the preset duration; the local training status includes the training amplitude of the aggregation model trained by the robot in the Mth aggregation cycle.

6. A data processing device, characterized in that: include: CPU, memory and input / output interface; The memory is a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method of any one of claims 1 to 2 or claim 4.

7. A computer storage medium, characterized in that: The computer storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the method according to any one of claims 1 to 2 or claim 4.

Images