A distributed multi-agent reinforcement learning algorithm for virtual-real transfer

Abstract

This invention discloses a distributed multi-agent reinforcement learning virtual-real transfer algorithm, comprising the following steps: Step 1: Building a simulation environment based on the actual task environment; Step 2: Initializing the multi-agent reinforcement learning network parameters, agent parameters, and experience pool; Step 3: Adding randomness to the physical parameters of the simulation environment in each segment of the simulation training; Step 4: Adding randomness to the observation information of each agent in the simulation environment; Step 5: Agents communicate with each other to obtain the observation information of other agents, merge it with their own observation information to form a state, input it into the reinforcement learning policy network, and the agent executes the action of the next moment according to the output of the reinforcement learning network; etc. The virtual-real transfer algorithm and platform involved in this solution have good performance capabilities for solving the virtual-real transfer problem in multi-agent reinforcement learning algorithms, and have great application value for the deployment of multi-agent reinforcement learning.

Description

Technical Field

[0001] This invention relates to an algorithm for virtual-real transfer in distributed multi-agent reinforcement learning, belonging to the field of reinforcement learning virtual-real transfer technology. Background Technology

[0002] The rapid development of computer science in recent years has driven the vigorous development of artificial intelligence (AI) technology. With the continuous maturation of technologies such as deep learning and reinforcement learning, AI has achieved remarkable success in fields such as image recognition, natural language processing, autonomous driving, and multi-agent systems. A multi-agent system is a collective of multiple unmanned systems that communicate and coordinate with each other through a communication network, thereby achieving more efficient and accurate task processing and completing designated goals.

[0003] Multi-agent reinforcement learning (MAL), as a machine learning technique, has garnered significant attention in recent years. Agents continuously interact with the dynamic environment, updating their value under different environmental states, and maximizing rewards through trial and error. MAL provides a highly adaptable and scalable method for the execution strategies of agents in multi-agent systems. Compared to traditional single-agent reinforcement learning, it possesses stronger robustness; even if one agent in the cluster fails, it does not affect the work of other agents, ensuring task completion to a certain extent. In intelligent manufacturing, this technology can effectively coordinate task allocation and collaboration among robots, improving factory production efficiency. In the field of autonomous driving, MAL can streamline traffic flow and alleviate traffic congestion. However, in practical applications, extensive training in real physical environments is impractical. A better solution is extensive training in simulations followed by deployment to real physical environments. However, the differences between simulation and real environments often result in poor deployment performance. Summary of the Invention

[0004] This invention establishes a multi-agent reinforcement learning simulation and physical platform. In the simulation, parameters of the domain randomized random environment and agent observations are used to iteratively train and output a model. The model is then deployed to the real environment. By using a Kalman filter, the invention solves the problem of how to quickly and efficiently transfer and deploy the multi-agent reinforcement learning model from the simulation environment to the real physical world, while ensuring that the task completion rate in the simulation and the real world is similar.

[0005] Technical solution

[0006] The technical solution of this invention is as follows: a distributed multi-agent reinforcement learning virtual-real transfer algorithm. In order to verify the performance and reliability of the algorithm, a multi-agent reinforcement learning virtual-real transfer simulation platform and a physical platform were also built.

[0007] The multi-agent reinforcement learning simulation platform includes the following modules: multi-agent reinforcement learning algorithm module, environment module, agent module, communication module, and data recording and evaluation module.

[0008] The multi-agent reinforcement learning algorithm module integrates various multi-agent reinforcement learning algorithms and achieves modular design through aligned input and output interfaces, including algorithms such as MADDPG, MATD3, QMIX, and VDN.

[0009] The environment module, based on ROS2 and Gazebo tools, can create simulation environments, including scenarios that agents may face, scenario modeling, tasks, obstacles, physics engines, etc.

[0010] The agent module is the execution module for multi-agent reinforcement learning, including but not limited to: robots, sensors, actuators, etc., which have kinematic and dynamic constraints of the real physical world.

[0011] The communication module is based on the subscription / publishing model of ROS2's data distribution service, providing a message passing interface that enables agents to communicate with each other and share information.

[0012] The data recording and evaluation module records key data during the simulation training process, including but not limited to the agent's state, actions, and rewards, and analyzes and compares the performance of different agents and algorithms to help users analyze the training structure and understand the agent's learning trajectory and behavior.

[0013] The multi-agent reinforcement learning physical platform includes the following modules: model deployment module, agent module, communication module, and data recording and evaluation module.

[0014] The model deployment module is used to load multi-agent reinforcement learning models onto the real agent module.

[0015] The agent module is an embedded control system for the actual actuator, used to control the agent's actions in real time. It contains decision control algorithms to enable the agent's actions to be executed accurately and efficiently. It also includes various sensor modules to collect its own observation data.

[0016] The communication module is based on the subscription / publishing format of the ROS2-based DDS data distribution service, which ensures that the intelligent agents in the physical world can communicate in real time and share observation information.

[0017] The data recording and evaluation module is used to record data of the multi-agent system during task execution and analyze and evaluate the model's strengths and weaknesses to facilitate subsequent improvements.

[0018] A distributed multi-agent reinforcement learning virtual-to-real transfer algorithm mainly consists of the following steps:

[0019] S1: Build a simulation environment based on the actual task environment, ensuring consistency between the physical and simulation environments. Define a clearly defined task, including but not limited to the number of agents, the conditions for task completion or failure, and the task scenario. Use Markov games to model the task, with the following formula:

[0020]

[0021] in Let N be the set of agents, S be the number of agents, A be the state space, O be the observation space of the agents, T be the state transition function, R be the reward function, Z be the observation function, and γ be the discount factor for the cumulative reward.

[0022] S2: Initialize the parameters θ of the multi-agent reinforcement learning network i The parameters of the agent include the agent's state space s. i ∈S, observation space o i ∈O, action space a i ∈A, reward function r i ∈R, strategy Where i is the identifier of the agent.

[0023] S3: In each segment of the agent's training in the simulation environment, the physical parameters need to be re-randomized. The simulation environment is developed based on the Gazebo tool, and the physical parameters include, but are not limited to, the object's mass, friction, and air resistance. The physical parameters follow a normal or uniform distribution. The formula is expressed as follows:

[0024]

[0025] Where μ represents the set of physical parameters that need to be initialized before the training segment, m i m represents the mass of agent i. r This indicates that the expression follows a pattern with a mean of 0 and a variance of σ. m 2 The normal distribution, f r This indicates that the frictional force follows an upper limit of f. r_max The lower limit is f r_min The uniform distribution of Cd r This indicates that air resistance follows an upper limit of Cd. r_max The lower limit is Cd r_min The uniform distribution.

[0026] S4: In the simulation environment, the observation and action information of the agent at each step needs to have a certain degree of randomness. The agent's observation information represents the information acquired by its own sensors, including but not limited to encoders, gyroscopes, lidar, and visual information. These parameters follow a normal distribution. The formula is as follows:

[0027] o′ i =o i +o r ,o r ~N(0,σ o 2 )

[0028] Where o′ i It is the randomized observation, o i It is the observation quantity acquired by the sensor, o r It follows a pattern with a mean of 0 and a variance of σ. o 2 It follows a normal distribution.

[0029] S5: Based on ROS2's DDS and Gazebo tools, agents in the simulation can exchange data via publish / subscribe. Agents obtain observations o′ from other agents through a communication topology network. other All observations are input into the policy network, which outputs and executes actions. The agent's actions represent the interface between the agent and the environment. The formula is as follows:

[0030] π i (a i |o′ i +o′ other ,θ i )

[0031] a′ i =a i +a r ,a r ~N(0,σ a 2 )

[0032] Where a′ i It is the randomized action, a i It is the observation acquired by the sensor, a r It follows a pattern with a mean of 0 and a variance of σ. a 2 It follows a normal distribution.

[0033] S6: Establish an experience pool of fixed size. The agent's trajectory information is an array consisting of the previous time step's state, action, reward value, and the next time step's state, expressed by the formula (s t ,a t ,rt ,s t+1 The weights are strictly allocated according to the reward value, with higher-weighted experiences having a greater probability of being sampled for training. The agent's policy learning can be described as an optimization problem, as shown in the following formula:

[0034]

[0035]

[0036]

[0037]

[0038] Where V π J(S) represents the state value. The policy function is the expectation of the state value. Eliminating the influence of state S, it depends only on the parameter θ of the policy network π. The better the policy, the larger J(θ). The state value function is the state s at time t. t and action a t Treating the observed values ​​as known values, and the state and actions after time t+1 as unknown variables, we eliminate these variables using the conditional expectation method, V π (S) depends on the current state s t It also depends on the parameters θ and Q of the policy network π. π (s t A t ) represents the action value function, where U t Let denot be a random variable, and the policy gradient be expressed in the following expected form:

[0039]

[0040] S7: Deploy the converged model on each physical agent. The physical agent possesses the same state space, observation space, action space, and reward function as the simulation agent; that is, the physical agent is identical to the simulated agent in terms of hardware sensor configuration, communication, and motion control. In addition, each physical agent needs a reinforcement learning environment to load the agent's model policy.

[0041] S8: The physical agent platform is built on the ROS2 platform. Communication between agents is based on the TCP protocol's DDS service. They subscribe to / publish observation information with each other and correct Kalman filter parameters. The Kalman gain K needs to be calculated. k Estimated value x k And error covariance matrix P k The formula is as follows:

[0042] K k =P′H T (HP′k H T +R) -1

[0043] x k =x′ k +K k (Z k -Hx′ k )

[0044] P k =(IK k H)P′ k

[0045] Where K k Here, P is the Kalman gain, H is the error matrix, R is the observation matrix, and Z is the prediction noise covariance matrix. k x is the observation value at time k. k It is the state at time k, x′ k The unoptimized predicted state at time k, and the agent's observations at time t. t It refers to x in the Kalman filter. k ;

[0046] S9: Use a Kalman filter to predict the observation information of other agents, and combine it with the observation information of the agent itself to form a state variable, which is then passed as input to the reinforcement learning model. The observation information of other agents needs to be predicted using a Kalman filter before being input into the reinforcement learning network, and the predicted state value needs to be calculated. and prediction error covariance matrix P′ k The formula is as follows:

[0047]

[0048] P′ k =AP k-1 A T +Q

[0049] Where A is the state transition matrix, B is the input control matrix, Q is the prediction noise covariance matrix, and u k-1 It represents the effect of external forces on the system at time k-1.

[0050] S10: The physical agent executes the action of the next moment according to the output of the reinforcement learning model, and repeats the execution and perception until the task is completed. Each agent in the task is a distributed deployment of reinforcement learning models. This is an end-to-end multi-agent system virtual-physical transfer method, which does not require manual feature extraction or intermediate design.

[0051] Compared with the prior art, the advantages of the present invention are as follows:

[0052] 1. Low resource costs and easier data acquisition: A large number of samples can be easily generated for training in a virtual environment, while collecting samples in a real environment is more difficult and expensive.

[0053] 2. Enhanced model generalization ability: In a multi-agent environment, domain randomization is used to expand the domain of the reinforcement learning training model and simulate the physical environment of the real deployment model, thereby enhancing the model's generalization ability.

[0054] 3. More accurate physical sensor data: The Kalman filter algorithm is used to iteratively update and predict the parameters of the communication data, ensuring the accuracy of the data.

[0055] 4. Fully Distributed: Based on a real-time distributed architecture of data distribution services, the central node is removed, which helps to improve the robustness of task execution even if a certain agent has a problem.

[0056] 5. A new solution is provided for the application of multi-agent reinforcement learning in the real world: The algorithm involved in this invention introduces domain randomization and Kalman filtering algorithm, which narrows the gap between the simulation environment and real physics, improves the generalization of the model, and provides a new solution for the application of multi-agent reinforcement learning in the real world. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the structure of the present invention.

[0058] Figure 2 This is a flowchart of the distributed multi-agent reinforcement learning virtual-real transfer algorithm of the present invention. Detailed Implementation

[0059] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples, so that the process of how the present invention uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly. It should be noted that, as long as there is no conflict, the various embodiments and features in the various embodiments of the present invention can be combined with each other, and the resulting technical solutions are all within the protection scope of the present invention.

[0060] Example: To verify the proposed distributed multi-agent reinforcement learning virtual-to-real transfer algorithm, a distributed multi-agent reinforcement learning virtual-to-real transfer platform was also built. The platform includes a simulation part and a physical part. The simulation platform includes a multi-agent reinforcement learning algorithm module, an agent module, an environment module, a communication module, and a data recording and evaluation module. The physical platform includes a model deployment module, an agent module, a communication module, and a data recording and evaluation module.

[0061] The multi-agent reinforcement learning simulation platform includes the following modules: multi-agent reinforcement learning algorithm module, environment module, agent module, communication module, and data recording and evaluation module.

[0062] The multi-agent reinforcement learning algorithm module integrates various multi-agent reinforcement learning algorithms and achieves modular design through aligned input and output interfaces, including algorithms such as MADDPG, MATD3, QMIX, and VDN.

[0063] The environment module, based on ROS2 and Gazebo tools, can create simulation environments, including scenarios that agents may face, scenario modeling, tasks, obstacles, physics engines, etc.

[0064] The agent module is the execution module for multi-agent reinforcement learning, including but not limited to: robots, sensors, actuators, etc., which have kinematic and dynamic constraints of the real physical world.

[0065] The communication module is based on the subscription / publishing model of ROS2's data distribution service, providing a message passing interface that enables agents to communicate with each other and share information.

[0066] The data recording and evaluation module records key data during the simulation training process, including but not limited to the agent's state, actions, and rewards, and analyzes and compares the performance of different agents and algorithms to help users analyze the training structure and understand the agent's learning trajectory and behavior.

[0067] The multi-agent reinforcement learning physical platform includes the following modules: model deployment module, agent module, communication module, and data recording and evaluation module.

[0068] The model deployment module is used to load multi-agent reinforcement learning models onto the real agent module.

[0069] The agent module is an embedded control system for the actual actuator, used to control the agent's actions in real time. It contains decision control algorithms to enable the agent's actions to be executed accurately and efficiently. It also includes various sensor modules to collect its own observation data.

[0070] The communication module is based on the subscription / publishing format of the ROS2-based DDS data distribution service, which ensures that the intelligent agents in the physical world can communicate in real time and share observation information.

[0071] The data recording and evaluation module is used to record data of the multi-agent system during task execution and analyze and evaluate the model's strengths and weaknesses to facilitate subsequent improvements.

[0072] Example 2: A distributed multi-agent reinforcement learning virtual-real transfer algorithm, mainly consisting of the following steps:

[0073] S1: Build a simulation environment based on the actual task environment, ensuring consistency between the physical and simulation environments. Define a clearly defined task, including but not limited to the number of agents, the conditions for task completion or failure, and the task scenario. Use Markov games to model the task, with the following formula:

[0074]

[0075] in Let N be the set of agents, S be the number of agents, A be the state space, O be the observation space of the agents, T be the state transition function, R be the reward function, Z be the observation function, and γ be the discount factor for the cumulative reward.

[0076] S2: Initialize the parameters θ of the multi-agent reinforcement learning network i The parameters of the agent include the agent's state space s. i ∈S, observation space o i ∈O, action space a i ∈A, reward function r i ∈R, strategy Where i is the identifier of the agent.

[0077] S3: In each segment of the agent's training in the simulation environment, the physical parameters need to be re-randomized. The simulation environment is developed based on the Gazebo tool, and the physical parameters include, but are not limited to, the object's mass, friction, and air resistance. The physical parameters follow a normal or uniform distribution. The formula is expressed as follows:

[0078]

[0079] Where μ represents the set of physical parameters that need to be initialized before the training segment, m i m represents the mass of agent i. r This indicates that the expression follows a pattern with a mean of 0 and a variance of σ. m 2 The normal distribution, f r This indicates that the frictional force follows an upper limit of f. r_max The lower limit is f r_min The uniform distribution of Cd r This indicates that air resistance follows an upper limit of Cd. r_max The lower limit is Cd r_min The uniform distribution.

[0080] S4: In the simulation environment, the observation and action information of the agent at each step needs to have a certain degree of randomness. The agent's observation information represents the information acquired by its own sensors, including but not limited to encoders, gyroscopes, lidar, and visual information. These parameters follow a normal distribution. The formula is as follows:

[0081] o′ i =o i +o r ,o r ~N(0,σ o 2 )

[0082] Where o′ i It is the randomized observation, o i It is the observation quantity acquired by the sensor, o r It follows a pattern with a mean of 0 and a variance of σ. o 2 It follows a normal distribution.

[0083] S5: Based on ROS2's DDS and Gazebo tools, agents in the simulation can exchange data via publish / subscribe. Agents obtain observations o′ from other agents through a communication topology network. oth All observations are input into the policy network, which outputs and executes actions. The agent's actions represent the interface between the agent and the environment. The formula is as follows:

[0084] π i (a i |o′ i +o′ othe ,θ i )

[0085] a′ i =a i +a r ,a r ~N(0,σ a 2 )

[0086] Where a′ i It is the randomized action, a i It is the observation acquired by the sensor, a r It follows a pattern with a mean of 0 and a variance of σ. a 2 It follows a normal distribution.

[0087] S6: Establish an experience pool of fixed size. The agent's trajectory information is an array consisting of the previous time step's state, action, reward value, and the next time step's state, expressed by the formula (s t ,a t ,rt ,s t+1 The weights are strictly allocated according to the reward value, with higher-weighted experiences having a greater probability of being sampled for training. The agent's policy learning can be described as an optimization problem, as shown in the following formula:

[0088]

[0089]

[0090]

[0091]

[0092] Where V π J(S) represents the state value. The policy function is the expectation of the state value. Eliminating the influence of state S, it depends only on the parameter θ of the policy network π. The better the policy, the larger J(θ). t and action a t Treating the observed values ​​as known values, and the state and actions after time t+1 as unknown variables, we eliminate these variables using the conditional expectation method, V π (S) depends on the current state s t It also depends on the parameters θ of the policy network π. π (s t A t ) represents the action value function, where U t This represents a random variable.

[0093] The policy gradient can be expressed in the following expected form:

[0094]

[0095] S7: Deploy the converged model on each physical agent. The physical agent possesses the same state space, observation space, action space, and reward function as the simulation agent; that is, the physical agent is identical to the simulated agent in terms of hardware sensor configuration, communication, and motion control. In addition, each physical agent needs a reinforcement learning environment to load the agent's model policy.

[0096] S8: The physical agent platform is built on the ROS2 platform. Communication between agents is based on the TCP protocol's DDS service. They subscribe to / publish observation information with each other and correct Kalman filter parameters. The Kalman gain K needs to be calculated. k Estimated value x k And error covariance matrix P k The formula is as follows:

[0097] K k =P′HT (HP′ k H T +R) -1

[0098] x k =x′ k +K k (Z k -Hx′ k )

[0099] P k =(IK k H)P′ k

[0100] Where K k Here, P is the Kalman gain, H is the error matrix, R is the observation matrix, and Z is the prediction noise covariance matrix. k x is the observation value at time k. k It is the state at time k, x′ k This is the unoptimized predicted state at time k. The agent's observations at time t are... t It refers to x in the Kalman filter. k .

[0101] S9: Use a Kalman filter to predict the observation information of other agents, and combine it with the observation information of the agent itself to form a state variable, which is then passed as input to the reinforcement learning model. The observation information of other agents needs to be predicted using a Kalman filter before being input into the reinforcement learning network, and the predicted state value needs to be calculated. and prediction error covariance matrix P′ k The formula is as follows:

[0102]

[0103] P′ k =AP k-1 A T +Q

[0104] Where A is the state transition matrix, B is the input control matrix, Q is the prediction noise covariance matrix, and u k-1 It represents the effect of external forces on the system at time k-1.

[0105] S10: The physical agent executes the action for the next time step based on the output of the reinforcement learning model, repeating this process until the task is completed. Each agent in the task is a distributed deployment of the reinforcement learning model. This is an end-to-end method for transferring virtual and physical connections in a multi-agent system, eliminating the need for manual feature extraction or intermediate design.

[0106] Example 3: To evaluate the performance of the algorithm, this invention was tested under a three-chasing-one cooperative hunting task. In a 4m*4m space, three moving chasing robots and one target escaping robot were randomly generated. The task was considered successful if two or more chasing robots reached the range of the target robot.

[0107] Some specific module designs are as follows:

[0108] Intelligent Agent: Each intelligent agent consists of a differential two-wheeled robot. Its motion control, observations, and reward values ​​are independent, and it can communicate with other robots, publish its own observations, and subscribe to the observations of others.

[0109] Observations: Observations are composed of data fusion of the observations of the entity itself and the observations of other agents obtained through communication. Specifically, they include angle difference, distance difference, and coordinate information of other agents.

[0110] Action: The target robot's escape strategy is obstacle avoidance using an artificial potential field. The action space is continuous, and its angular velocity v and linear velocity w are designed as follows:

[0111]

[0112]

[0113] Where f x and f y It is the resultant force acting on the target robot in the Cartesian coordinate system, where k1 is the linear velocity constant and k2 is the angular velocity constant.

[0114] To reduce training time and improve the convergence speed of multi-agent reinforcement learning algorithms, the action space of the pursuit robot is set to be discrete, with its linear velocity v and angular velocity w as follows:

[0115]

[0116]

[0117] Where V max and W max These are the maximum linear velocity and angular velocity, and x is the motion.

[0118] Reward Function: The reward function is a crucial component in multi-agent reinforcement learning algorithms. In this cooperative encirclement task, the reward function for agent i is defined as follows:

[0119]

[0120] Where the distance d between agent i and target is... im Less than d min Receive reward rc ,r h The reward for a successful encirclement is r; the reward for a single step in other situations is r. s r s The single-step reward can be expressed by the following formula:

[0121]

[0122] Where k1 and k2 are adjustment parameters. σ(d) and σ(d) are the mean and standard deviation among agents.

[0123] The experimental results above show that using domain randomization to train the model on the simulation platform and employing Kalman filtering and buffer pooling in the real environment ensures the stability of communication in the multi-agent system, enhances the model's generalization ability, and verifies that our proposed algorithm and platform can effectively improve the ability of multi-agent reinforcement learning to transfer between virtual and real environments.

[0124] While the embodiments disclosed in this invention are as described above, the content is merely for the purpose of facilitating understanding of the invention and is not intended to limit the invention. Any person skilled in the art to which this invention pertains may make any modifications and variations in form and detail of the implementation without departing from the spirit and scope disclosed herein; however, the scope of patent protection for this invention shall still be determined by the scope defined in the appended claims.

Claims

1. A distributed multi-agent reinforcement learning virtual-real transfer algorithm, characterized in that: Includes the following steps: S1: Build a simulation environment based on the actual task environment; S2: Initialize the parameters of the multi-agent reinforcement learning network, agent parameters, and experience pool; S3: In each segment of the simulation training, randomness is added to the physical parameters of the simulation environment; S4: In a simulation environment, add randomness to the observation information of each step of the agent; S5: Agents communicate with each other to obtain observation information from other agents, fuse it with their own information to form observation data, and input it into the reinforcement learning policy network. The agents then execute the next action based on the output of the reinforcement learning network. S6: Divide the agent's trajectory information into quadruples and insert them into the experience pool in chronological order. Update the model using priority experience replay and policy gradient methods until the model converges. S7: Deploy the converged model on each physical agent; S8: Agents communicate with each other to obtain observation information from other agents, store the obtained information in a buffer pool in chronological order, and update the Kalman filter parameters; S9: Use a Kalman filter to predict the observation information of other agents, and combine it with its own observation information to form a state variable, which is then passed as input to the reinforcement learning model; S10: The physical intelligent agent executes the action of the next moment according to the output of the reinforcement learning model, and repeats the execution and perception until the task is completed; In step S6, a fixed-size experience pool is established. The agent's trajectory information is an array consisting of the previous state, action, reward value, and next state, expressed by the formula: The weights are strictly allocated according to the reward value. Experiences with higher weights have a greater probability of being sampled for training. The agent's policy learning can be described as an optimization problem, as shown in the following formula: in It is the state value; the policy function calculates the expectation of the state value and eliminates the state. The factors depend solely on the policy network. parameters The better the strategy, the better. The larger the value, the more the state value function represents the state at time t. and actions Treating the observed values ​​as known values, and the state and actions after time t+1 as unknown variables, we use the conditional expectation method to eliminate these variables. It depends on the current state It also depends on policy networks. parameters , Denotes the action value function, where Let denot be a random variable, and the policy gradient be expressed in the following expected form: ； In step S8, the physical intelligent agent platform is built on the ROS2 platform. Communication between the agents is based on the DDS service of the TCP protocol. They subscribe to / publish observation information with each other and correct the Kalman filter parameters, which requires calculating the Kalman gain. Estimated value And error covariance matrix The formula is as follows: in It is Kalman gain. It is the error matrix. It is an observation matrix. It is the prediction noise covariance matrix. These are the observations at time k. It is the state at time k. The unoptimized predicted state at time k, and the agent's observations at time t. It is the Kalman filter ; In step S9, the observation information from other agents needs to be predicted using a Kalman filter before being input into the reinforcement learning network, and the predicted state values ​​need to be calculated. and prediction error covariance matrix The formula is as follows: in It is the state transition matrix. Q is the input control matrix, and Q is the prediction noise covariance matrix. It represents the effect of external forces on the system at time k-1.

2. The distributed multi-agent reinforcement learning virtual-real transfer algorithm as described in claim 1, characterized in that: In step S1, the consistency of tasks in the physical environment and the simulation environment is ensured. A clearly defined task is established, including the number of agents in the task, the conditions for task completion or failure, and the task scenario. Markov game theory is used to model the task, as shown in the following formula: in For a collection of intelligent agents, For the number of intelligent agents, Representing the state space, Represents the action space. Represents the observation space of the intelligent agent. Represents the state transition function. Represents the reward function, Represents the observation function, This represents the discount factor for cumulative rewards.

3. The distributed multi-agent reinforcement learning virtual-real transfer algorithm as described in claim 1, characterized in that: In step S2, the parameters of the multi-agent reinforcement learning network are initialized. The parameters of the agent include the agent's state space. Observation space Action space Reward function ,Strategy ,in It is the identifier of the intelligent agent.

4. The distributed multi-agent reinforcement learning virtual-real transfer algorithm as described in claim 1, characterized in that: In step S3, the physical parameters need to be re-randomized in each segment of the agent training in the simulation environment. The simulation environment is developed based on the Gazebo tool, and the physical parameters include the object's mass, friction, and air resistance. The physical parameters follow a normal distribution or a uniform distribution, as expressed by the following formula: in This represents the set of physical parameters that need to be initialized before the training segment. Represents intelligent agents quality Indicates that it follows the mean. The variance is The normal distribution This indicates that the frictional force follows an upper limit. The lower limit is Uniform distribution This indicates that air resistance follows the upper limit. The lower limit is The uniform distribution.

5. The distributed multi-agent reinforcement learning virtual-real transfer algorithm as described in claim 1, characterized in that: In step S4, the observation and action information of the agent in the simulation environment needs to have a certain degree of randomness. The observation information of the agent represents the information obtained by the agent's own sensors, including encoders, gyroscopes, lidar, visual information, etc. These parameters follow a normal distribution, as expressed by the following formula: in These are the observations after randomization. These are the observations acquired by the sensors. Does it follow the mean? The variance is It follows a normal distribution.

6. The distributed multi-agent reinforcement learning virtual-real transfer algorithm as described in claim 1, characterized in that: In step S5, based on the ROS2-based DDS and Gazebo tools, the agents in the simulation exchange data through a publish / subscribe mechanism, and the agents obtain the observations of other agents through a communication topology network. All observations are input into the policy network, which outputs and executes actions. The agent's actions represent the interface between the agent and the environment, as expressed in the following formula: in It is the randomized action. These are the observations acquired by the sensors. Does it follow the mean? The variance is It follows a normal distribution.

7. The distributed multi-agent reinforcement learning virtual-real transfer algorithm as described in claim 1, characterized in that: In step S7, the physical intelligent agent has the same state space, observation space, action space and reward function as the simulation environment. That is, the physical intelligent agent is the same as the simulated intelligent agent in terms of hardware sensor configuration, communication and motion control. In addition, each physical intelligent agent also needs to have a reinforcement learning environment to load the agent model policy.

8. A distributed multi-agent reinforcement learning virtual-to-real transfer platform, implementing the transfer algorithm described in any one of claims 1-7, wherein the platform includes a simulation part and a physical part, the simulation platform includes a multi-agent reinforcement learning algorithm module, an agent module, an environment module, a communication module, and a data recording and evaluation module, and the physical platform includes a model deployment module, an agent module, a communication module, and a data recording and evaluation module.

Images