Multi-agent robot collaborative control method and system based on distributed reinforcement learning

Abstract

The present invention discloses a multi-agent robot collaborative control method based on distributed reinforcement learning, which defines state space, action space and reward function by constructing a multi-agent reinforcement learning algorithm environment; constructs a distributed reinforcement learning training framework, including a sampling module, a training module, a scheduling management and storage module; based on the "actor-critic" architecture, constructs an independent actor network and critic network for each robot agent; through the distributed training framework, realizes sample data sharing of the sampling module and neural network parameter sharing and collaborative training of the training module; finally, loads the trained neural network into the multi-agent robot system, and performs collaborative control according to environmental changes and task requirements; thereby, the robot agents of each node can share experience, and perform collaborative training through neural network parameter sharing, thereby realizing a collaborative control strategy and improving the overall performance and robustness of the system.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a multi-agent robot collaborative control method and system based on distributed reinforcement learning. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and robotics, Multi-Agent Robot Systems (MARS) have shown unprecedented potential and application value in many fields such as collaborative work, resource optimization and allocation, and exploration of unknown environments due to their unique advantages. Such systems can efficiently and collaboratively complete complex tasks by integrating multiple agents with autonomous decision-making and interactive capabilities, thereby improving overall work efficiency and system robustness.

[0003] In the early development stage of multi-agent robot systems, centralized control methods were a common choice. In this control mode, a central control node is responsible for collecting the status information, task requirements, and environmental data of all agents, and formulates a global control strategy based on this information, and then sends specific control instructions to each agent. The advantage of this method is that it can achieve unified planning and scheduling of multi-agent behaviors, ensuring the orderliness and consistency of the overall system.

[0004] However, with the deepening of the application of multi-agent robot systems and the expansion of their scale, the limitations of centralized control methods have gradually emerged: first, the scalability of the system is limited. As the number of agents increases, the amount of data that the central control node needs to process increases sharply, resulting in increased computing burden and communication delays, which in turn affects the overall performance of the system; second, the system has low fault tolerance. The central control node is the core of the entire system. Once it fails or is damaged, the entire system will be paralyzed and it will be difficult to continue to perform tasks; in addition, centralized control methods may also lead to excessive dependence between agents, lack of autonomous decision-making and adaptability, and difficulty in coping with complex and changing environments and task requirements. Summary of the invention

[0005] In view of the above problems, the present invention provides a multi-agent robot collaborative control method and system based on distributed reinforcement learning, exploring a more efficient, flexible and robust distributed control strategy to replace the centralized control method. By having each agent independently learn in the environment and share experience with each other, the flexibility and adaptability of the system can be improved while reducing dependence on a single control node, thereby effectively overcoming the limitations of the centralized control method.

[0006] To achieve the above object, the present invention provides a multi-agent robot collaborative control method based on distributed reinforcement learning, which comprises the following steps:

[0007] Step a. Construct a multi-agent reinforcement learning algorithm environment according to the robot collaborative control scenario, and define the state space, action space and reward function; wherein the state space includes the position coordinates, speed information, environmental perception information and task progress information of the robot body in the environment; the action space includes movement instructions and operation instructions; the reward function is constructed by combining task completion, collaborative efficiency and resource consumption;

[0008] Step b. Construct a distributed reinforcement learning training framework, including a sampling module, a training module, a scheduling management and storage module; wherein the sampling module is used to interact with the environment to collect sample data of state, action, and reward, the training module is used to train the sample data and update the neural network parameters, and the scheduling management and storage module is used to schedule and manage the communication and storage between each distributed node;

[0009] Step c. Based on the “actor-critic” architecture, an independent actor network and critic network are constructed for each robot agent; the actor network is used to generate action strategies according to the current state of the robot agent, and the critic network is used to evaluate the action value;

[0010] Step d. using a distributed training framework, sharing sample data of the sampling module and sharing neural network parameters of the training module are realized, and collaborative training of the actor network and the critic network is performed using the shared data;

[0011] Step e. Load the trained neural network into the multi-agent robot system and perform collaborative control according to environmental changes and task requirements.

[0012] Preferably, in the step a:

[0013] In the state space, the position coordinates refer to the specific position coordinates of the robot body in the environment, the speed information includes the movement speed and rotation speed of the robot body, the environmental perception information includes the obstacle position coordinates, the position coordinates of other robot bodies, and the position coordinates of the working area, and the task progress information refers to the progress or stage of the task currently being executed; and the state space also includes timing information and historical state information;

[0014] The movement instructions of the action space include forward, backward, left turn, and right turn, and the operation instructions of the action space include grab, release, and rotation;

[0015] In the reward function, the task completion reward is a corresponding reward given according to the task goal; the collaborative efficiency reward is used to encourage collaborative work between robots; and the resource consumption reward includes the robot's energy consumption factor and time cost factor.

[0016] Preferably, the reward function specifically includes:

[0017] a1.Task completion rewards:

[0018] According to the task objectives, the robot is given a positive reward when it successfully completes the task, and a negative reward when it fails to complete the task. The weight of the positive reward is greater than the weight of the negative reward.

[0019] a2. Collaborative efficiency rewards:

[0020] Collaborative behavior reward: When the robot completes collaborative tasks with other robots, it is given a positive reward; the collaborative tasks include carrying heavy objects together with other robots and avoiding collisions with other robots; if the robot collides with other robots or obstacles, it is given a negative reward; the weight of the negative reward is greater than the weight of the positive reward;

[0021] a3. Resource consumption rewards:

[0022] According to the robot's energy consumption, a preset negative energy consumption reward is given; if the task completion time is too long, a negative time cost reward is given; among them, the weight of the negative time cost reward is greater than the weight of the negative energy consumption reward;

[0023] Also includes:

[0024] a4. Position and speed rewards:

[0025] A gradually increasing positive target approach reward is given based on the distance between the robot's current position and the target position; a positive speed optimization reward is given if the robot moves with higher efficiency without sacrificing safety; the weight of the target approach reward is greater than the weight of the speed optimization reward;

[0026] a5.Environmental Perception Rewards:

[0027] If the robot successfully avoids colliding with an obstacle, it is given a positive obstacle avoidance reward; when the robot enters the designated work area, it is given a positive work area entry reward; the weight of the work area entry reward is greater than the weight of the obstacle avoidance reward;

[0028] a6. Timing and historical status rewards:

[0029] If the robot's behavior is temporally coherent and helps complete the task, give positive rewards;

[0030] Furthermore, when a reward of a specified type reaches a preset score or number of times, the reward weight of that reward type is increased.

[0031] Preferably, in the step b, the sampling module collects sample data by executing the action strategy output by the actor network, and stores the sample data as training samples by continuously executing actions and observing environmental feedback; the training module receives the training samples for training, calculates the action value using the critic network, and updates the parameters of the actor network and the critic network using the training results through the back propagation algorithm; at the same time, the training module communicates with the scheduling management and storage module to obtain more training data or update status information, and stores the training samples, intermediate results generated by the training process or the final model in the scheduling management and storage module.

[0032] Preferably, in the step c:

[0033] The actor network, the input information includes the robot's local observation information, environmental observation information, the status of other robots and shared information; the output information is the action probability distribution or decision action of each possible action taken by the robot under the current observation; wherein the local observation information includes the robot's sensor data;

[0034] The critic network inputs information including local observation information of the robot, action probability distribution or decision action of the actor network, and current action of the robot; and outputs information including action value of the current action.

[0035] Preferably, the actor-critic algorithm is a reinforcement learning method that combines policy gradient and value function estimation, the actor network selects action strategies through a policy gradient algorithm, and the critic network evaluates action values ​​through value function estimation.

[0036] Preferably, the step c further comprises:

[0037] c1. Build a master-slave or parameter server distributed training architecture; the master node or parameter server is used to store and update global parameters, and the slave node or worker node is used for data sampling, local gradient calculation and parameter update;

[0038] c2. Setting an experience replay buffer for storing sample data sampled from the environment; wherein the data in the experience replay buffer is globally shared, and the slave node or the worker node samples data from the experience replay buffer for training the actor network and the critic network;

[0039] c3. The neural network parameters of the actor network and the critic network are centrally managed and updated on the master node or parameter server. The slave nodes or working nodes upload local gradients to the master node or parameter server during the training process. After the master node or parameter server aggregates and updates the parameters, it sends the updated parameters to the slave nodes or working nodes.

[0040] Preferably, the collaborative training process of step d specifically includes the following steps:

[0041] d1. Initialize the network:

[0042] Initialize an actor network and a critic network for each agent;

[0043] d2. Sampling and experience collection:

[0044] The actor network of each agent outputs a control strategy based on the current state and sends control instructions to the robot body; the sample data generated by the agent during the execution of actions in the simulation environment, the sample data includes state values, observation values, and reward values; and is sent to the global controller and saved in the experience pool in an asynchronous manner;

[0045] d3. Network training and updating:

[0046] In each training round, the training module obtains sample data from the experience pool for calculation. The critic network predicts the action value based on the current state and current action, uses the predicted value and the actual reward to calculate the mean square error MSE loss function, and updates the parameters of the critic network through the back-propagation algorithm; the actor network uses the action value information provided by the critic network, combined with the policy gradient method to calculate the gradient of the actor network parameters, and uses the gradient descent method to update the parameters of the actor network to optimize the strategy; the training module sends the updated parameters of the critic network and the actor network to the sampling module through the global controller to update the parameters of the network;

[0047] d4. Agent evaluation and adjustment:

[0048] Evaluate the performance of the trained multi-agents in a simulation environment to observe whether the agents can effectively collaborate and achieve the optimization goals; and adjust the network structure and parameters of the agents based on the evaluation results.

[0049] Preferably, the step e further comprises:

[0050] e1. Export the trained neural network as a TensorFlow Lite model or a TensorFlow.js model;

[0051] e2. Load the exported neural network model into the control module of the robot system;

[0052] e3. Integrate and coordinate the neural network model with the perception module, decision-making and planning module, and communication module of the robot system.

[0053] In addition, the present invention also provides a multi-agent robot collaborative control system based on distributed reinforcement learning, the system includes a memory, a processor, and a multi-agent robot collaborative control program based on distributed reinforcement learning stored in the memory and executable on the processor, and the multi-agent robot collaborative control program based on distributed reinforcement learning, when executed by the processor, implements the steps of the multi-agent robot collaborative control method based on distributed reinforcement learning as described in any one of the above items.

[0054] The beneficial effects of the present invention are:

[0055] (1) The present invention uses a distributed reinforcement learning algorithm to enable the robot agents at each node to share experience and conduct collaborative training through neural network parameter sharing, thereby realizing a collaborative control strategy and improving the overall performance and robustness of the system.

[0056] (2) The distributed training framework of the present invention provides the functions of task scheduling, distributed state management and high-performance computing, which can simplify parallel computing, accelerate model training, and realize an efficient computing environment through distributed task management.

[0057] (3) Through distributed reinforcement learning, the node robots can collaboratively train optimization strategies to improve task completion efficiency.

[0058] (4) The present invention adopts a distributed computing architecture to reduce communication volume and computational complexity.

[0059] (5) The present invention adopts a distributed reinforcement learning algorithm to learn efficient resource allocation and collaborative control strategies, thereby improving the overall performance and efficiency of the system.

[0060] (6) The reward function scheme of the present invention can produce many beneficial effects in the multi-agent robot system, including improving the task completion rate, coordination efficiency, resource utilization efficiency, safety and stability. These effects work together on the system and help improve the overall performance and reliability of the multi-agent robot system. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0062] Figure 1 This is an architecture diagram of the distributed training framework of the present invention;

[0063] Among them, Actor represents the sampling module, Learner represents the training module, ENV represents the environment, input represents the model input, and network represents the neural network. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the following will be clearly and completely described in combination with the specific technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0065] With the rapid development of robotics technology, Multi-Agent Robot Systems (MARS) have shown great potential in collaborative operations, resource allocation, environmental exploration and other fields. However, traditional centralized control methods have problems such as poor scalability and low fault tolerance in multi-agent systems, and are difficult to adapt to complex and changing environments. Distributed Reinforcement Learning (DRL) is an effective means to solve multi-agent collaborative control. By allowing each node agent to share experience and collaborate in training, it can improve the flexibility and adaptability of the system while reducing dependence on a single control node.

[0066] However, in traditional distributed systems, although the training process can be accelerated through parallel computing, the convergence speed may be limited because different robots may have data inconsistencies due to different observed environmental information or different processing speeds.

[0067] Based on this, the present invention provides a multi-agent robot collaborative control method based on distributed reinforcement learning, which comprises the following steps:

[0068] Step a. Reinforcement learning algorithm environment modeling:

[0069] According to the robot collaborative control scenario, a multi-agent reinforcement learning algorithm environment is constructed, and the state space, action space and reward function are defined; the state space includes the position coordinates, speed information, environmental perception information and task progress information of the robot body in the environment; the action space includes movement instructions and operation instructions; the reward function is constructed by combining task completion, collaborative efficiency and resource consumption;

[0070] Step b. Build a distributed reinforcement learning training framework:

[0071] Construct a distributed reinforcement learning training framework, including a sampling module, a training module, a scheduling management and storage module; the sampling module is used to interact with the environment to collect sample data of state, action and reward, the training module is used to train the sample data and update the neural network parameters, and the scheduling management and storage module is used to schedule and manage the communication and storage between distributed nodes;

[0072] Step c. Build a reinforcement learning training neural network:

[0073] Based on the "actor-critic" architecture, an independent actor network and critic network are built for each robot agent. The actor network is used to generate action strategies based on the current state of the robot agent, and the critic network is used to evaluate the value of the action.

[0074] Step d. Collaborative control strategy learning:

[0075] Through a distributed training framework, sample data sharing of a sampling module and neural network parameter sharing of a training module are realized, and the actor network and the critic network are collaboratively trained using the shared data;

[0076] Step e. Deployment and application:

[0077] The trained neural network is loaded into the multi-agent robot system to perform collaborative control according to environmental changes and task requirements.

[0078] In this embodiment, in step a:

[0079] In the state space, the position coordinates refer to the specific position coordinates of the robot body in the environment, the speed information includes the movement speed and rotation speed of the robot body, the environmental perception information includes the obstacle position coordinates, the position coordinates of other robot bodies, and the position coordinates of the working area, and the task progress information refers to the progress or stage of the task currently being executed; and the state space also includes timing information and historical state information;

[0080] The movement instructions of the action space include forward, backward, left turn, and right turn, and the operation instructions of the action space include grab, release, and rotation;

[0081] In the reward function, the task completion reward is a corresponding reward given according to the task goal; the collaborative efficiency reward is used to encourage collaborative work between robots; and the resource consumption reward includes the robot's energy consumption factor and time cost factor.

[0082] Reinforcement learning environment modeling is to build a multi-agent reinforcement learning algorithm environment based on the robot collaborative control scenario, including state space, action space and reward function. In addition to basic attributes and motion states, the state space can also introduce timing information or historical state information to fully reflect the state changes of the robot team at different time points and the interaction history between each other. The action space covers the robot's motion instructions, including forward, backward, left and right turns, grasping, releasing, etc. By coordinating these actions, close collaboration and efficient task execution between robots can be achieved. The construction of the reward function should be combined with specific collaborative tasks, while fully considering dynamically changing environmental factors and possible external interference. Such a reward mechanism can motivate the robot team to continuously optimize its collaborative control strategy in a complex and changing environment, and improve the efficiency and success rate of task completion.

[0083] In this embodiment, the reward function specifically includes:

[0084] a1.Task completion rewards:

[0085] According to the task goal, the robot is given a positive reward when it successfully completes the task, and a negative reward when it fails. The weight of the positive reward is greater than the weight of the negative reward. This reward can motivate the robot to complete the task goal first. The weight setting of positive rewards greater than negative rewards makes the robot more inclined to complete the task and obtain higher cumulative rewards. This helps to improve the overall performance of the system and the task completion rate.

[0086] a2. Collaborative efficiency rewards:

[0087] Collaborative behavior rewards: When a robot completes collaborative tasks with other robots, it will be given a positive reward; the collaborative tasks include carrying heavy objects with other robots and avoiding collisions with other robots; if the robot collides with other robots or obstacles, it will be given a negative reward; it helps to improve the collaborative efficiency of the system and reduce resource waste and conflicts. Among them, the weight of negative rewards is greater than the weight of positive rewards; by giving larger negative rewards, the robot is prompted to act more cautiously, reducing the occurrence of collision accidents, and ensuring the safety and stability of the system.

[0088] a3. Resource consumption rewards:

[0089] According to the robot's energy consumption, a preset negative energy consumption reward is given; if the task takes too long to complete, a negative time cost reward is given; the weight of the negative time cost reward is greater than the weight of the negative energy consumption reward; the negative energy consumption reward can motivate the robot to optimize energy consumption, extend battery life, and reduce energy waste; the negative time cost reward prompts the robot to complete the task as quickly as possible and improve overall efficiency. By adjusting the weight of the negative time cost reward, the importance of time efficiency can be further emphasized.

[0090] Also includes:

[0091] a4. Position and speed rewards:

[0092] According to the distance between the robot's current position and the target position, a gradually increasing positive target approach reward is given; if the robot moves with higher efficiency without sacrificing safety, a positive speed optimization reward is given; the weight of the target approach reward is greater than the weight of the speed optimization reward; the target approach reward can guide the robot to quickly approach the target position and improve the speed of task completion. The speed optimization reward can encourage the robot to move with higher efficiency while ensuring safety, which helps to improve the overall response speed and flexibility of the system.

[0093] a5.Environmental Perception Rewards:

[0094] If the robot successfully avoids collision with an obstacle, a positive obstacle avoidance reward is given; when the robot enters the designated work area, a positive work area entry reward is given; the weight of the work area entry reward is greater than the weight of the obstacle avoidance reward; the positive obstacle avoidance reward can enhance the robot's environmental perception ability, reduce collisions with obstacles, and improve the safety and reliability of the system. The positive work area entry reward can guide the robot to enter the designated work area to ensure that the task can be executed correctly. By adjusting the weight of the work area entry reward, the importance of task execution can be further emphasized.

[0095] a6. Timing and historical status rewards:

[0096] If the robot's behavior is coherent in time and helps complete the task, a positive reward will be given; this reward can encourage the robot to maintain the coherence and consistency of behavior, which helps to improve the stability and predictability of the system. By considering historical states, the robot can better adapt to environmental changes and make more reasonable decisions.

[0097] In this embodiment, when the specified type of reward reaches the preset score or number of times, the reward weight of this type of reward is increased; thereby further emphasizing the importance of this type of reward and guiding the robot to pay more attention to this aspect of performance. This dynamic adjustment mechanism makes the reward function more flexible and adaptable, and can better adapt to different tasks and environmental requirements.

[0098] The distributed reinforcement learning algorithm framework adopts a modular design concept. Its structure includes: (1) Actor sampling module: responsible for interacting with the environment to collect samples; (2) Learner training module: responsible for updating the network; (3) Scheduling management and storage module: responsible for scheduling and managing communications between different nodes, divided into local scheduling, object storage and global control storage. The framework architecture supports multiple GPU communication backends, including MPI, NCCL and GLOO.

[0099] In this embodiment, in the step b, the sampling module collects sample data by executing the action strategy output by the actor network, and stores the sample data as training samples by continuously executing actions and observing environmental feedback; the training module receives the training samples for training, calculates the action value using the critic network, and updates the parameters of the actor network and the critic network through the back propagation algorithm with the training results; at the same time, the training module communicates with the scheduling management and storage module to obtain more training data or update status information, and stores the training samples, intermediate results generated by the training process or the final model in the scheduling management and storage module.

[0100] The reinforcement learning training neural network is constructed based on the "actor-critic" method, which combines policy gradient and value function estimation. In a multi-agent system, each robot can be regarded as an independent agent, using the actor-critic method to independently learn its strategy and work together by sharing observation information or global state. A deep neural network is used to approximate the policy function (actor network) and the value function (critic network), where the actor network is responsible for learning independent strategies, that is, selecting actions based on the current robot state to maximize the long-term expected return; the critic network is responsible for learning the value function, that is, evaluating the expected cumulative reward of taking an action in a given state. The input of the actor network is the local observation information of the robot, including the robot's sensor data (such as position, speed, acceleration, etc.), environmental observations (such as obstacle positions, target positions, etc.) and possible communication information (such as the state of other robots), and the output is the probability distribution of the robot taking each action under the current observation. The input of the critic network is the local observation information of the robot and the action probability distribution of the actor network, and the output is the action value, which is used to evaluate the quality of the action.

[0101] In this embodiment, in the step c:

[0102] The actor network, the input information includes the robot's local observation information, environmental observation information, the status of other robots and shared information; the output information is the action probability distribution or decision action of each possible action taken by the robot under the current observation; wherein the local observation information includes the robot's sensor data;

[0103] The critic network inputs information including local observation information of the robot, action probability distribution or decision action of the actor network, and current action of the robot; and outputs information including action value of the current action.

[0104] In this embodiment, the actor-critic algorithm is a reinforcement learning method that combines policy gradient and value function estimation. The actor network selects action strategies through the policy gradient algorithm, and the critic network evaluates action values ​​through value function estimation.

[0105] In this embodiment, the step c further comprises:

[0106] c1. Build a master-slave or parameter server distributed training architecture; the master node or parameter server is used to store and update global parameters, and the slave node or worker node is used for data sampling, local gradient calculation and parameter update;

[0107] c2. Setting an experience replay buffer for storing sample data sampled from the environment; wherein the data in the experience replay buffer is globally shared, and the slave node or the worker node samples data from the experience replay buffer for training the actor network and the critic network;

[0108] c3. The neural network parameters of the actor network and the critic network are centrally managed and updated on the master node or parameter server. The slave nodes or working nodes upload local gradients to the master node or parameter server during the training process. After the master node or parameter server aggregates and updates the parameters, it sends the updated parameters to the slave nodes or working nodes.

[0109] The collaborative control strategy adopts large-scale distributed learning, and performs collaborative training through experience sharing of sampling nodes and neural network parameter sharing of training nodes to achieve collaborative control of multi-agent robots. In this embodiment, the collaborative training process of step d specifically includes the following steps:

[0110] d1. Initialize the network:

[0111] Initialize an actor network and a critic network for each agent.

[0112] d2. Sampling and experience collection:

[0113] The actor network of each agent outputs a control strategy based on the current state and sends control instructions to the robot body; the sample data generated by the agent during the execution of actions in the simulation environment, the sample data includes state values, observation values, and reward values; and is sent to the global controller and saved in the experience pool in an asynchronous manner;

[0114] In this embodiment, the actor network of each agent outputs the optimal control strategy based on the current robot state and environmental information, and sends control instructions to the robot, allowing the agent to interact in the environment and observe the new state and reward obtained after executing the action. In the process of continuous interaction between the sampling module and the simulation environment, a large number of samples (reward values, state values, observation values, etc.) are generated, and the samples are sent to the global controller asynchronously and saved in the experience pool.

[0115] d3. Network training and updating:

[0116] In each training round, the training module obtains sample data from the experience pool for calculation. The critic network predicts the action value based on the current state and current action, uses the predicted value and the actual reward to calculate the mean square error MSE loss function, and updates the parameters of the critic network through the back-propagation algorithm; the actor network uses the action value information provided by the critic network, combined with the policy gradient method to calculate the gradient of the actor network parameters, and uses the gradient descent method to update the parameters of the actor network to optimize the strategy; the training module sends the updated parameters of the critic network and the actor network to the sampling module through the global controller to update the parameters of the network;

[0117] In this embodiment, the training node obtains batch samples from the experience pool to calculate and update the parameters, and then sends the updated parameters to the sampling node through the global controller to update the parameters of the network. The critic network evaluates the value of the action based on the current state and action, uses the predicted value and actual reward of the critic network to calculate the mean square error (MSE) loss function, and updates the parameters of the critic network through back propagation. Using the action value information provided by the critic network, the gradient of the actor network parameters is calculated in combination with the policy gradient method, and the gradient descent method is used to update the parameters of the actor network to optimize the strategy.

[0118] d4. Agent evaluation and adjustment:

[0119] The performance of the trained multi-agent is evaluated in a simulation environment to observe whether the agents can effectively collaborate and achieve the optimization goal; and the network structure and parameters of the agents are adjusted according to the evaluation results. In this embodiment, the network structure, learning rate, discount factor and other parameters of the agents are adjusted according to the evaluation results to further improve the performance.

[0120] In the overall training process, the sampling process is run on the CPU, and each node has an independent sampling module and simulation environment, while the training process is run on the GPU, using the Horovod method and the Ring-Allreduce algorithm to achieve efficient computing and improve the efficiency of the distributed system. This distributed learning framework supports hybrid resource simultaneous training, which can maximize the use of CPU and GPU resources and improve the gradient update speed.

[0121] In this embodiment, the step e further includes:

[0122] e1. Export the trained neural network as a TensorFlow Lite model or a TensorFlow.js model;

[0123] e2. Load the exported neural network model into the control module of the robot system;

[0124] e3. Integrate and coordinate the neural network model with the perception module, decision-making and planning module, and communication module of the robot system.

[0125] This embodiment loads the trained neural network into the actual multi-agent robot system to achieve collaborative control according to environmental changes and task requirements. The deployment methods include executable format and source code form. The executable format refers to exporting the model's calculation graph into a certain executable format, such as TensorFlow Lite model or TensorFlow.js model, and then executing it on the client device. The source code form refers to exporting the model's calculation graph into source code in other languages, such as C++ code, and then compiling and executing it on the client.

[0126] In addition, the present invention also provides a multi-agent robot collaborative control system based on distributed reinforcement learning, the system includes a memory, a processor, and a multi-agent robot collaborative control program based on distributed reinforcement learning stored in the memory and executable on the processor, and the multi-agent robot collaborative control program based on distributed reinforcement learning, when executed by the processor, implements the steps of the multi-agent robot collaborative control method based on distributed reinforcement learning as described in any one of the above items.

[0127] Among them, the memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as an image playback function, etc.), etc.; the data storage area can store data created according to the use of the system, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory can also include a memory controller to provide the processor and the input unit with access to the memory.

[0128] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0129] Furthermore, in this document, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

[0130] The above description shows and describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the invention, through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not depart from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A multi-agent robot collaborative control method based on distributed reinforcement learning, characterized in that: The following steps are involved: Step a. Construct a multi-agent reinforcement learning algorithm environment according to the robot collaborative control scenario, and define the state space, action space and reward function; wherein the state space includes the position coordinates, speed information, environmental perception information and task progress information of the robot body in the environment; the action space includes movement instructions and operation instructions; the reward function is constructed by combining task completion, collaborative efficiency and resource consumption; Step b. Construct a distributed reinforcement learning training framework, including a sampling module, a training module, a scheduling management and storage module; wherein the sampling module is used to interact with the environment to collect sample data of state, action, and reward, the training module is used to train the sample data and update the neural network parameters, and the scheduling management and storage module is used to schedule and manage the communication and storage between each distributed node; Step c. Based on the "actor-critic" architecture, an independent actor network and critic network are constructed for each robot agent; the actor network is used to generate action strategies according to the current state of the robot agent, and the critic network is used to evaluate the action value; Step d. using a distributed training framework, sharing sample data of the sampling module and sharing neural network parameters of the training module are realized, and collaborative training of the actor network and the critic network is performed using the shared data; Step e. Load the trained neural network into the multi-agent robot system and perform collaborative control according to environmental changes and task requirements; The step c further comprises: c1. Build a master-slave or parameter server distributed training architecture; the master node or parameter server is used to store and update global parameters, and the slave node or worker node is used for data sampling, local gradient calculation and parameter update; c2. Setting an experience replay buffer for storing sample data sampled from the environment; wherein the data in the experience replay buffer is globally shared, and the slave node or the worker node samples data from the experience replay buffer for training the actor network and the critic network; c3. The neural network parameters of the actor network and the critic network are centrally managed and updated on the master node or parameter server. The slave nodes or working nodes upload local gradients to the master node or parameter server during the training process. After the master node or parameter server aggregates and updates the parameters, it sends the updated parameters to the slave nodes or working nodes.

2. The method according to claim 1, characterized in that In the step a: In the state space, the position coordinates refer to the specific position coordinates of the robot body in the environment, the speed information includes the movement speed and rotation speed of the robot body, the environmental perception information includes the obstacle position coordinates, the position coordinates of other robot bodies, and the position coordinates of the working area, and the task progress information refers to the progress or stage of the task currently being executed; and the state space also includes timing information and historical state information; The movement instructions of the action space include forward, backward, left turn, and right turn, and the operation instructions of the action space include grab, release, and rotation; In the reward function, the task completion reward is a corresponding reward given according to the task goal; the collaborative efficiency reward is used to encourage collaborative work between robots; and the resource consumption reward includes the robot's energy consumption factor and time cost factor.

3. The method according to claim 2, characterized in that The reward function specifically includes: a1.Task completion rewards: According to the task objectives, the robot is given a positive reward when it successfully completes the task, and a negative reward when it fails to complete the task. The weight of the positive reward is greater than the weight of the negative reward. a2. Collaborative efficiency rewards: Collaborative behavior reward: When the robot completes collaborative tasks with other robots, it is given a positive reward; the collaborative tasks include carrying heavy objects together with other robots and avoiding collisions with other robots; if the robot collides with other robots or obstacles, it is given a negative reward; the weight of the negative reward is greater than the weight of the positive reward; a3. Resource consumption rewards: According to the robot's energy consumption, a preset negative energy consumption reward is given; if the task completion time is too long, a negative time cost reward is given; among them, the weight of the negative time cost reward is greater than the weight of the negative energy consumption reward; Also includes: a4. Position and speed rewards: A gradually increasing positive target approach reward is given based on the distance between the robot's current position and the target position; a positive speed optimization reward is given if the robot moves with higher efficiency without sacrificing safety; the weight of the target approach reward is greater than the weight of the speed optimization reward; a5.Environmental Perception Rewards: If the robot successfully avoids colliding with an obstacle, it is given a positive obstacle avoidance reward; when the robot enters the designated work area, it is given a positive work area entry reward; the weight of the work area entry reward is greater than the weight of the obstacle avoidance reward; a6. Timing and historical status rewards: If the robot's behavior is temporally coherent and helps complete the task, give positive rewards; Furthermore, when a reward of a specified type reaches a preset score or number of times, the reward weight of that reward type is increased.

4. The method according to claim 1, characterized in that In the step b, the sampling module collects sample data by executing the action strategy output by the actor network, and stores the sample data as training samples by continuously executing actions and observing environmental feedback; the training module receives the training samples for training, calculates the action value using the critic network, and updates the parameters of the actor network and the critic network through the back propagation algorithm with the training results; at the same time, the training module communicates with the scheduling management and storage module to obtain more training data or update status information, and stores the training samples, intermediate results generated by the training process or the final model in the scheduling management and storage module.

5. The method according to claim 1, characterized in that In the step c: The actor network, the input information includes the robot's local observation information, environmental observation information, the status of other robots and shared information; the output information is the action probability distribution or decision action of each possible action taken by the robot under the current observation; wherein the local observation information includes the robot's sensor data; The critic network inputs information including local observation information of the robot, action probability distribution or decision action of the actor network, and current action of the robot; and outputs information including action value of the current action.

6. The method according to claim 1 or 5, characterized in that: The actor-critic algorithm is a reinforcement learning method that combines policy gradient and value function estimation. The actor network selects action strategies through the policy gradient algorithm, and the critic network evaluates action values ​​through value function estimation.

7. The method according to any one of claims 1 to 5, characterized in that: The collaborative training process of step d specifically includes the following steps: d1. Initialize the network: Initialize an actor network and a critic network for each agent; d2. Sampling and experience collection: The actor network of each agent outputs a control strategy based on the current state and sends control instructions to the robot body; the sample data generated by the agent during the execution of actions in the simulation environment, the sample data includes state values, observation values, and reward values; and is sent to the global controller and saved in the experience pool in an asynchronous manner; d3. Network training and updating: In each training round, the training module obtains sample data from the experience pool for calculation. The critic network predicts the action value based on the current state and current action, uses the predicted value and the actual reward to calculate the mean square error (MSE) loss function, and updates the parameters of the critic network through the back-propagation algorithm. The actor network uses the action value information provided by the critic network and combines it with the policy gradient method to calculate the gradient of the actor network parameters, and uses the gradient descent method to update the parameters of the actor network to optimize the strategy. The training module sends the updated parameters of the critic network and the actor network to the sampling module through the global controller to update the parameters of the network; d4. Agent evaluation and adjustment: Evaluate the performance of the trained multi-agents in a simulation environment to observe whether the agents can effectively collaborate and achieve the optimization goals; and adjust the network structure and parameters of the agents based on the evaluation results.

8. The method according to any one of claims 1 to 5, characterized in that: The step e further comprises: e1. Export the trained neural network as a TensorFlow Lite model or a TensorFlow.js model; e2. Load the exported neural network model into the control module of the robot system; e3. Integrate and coordinate the neural network model with the perception module, decision-making and planning module, and communication module of the robot system.

9. A multi-agent robot collaborative control system based on distributed reinforcement learning, characterized in that: The system includes a memory, a processor, and a multi-agent robot collaborative control program based on distributed reinforcement learning stored in the memory and executable on the processor. When the multi-agent robot collaborative control program based on distributed reinforcement learning is executed by the processor, the steps of the multi-agent robot collaborative control method based on distributed reinforcement learning as described in any one of claims 1 to 8 are implemented.

Images