Reinforcement learning training method and system for robot agents in complex scenarios

Abstract

The present invention discloses a reinforcement learning training method for a robot agent in complex scenarios. The method comprises constructing a reinforcement learning algorithm environment and a state transition model for a multi-agent system including complex scenarios such as obstacles, dynamic targets and environmental constraints, and adopting a reinforcement learning algorithm to construct an independent neural network for each robot agent. The vectors of the given state, timing information and historical state information of the agent are used as inputs of the state transition model, and the state transition probability and reward information of the predicted state are output. The action strategy and action value output by the neural network are executed in the state transition model to generate new predicted state and reward information. The hyperparameters of the neural network are adjusted, and the decision network and the evaluation network are collaboratively trained using a large-scale distributed training architecture to realize parallel learning and collaborative decision-making of multiple agents, thereby improving the performance and generalization ability of the agents in complex scenarios.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for training robot agent reinforcement learning in complex scenarios. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and robotics, Multi-Agent Robot Systems (MARS) have shown unprecedented potential and application value in many fields such as collaborative work, resource optimization and allocation, and exploration of unknown environments due to their unique advantages. Such systems can efficiently and collaboratively complete complex tasks by integrating multiple agents with autonomous decision-making and interactive capabilities, thereby improving overall work efficiency and system robustness.

[0003] Currently, there is a growing demand for the application of robotic agents in complex scenarios, including but not limited to city streets, factory production lines, and natural disaster rescue sites, which are often highly dynamic, uncertain, and diverse.

[0004] Traditional training methods are mostly based on preset rules and models, which lack adaptability to environmental changes and task requirements. When the environment changes or task requirements change, the robot agent may not be able to respond effectively, resulting in performance limitations or even task failure.

[0005] Specifically include:

[0006] On the one hand, the environmental states in complex scenes are often high-dimensional and continuous, which makes the exploration of state space extremely difficult. Traditional training methods are often inefficient when dealing with high-dimensional state spaces and may even fall into local optimal solutions.

[0007] On the other hand, task requirements in complex scenarios are usually diverse and uncertain. Robot agents need to flexibly switch between different tasks and adjust their strategies according to task requirements. However, traditional training methods often have difficulty handling this diversity and uncertainty, resulting in poor performance of agents when switching tasks.

[0008] In addition, robotic agents in complex scenarios often need to interact with other agents. This interaction may involve a variety of relationships such as competition, cooperation, or coordination. Traditional training methods often only consider the optimization of a single agent and ignore the interaction between agents, which may lead to conflicts and competition between agents and reduce overall performance.

[0009] Therefore, the research on reinforcement learning training of robot agents for complex scenarios has important theoretical significance and practical application value. Summary of the invention

[0010] In view of the above problems, the present invention provides a robot agent reinforcement learning training method and system for complex scenarios to improve the adaptability and flexibility of the robot in complex scenarios.

[0011] To achieve the above object, the present invention provides a robot agent reinforcement learning training method for complex scenarios, which comprises the following steps:

[0012] Step a. Build a scene model:

[0013] According to the robot application scenario, build a complex scene model containing obstacles, dynamic targets and environmental constraints;

[0014] Step b. Build a reinforcement learning algorithm environment:

[0015] Based on the complex scenario model, a reinforcement learning algorithm environment suitable for a multi-agent system is constructed; and the state space, action space, state transition probability and reward function are defined in the environment; wherein the state space includes the position coordinates, speed information, environmental perception information and task progress information of the robot body in the environment; the action space includes movement instructions and operation instructions; the reward function is constructed by combining task completion, collaborative efficiency and resource consumption; the state transition probability represents the probability of the current state being transferred to the predicted state through the trained action strategy;

[0016] Step c. Construct a state transition model:

[0017] A state transition model is constructed according to the state transition probability, time series information, and historical state information, and the vector of the given state, time series information, and historical state information of the agent is used as the input of the state transition model, and the state transition probability and reward information of the predicted state are output;

[0018] Step d. Build a neural network model:

[0019] A reinforcement learning algorithm is used to construct an independent neural network for each robot agent; the neural network includes a decision network and an evaluation network, the decision network is used to generate a predicted action strategy according to the current state, current action, state transition probability and reward information of the robot agent, and the evaluation network is used to evaluate the action value of the action strategy; the action strategy and action value output by the neural network are executed in the state transition model and generate new predicted state and reward information;

[0020] Step e. Training of multi-agent collaborative strategy:

[0021] By adjusting the hyperparameters of the neural network and utilizing a large-scale distributed training architecture to collaboratively train the decision network and the evaluation network, parallel learning and collaborative decision-making of multiple agents can be achieved.

[0022] Preferably, in step a: the obstacles include static obstacles and dynamic obstacles, and the shape, size, and position of the obstacles are collected by a camera, and an obstacle model is constructed; the dynamic target is a target whose motion trajectory and speed are collected in real time by a radar or a camera, and a motion model is used to simulate the motion mode of the dynamic target; the environmental constraints include energy constraints, coverage constraints, obstacle avoidance constraints, coordination constraints, and task constraints, and the environmental constraint data are collected by sensors or manually input and integrated into the scene model.

[0023] Preferably, in the step b:

[0024] In the state space, the position coordinates refer to the specific position coordinates of the robot body in the environment, the speed information includes the movement speed and rotation speed of the robot body, the environmental perception information includes the obstacle position coordinates, the position coordinates of other robot bodies, and the position coordinates of the working area, and the task progress information refers to the progress or stage of the task currently being executed; and the state space also includes timing information and historical state information;

[0025] The movement instructions of the action space include forward, backward, left turn, and right turn, and the operation instructions of the action space include grab, release, and rotation;

[0026] In the reward function, the task completion reward is a corresponding reward given according to the task goal; the collaborative efficiency reward is used to encourage collaborative work between robots; and the resource consumption reward includes the robot's energy consumption factor and time cost factor.

[0027] Preferably, the reward function specifically includes:

[0028] Step b1. Task completion rewards:

[0029] According to the task objectives, the robot is given a positive reward when it successfully completes the task, and a negative reward when it fails to complete the task. The weight of the positive reward is greater than the weight of the negative reward.

[0030] Step b2. Collaborative efficiency reward:

[0031] Collaborative behavior reward: When the robot completes collaborative tasks with other robots, it is given a positive reward; the collaborative tasks include carrying heavy objects together with other robots and avoiding collisions with other robots; if the robot collides with other robots or obstacles, it is given a negative reward; the weight of the negative reward is greater than the weight of the positive reward;

[0032] Step b3. Resource consumption rewards:

[0033] According to the robot's energy consumption, a preset negative energy consumption reward is given; if the task completion time is too long, a negative time cost reward is given; among them, the weight of the negative time cost reward is greater than the weight of the negative energy consumption reward;

[0034] Also includes:

[0035] Step b4. Position and speed rewards:

[0036] A gradually increasing positive target approach reward is given based on the distance between the robot's current position and the target position; a positive speed optimization reward is given if the robot moves with higher efficiency without sacrificing safety; the weight of the target approach reward is greater than the weight of the speed optimization reward;

[0037] Step b5. Environmental perception reward:

[0038] If the robot successfully avoids colliding with an obstacle, it is given a positive obstacle avoidance reward; when the robot enters the designated work area, it is given a positive work area entry reward; the weight of the work area entry reward is greater than the weight of the obstacle avoidance reward;

[0039] Step b6. Timing and historical state rewards:

[0040] If the robot's behavior is temporally coherent and helps complete the task, give positive rewards;

[0041] Furthermore, when a reward of a specified type reaches a preset score or number of times, the reward weight of that reward type is increased.

[0042] Preferably, in step c: the state transfer model is composed of a state transfer probability distribution, and the state transfer probability distribution uses a state transfer matrix in a Markov decision process MDP.

[0043] Preferably, in the step c, constructing the state transition model further comprises the following steps:

[0044] Step c1. According to the state space and action space, collect the possible actions that the agent may take in each state to obtain state-action pairs;

[0045] Step c2. For each state-action pair, calculate the probability of the state transferring to each possible next state after taking the action, and obtain the corresponding state transition probability;

[0046] Step c3. Create a state transfer matrix according to the state-action pair, and fill the state transfer matrix with the state transfer probability to obtain a state transfer model;

[0047] Step c4. The action strategy output during the training of the neural network model is executed in the state transfer model and the new predicted state generated is input into the state transfer model to obtain an updated state transfer model.

[0048] Preferably, in the step d, constructing a neural network model comprises:

[0049] Two independent neural networks are initialized for each robot agent: a decision network and an evaluation network; the input of the decision network is the current state, the current action, the estimate of the state transition probability and the historical record of reward information, and the output is the predicted action strategy; the input of the evaluation network is the state-action pair formed by the current state and the action strategy generated by the decision network, and the output is the action value of the state-action pair, which is the expected cumulative reward;

[0050] A replay buffer is established for storing the interaction experience between the agent and the environment. The storage data of the replay buffer adopts a four-tuple, including: state, action, reward, and next state.

[0051] Preferably, in step e, the multi-agent collaborative training process specifically includes the following steps:

[0052] Step e1. Each agent interacts with the environment to form experience data, and saves the experience data to the playback buffer;

[0053] Step e2. Each agent uses the experience data to learn the state transition model and reward information of the environment, uses the state transition model to generate additional simulation data, and saves the simulation data to the playback buffer;

[0054] Step e3. The empirical data and the simulated data are both used to train the neural network through a reinforcement learning algorithm;

[0055] Step e4. Evaluate the performance of the trained multi-agent system to see whether the agents can collaborate effectively and achieve the optimization goal; and adjust the network structure and parameters of the agents based on the evaluation results.

[0056] Preferably, the hyperparameters of the neural network include: learning rate, batch size, number of iterations, activation function; parallel learning and collaborative decision-making of multiple agents also include:

[0057] Divide a large-scale data set into multiple subsets and process the subsets in parallel on multiple computing nodes, each computing node independently training the decision network and evaluation network of the intelligent agent;

[0058] Use parameter servers to store and synchronously update model parameters so that the model parameters on each computing node remain consistent;

[0059] Share the state, action, and reward information of each agent with other agents through network communication;

[0060] Use multi-objective optimization or game theory methods to coordinate the objective functions and strategy choices of different agents, and gradually converge to the optimal collaborative strategy by iteratively updating the strategy of each agent.

[0061] Preferably, the reinforcement learning algorithm adopts a combination of a policy gradient algorithm and a value decomposition algorithm, wherein the policy gradient algorithm is used for continuous action space and collaborative tasks, and the value decomposition algorithm is used for discrete action space and collaborative tasks.

[0062] In addition, the present invention also provides a robot agent reinforcement learning training system for complex scenes, the system comprising a memory, a processor, and a robot agent reinforcement learning training program for complex scenes stored in the memory and executable on the processor, the robot agent reinforcement learning training program for complex scenes implementing the steps of the robot agent reinforcement learning training method for complex scenes as described in any one of the above items when executed by the processor.

[0063] The beneficial effects of the present invention are:

[0064] (1) By constructing a complex scene model containing obstacles, dynamic targets and environmental constraints, the method of the present invention can simulate diverse environments in the real world, thereby significantly enhancing the adaptability and coping ability of the robot in complex scenes.

[0065] (2) The present invention introduces a state transition model and considers the influence of timing information or historical state in the state transition modeling. It can more accurately predict the future state and reward information of the intelligent agent in a given state based on the timing information and historical state information. This enhances the robot's ability to predict future environmental changes, thereby improving the accuracy and flexibility of its decision-making.

[0066] (3) The present invention constructs an independent neural network for each robot agent, including a decision network and an evaluation network, and trains them through a reinforcement learning algorithm, adjusts the hyperparameters of the neural network, and uses a large-scale distributed training architecture for collaborative training. The method of the present invention realizes parallel learning and collaborative decision-making of multiple agents, helps to improve the collaborative work efficiency of robots in a multi-agent system, and enhances the flexibility and robustness of the overall system.

[0067] (4) The present invention designs a highly robust reward function that can adapt to environmental changes or external factors in dynamic environments, ensuring that the intelligent agent can continuously obtain effective learning signals.

[0068] (5) The present invention improves the system's adaptability and robustness to complex environments through collaborative learning and decision-making among multiple agents.

[0069] (6) The present invention accelerates the training process and reduces the training time through a distributed learning framework and a parallel training method.

[0070] (7) The present invention is applicable to multi-agent systems of different scales and has good flexibility and scalability.

[0071] (8) The present invention constructs a state transition model by introducing the state transition matrix in the Markov decision process (MDP), which brings about the beneficial effects of accurately describing state transition, improving decision accuracy, enhancing learning stability, supporting multi-agent collaboration, and facilitating model optimization and training. These effects jointly improve the adaptability and flexibility of robots in complex scenarios, providing strong support for the widespread application of robots. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the following will be clearly and completely described in combination with the specific technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0073] With the rapid development of robotics technology, Multi-Agent Robot Systems (MARS) have shown great potential in collaborative work, resource allocation, environmental exploration, etc. However, traditional reinforcement learning training methods face many challenges when dealing with multi-agent systems, such as high-dimensional state space, instability, and coordination problems between agents.

[0074] Based on this, the present invention provides a robot agent reinforcement learning training method for complex scenarios, which includes the following steps:

[0075] Step a. Build a scene model:

[0076] According to the robot application scenario, build a complex scene model containing obstacles, dynamic targets and environmental constraints;

[0077] Step b. Build a reinforcement learning algorithm environment:

[0078] Based on the complex scenario model, a reinforcement learning algorithm environment suitable for a multi-agent system is constructed; and the state space, action space, state transition probability and reward function are defined in the environment; wherein the state space includes the position coordinates, speed information, environmental perception information and task progress information of the robot body in the environment; the action space includes movement instructions and operation instructions; the reward function is constructed by combining task completion, collaborative efficiency and resource consumption; the state transition probability represents the probability of the current state being transferred to the predicted state through the trained action strategy;

[0079] Step c. Construct a state transition model:

[0080] A state transition model is constructed according to the state transition probability, time series information, and historical state information, and the vector of the given state, time series information, and historical state information of the agent is used as the input of the state transition model, and the state transition probability and reward information of the predicted state are output;

[0081] Step d. Build a neural network model:

[0082] A reinforcement learning algorithm is used to construct an independent neural network for each robot agent; the neural network includes a decision network and an evaluation network, the decision network is used to generate a predicted action strategy according to the current state, current action, state transition probability and reward information of the robot agent, and the evaluation network is used to evaluate the action value of the action strategy; the action strategy and action value output by the neural network are executed in the state transition model and generate new predicted state and reward information;

[0083] Step e. Training of multi-agent collaborative strategy:

[0084] By adjusting the hyperparameters of the neural network and utilizing a large-scale distributed training architecture to collaboratively train the decision network and the evaluation network, parallel learning and collaborative decision-making of multiple agents can be achieved.

[0085] In this embodiment, in step a: the obstacles include static obstacles and dynamic obstacles, the shape, size, and position of the obstacles are collected by cameras, and an obstacle model is constructed; the dynamic target is to collect the target's motion trajectory and speed in real time by radar or camera, and use the motion model to simulate the motion mode of the dynamic target; the environmental constraints include energy constraints, coverage constraints, obstacle avoidance constraints, coordination constraints, and task constraints, and the environmental constraint data are collected by sensors or manually input and integrated into the scene model. In this embodiment, the construction of the scene model also includes setting the background, participants, and executable actions, as well as constraints such as terrain and climate that affect the freedom and difficulty of action.

[0086] In this embodiment, in the step b:

[0087] In the state space, the position coordinates refer to the specific position coordinates of the robot body in the environment, the speed information includes the movement speed and rotation speed of the robot body, the environmental perception information includes the obstacle position coordinates, the position coordinates of other robot bodies, and the position coordinates of the working area, and the task progress information refers to the progress or stage of the task currently being executed; and the state space also includes timing information and historical state information;

[0088] The movement instructions of the action space include forward, backward, left turn, and right turn, and the operation instructions of the action space include grab, release, and rotation;

[0089] In the reward function, the task completion reward is a corresponding reward given according to the task goal; the collaborative efficiency reward is used to encourage collaborative work between robots; and the resource consumption reward includes the robot's energy consumption factor and time cost factor.

[0090] In this embodiment, the reward function specifically includes:

[0091] Step b1. Task completion rewards:

[0092] According to the task objectives, the robot is given a positive reward when it successfully completes the task, and a negative reward when it fails to complete the task. The weight of the positive reward is greater than the weight of the negative reward.

[0093] Step b2. Collaborative efficiency reward:

[0094] Collaborative behavior reward: When the robot completes collaborative tasks with other robots, it is given a positive reward; the collaborative tasks include carrying heavy objects together with other robots and avoiding collisions with other robots; if the robot collides with other robots or obstacles, it is given a negative reward; the weight of the negative reward is greater than the weight of the positive reward;

[0095] Step b3. Resource consumption rewards:

[0096] According to the robot's energy consumption, a preset negative energy consumption reward is given; if the task completion time is too long, a negative time cost reward is given; among them, the weight of the negative time cost reward is greater than the weight of the negative energy consumption reward;

[0097] Also includes:

[0098] Step b4. Position and speed rewards:

[0099] A gradually increasing positive target approach reward is given based on the distance between the robot's current position and the target position; a positive speed optimization reward is given if the robot moves with higher efficiency without sacrificing safety; the weight of the target approach reward is greater than the weight of the speed optimization reward;

[0100] Step b5. Environmental perception reward:

[0101] If the robot successfully avoids colliding with an obstacle, it is given a positive obstacle avoidance reward; when the robot enters the designated work area, it is given a positive work area entry reward; the weight of the work area entry reward is greater than the weight of the obstacle avoidance reward;

[0102] Step b6. Timing and historical state rewards:

[0103] If the robot's behavior is temporally coherent and helps complete the task, give positive rewards;

[0104] Furthermore, when a reward of a specified type reaches a preset score or number of times, the reward weight of that reward type is increased.

[0105] In this embodiment, in the step c: the state transfer model is composed of a state transfer probability distribution, and the state transfer probability distribution uses a state transfer matrix in a Markov decision process MDP.

[0106] In reinforcement learning, the state transition probability distribution describes the transition of the environment after the agent takes actions in different states. That is, the state transition model is used to predict the probability distribution of the next state under a given state. The input is a given state, which can be added with time series information or a vector of historical states, and the output is the probability distribution of the next state.

[0107] In this embodiment, in the step c, constructing the state transition model further includes the following steps:

[0108] Step c1. According to the state space and action space, collect the possible actions that the agent may take in each state to obtain state-action pairs;

[0109] Step c2. For each state-action pair, calculate the probability of the state transferring to each possible next state after taking the action, and obtain the corresponding state transition probability;

[0110] Step c3. Create a state transfer matrix according to the state-action pair, and fill the state transfer matrix with the state transfer probability to obtain a state transfer model;

[0111] Step c4. The action strategy output during the training of the neural network model is executed in the state transfer model and the new predicted state generated is input into the state transfer model to obtain an updated state transfer model.

[0112] The state transition matrix can accurately describe the transition of the robot after performing different actions in different states. This helps the robot to more accurately predict future states, more accurately evaluate the future value of different actions, and thus select the optimal action.

[0113] Even if the environment changes, the robot can quickly adapt to the new state transition probabilities based on the state transition matrix, thereby maintaining the continuity and stability of learning.

[0114] In a multi-agent system, each agent needs to consider the states and actions of other agents. The state transition matrix can describe the state transition relationship between different agents, realize the collaborative decision-making and parallel learning of multiple agents, and improve the flexibility and efficiency of the overall system.

[0115] In this embodiment, in the step d, constructing a neural network model includes:

[0116] Two independent neural networks are initialized for each robot agent: a decision network and an evaluation network; the input of the decision network is the current state, the current action, the estimate of the state transition probability and the historical record of reward information, and the output is the predicted action strategy; the input of the evaluation network is the state-action pair formed by the current state and the action strategy generated by the decision network, and the output is the action value of the state-action pair, which is the expected cumulative reward;

[0117] A replay buffer is established for storing the interaction experience between the agent and the environment. The storage data of the replay buffer adopts a four-tuple, including: state, action, reward, and next state.

[0118] Among them, the decision network or evaluation network can use a combination of RNN, LSTM, GRU, fully connected neural network, etc.

[0119] In this embodiment, in step e, the multi-agent collaborative training process specifically includes the following steps:

[0120] Step e1. Each agent interacts with the environment to form experience data, and saves the experience data to the playback buffer;

[0121] Step e2. Each agent uses the experience data to learn the state transition model and reward information of the environment, uses the state transition model to generate additional simulation data, and saves the simulation data to the playback buffer;

[0122] Step e3. The empirical data and the simulated data are both used to train the neural network through a reinforcement learning algorithm;

[0123] Step e4. Evaluate the performance of the trained multi-agent system to see whether the agents can collaborate effectively and achieve the optimization goal; and adjust the network structure and parameters of the agents based on the evaluation results.

[0124] In this embodiment, a distributed learning framework is constructed using a modular design concept, and its structure includes:

[0125] (1) The deduction module Actor: It is responsible for interacting with the environment ENV to collect samples. The input is the situation data of the environment, and it performs calculations based on the intelligent algorithm, and finally outputs the decision action. Env represents the internal environment of reinforcement learning and is the entrance to the intelligent algorithm. It mainly obtains situation data from the simulation environment, characterizes the situation data so that it can be used as the input of the intelligent algorithm, and provides the reward information required by the intelligent algorithm based on the situation data and reward function. Finally, the decision information of the intelligent algorithm is input into the simulation environment for real-time deduction. Env is the bridge connecting the reinforcement learning algorithm and the reinforcement learning engine.

[0126] (2) Learning module Learner: Responsible for updating the policy network; the learning module takes out batches of experience data from the experience pool, evaluates the decision-making quality of the intelligent algorithm of the deduction module with the goal of maximizing the reward, and is used to guide the improvement of the decision-making ability of the intelligent algorithm. It inputs batch data, outputs value scores, and updates the network.

[0127] (3) Scheduling management and storage module: responsible for scheduling and managing communications between different nodes, divided into local scheduling, object storage and global control storage.

[0128] Among them, each computing node includes a combination of a deduction module Actor and a learning module Learner; the framework architecture supports multiple GPU communication backends, including MPI, NCCL and GLOO.

[0129] Similarly, the training of a neural network consists of two stages:

[0130] (1) Sampling phase: During the continuous interaction between the Actor and ENV, a large number of samples (reward values, state values, observation values, etc.) are generated. The samples are sent to the global controller asynchronously. This process is called sampling.

[0131] (2) Learning phase: The learner obtains batch samples from the experience pool to calculate gradients and update parameters, and then sends the updated parameters to the sampled actor through the global controller to update the actor's neural network parameters. The sampling process runs on the CPU, and each node has an independent actor and ENV, while the learning process runs on the GPU, using the Horovod method and the Ring-Allreduce algorithm to achieve efficient computing and improve the efficiency of the distributed system. This distributed learning framework supports hybrid resource simultaneous training, which can maximize the use of CPU and GPU resources and improve the gradient update speed.

[0132] During the learning and training process, the agent collects samples from the exploration environment experience and high-quality experience with a preset probability, takes each agent's own state information and environmental information as the input of the neural network, and takes the coordinated actions of multiple agents as the output to complete the training of action strategies. Through the distributed learning framework, collaborative decision-making and strategy optimization among multiple agents are achieved. Through communication and sharing of experience, agents continuously adjust their own strategies to achieve the goal of optimizing overall performance.

[0133] In this embodiment, the hyperparameters of the neural network include: learning rate, batch size, number of iterations, and activation function; wherein:

[0134] The learning rate determines the speed at which the network weights are updated. A smaller learning rate can make the training process more stable, but the convergence speed is slower; a larger learning rate can accelerate convergence, but may cause training instability. In this embodiment, the convergence speed and system stability are balanced by learning rate decay or adaptive learning rate methods (such as Adam optimizer).

[0135] The batch size determines the number of samples used each time the weights are updated. The choice of batch size affects memory usage, training speed, and the accuracy of gradient estimation. A larger batch size can speed up training, but may lead to insufficient memory and inaccurate gradient estimation. A smaller batch size can improve the accuracy of gradient estimation, but the training speed is slower. In this embodiment, a suitable batch size is selected based on hardware resources and data set size.

[0136] Epochs refers to the number of times the entire data set is traversed. Too many iterations may lead to overfitting, while too few iterations may lead to underfitting of the model. In this embodiment, the appropriate number of iterations is determined by early stopping or performance on the validation set.

[0137] Activation functions determine how a neural network processes input and output signals. Different activation functions have different characteristics. For example, the ReLU function is simple to calculate during forward propagation and does not cause gradient disappearance, while the Sigmoid function is suitable for the output layer of a binary classification problem. In this embodiment, a suitable activation function is selected according to the task requirements and model architecture.

[0138] The parallel learning and collaborative decision-making in this embodiment also include:

[0139] Divide a large-scale data set into multiple subsets and process the subsets in parallel on multiple computing nodes, each computing node independently training the decision network and evaluation network of the intelligent agent;

[0140] Use parameter servers to store and synchronously update model parameters so that the model parameters on each computing node remain consistent;

[0141] Share the state, action, and reward information of each agent with other agents through network communication;

[0142] Use multi-objective optimization or game theory methods to coordinate the objective functions and strategy choices of different agents, and gradually converge to the optimal collaborative strategy by iteratively updating the strategy of each agent.

[0143] In this embodiment, the reinforcement learning algorithm adopts a combination of a policy gradient algorithm and a value decomposition algorithm, wherein the policy gradient algorithm is used for continuous action space and collaborative tasks, and the value decomposition algorithm is used for discrete action space and collaborative tasks. The policy gradient algorithm adopts MADDPG or COMA, and the value decomposition algorithm adopts QMIX or VDN, and each algorithm adopts its own loss formula. The details are as follows:

[0144] MADDPG (Multi-Agent Deep Deterministic Policy Gradient) is a multi-agent reinforcement learning algorithm. It is an extension of the deep deterministic policy gradient (DDPG) algorithm. MADDPG is mainly used to solve cooperation and competition problems in multi-agent environments.

[0145] COMA (Counterfactual Multi-Agent Policy Gradients) is a policy gradient-based multi-agent reinforcement learning algorithm that solves the problem of credit assignment (or reward assignment) in a multi-agent environment. The core idea of ​​COMA is to assign an independent reward to each agent to reflect its contribution to the global behavior decision.

[0146] QMIX (Q-value function Decomposition for Cooperative Multi-Agent Learning) is a multi-agent reinforcement learning algorithm based on value decomposition, which focuses on solving cooperative multi-agent tasks. The core idea of ​​QMIX is to decompose the global Q-value function into a nonlinear combination of the local Q-value functions of each agent, so as to learn efficient collaborative strategies.

[0147] VDN (Value Decomposition Networks) is an early multi-agent reinforcement learning algorithm based on value decomposition, which also focuses on solving cooperative multi-agent tasks. The core idea of ​​VDN is to simply add the global Q-value function to the sum of the local Q-value functions of each agent.

[0148] In addition, the reinforcement learning algorithm described in this embodiment also introduces an online learning and strategy update mechanism: in order to solve the problem that traditional offline learning may not be able to adapt to real-time changes in the environment, an online learning mechanism is introduced: the agent obtains data through real-time interaction with the environment; that is, the agent needs to continuously perform actions in the environment, observe changes in the state of the environment, and receive immediate feedback from the environment (such as rewards). Through online learning, the agent can accumulate a large amount of real-time interaction data and update the strategy parameters in time to adapt to new environmental characteristics. In this embodiment, the online reinforcement learning algorithm adopts Q-learning or DQN.

[0149] Finally, the trained neural network is loaded into the actual multi-agent system to achieve collaborative control and autonomous decision-making of multi-agents in complex environments. By evaluating the performance of the trained multi-agents in a simulation environment, it is observed whether the agents can effectively collaborate and achieve the optimization goal; and the network structure and parameters of the agents are adjusted according to the evaluation results; in this embodiment, the parameters such as the network structure, learning rate, and discount factor of the agents are adjusted according to the evaluation results to further improve the performance.

[0150] In the overall training process, the sampling process is run on the CPU, and each node has an independent sampling module and simulation environment, while the training process is run on the GPU, using the Horovod method and the Ring-Allreduce algorithm to achieve efficient computing and improve the efficiency of the distributed system. This distributed learning framework supports hybrid resource simultaneous training, which can maximize the use of CPU and GPU resources and improve the gradient update speed.

[0151] In this embodiment, the trained neural network is exported and loaded into the control module of the robot system, and the neural network model is integrated and coordinated with the perception module, decision-making and planning module, and communication module of the robot system, so as to achieve collaborative control according to environmental changes and task requirements. The deployment method includes executable format and source code form. The executable format refers to exporting the calculation graph of the model into a certain executable format, such as TensorFlow Lite model or TensorFlow.js model, and then executing it on the client device. The source code form refers to exporting the calculation graph of the model into source code in other languages, such as C++ code, and then compiling and executing it on the client.

[0152] In addition, the present invention also provides a robot agent reinforcement learning training system for complex scenes, the system comprising a memory, a processor, and a robot agent reinforcement learning training program for complex scenes stored in the memory and executable on the processor, the robot agent reinforcement learning training program for complex scenes implementing the steps of the robot agent reinforcement learning training method for complex scenes as described in any one of the above items when executed by the processor.

[0153] Among them, the memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as an image playback function, etc.), etc.; the data storage area can store data created according to the use of the system, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory can also include a memory controller to provide the processor and the input unit with access to the memory.

[0154] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0155] Furthermore, in this document, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

[0156] The above description shows and describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the invention, through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not depart from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A robot agent reinforcement learning training method for complex scenarios, characterized in that: The following steps are involved: Step a. Build a scene model: According to the robot application scenario, build a complex scene model containing obstacles, dynamic targets and environmental constraints; Step b. Build a reinforcement learning algorithm environment: Based on the complex scenario model, a reinforcement learning algorithm environment suitable for a multi-agent system is constructed; and the state space, action space, state transition probability and reward function are defined in the environment; wherein the state space includes the position coordinates, speed information, environmental perception information and task progress information of the robot body in the environment; the action space includes movement instructions and operation instructions; the reward function is constructed by combining task completion, collaborative efficiency and resource consumption; the state transition probability represents the probability of the current state being transferred to the predicted state through the trained action strategy; Step c. Construct a state transition model: A state transition model is constructed according to the state transition probability, time series information, and historical state information, and the vector of the given state, time series information, and historical state information of the agent is used as the input of the state transition model, and the state transition probability and reward information of the predicted state are output; Step d. Build a neural network model: A reinforcement learning algorithm is used to construct an independent neural network for each robot agent; the neural network includes a decision network and an evaluation network, the decision network is used to generate a predicted action strategy according to the current state, current action, state transition probability and reward information of the robot agent, and the evaluation network is used to evaluate the action value of the action strategy; the action strategy and action value output by the neural network are executed in the state transition model and generate new predicted state and reward information; Step e. Training of multi-agent collaborative strategy: By adjusting the hyperparameters of the neural network, the decision network and the evaluation network are collaboratively trained using a large-scale distributed training architecture to achieve parallel learning and collaborative decision-making of multiple agents; In the step d, constructing a neural network model includes: Two independent neural networks are initialized for each robot agent: a decision network and an evaluation network; the input of the decision network is the current state, the current action, the estimate of the state transition probability and the historical record of reward information, and the output is the predicted action strategy; the input of the evaluation network is the state-action pair formed by the current state and the action strategy generated by the decision network, and the output is the action value of the state-action pair, which is the expected cumulative reward; A replay buffer is established for storing the interaction experience between the agent and the environment. The storage data of the replay buffer adopts a four-tuple, including: state, action, reward, and next state.

2. The method according to claim 1, characterized in that In the step a, the obstacles include static obstacles and dynamic obstacles, and the shape, size, and position of the obstacles are collected by a camera, and an obstacle model is constructed; the dynamic target is a target whose motion trajectory and speed are collected in real time by a radar or a camera, and a motion model is used to simulate the motion mode of the dynamic target; the environmental constraints include energy constraints, coverage constraints, obstacle avoidance constraints, coordination constraints, and task constraints, and the environmental constraint data are collected by sensors or manually input and integrated into the scene model.

3. The method according to claim 1, characterized in that In the step b: In the state space, the position coordinates refer to the specific position coordinates of the robot body in the environment, the speed information includes the movement speed and rotation speed of the robot body, the environmental perception information includes the obstacle position coordinates, the position coordinates of other robot bodies, and the position coordinates of the working area, and the task progress information refers to the progress or stage of the task currently being executed; and the state space also includes timing information and historical state information; The movement instructions of the action space include forward, backward, left turn, and right turn, and the operation instructions of the action space include grab, release, and rotation; In the reward function, the task completion reward is a corresponding reward given according to the task goal; the collaborative efficiency reward is used to encourage collaborative work between robots; and the resource consumption reward includes the robot's energy consumption factor and time cost factor.

4. The method according to claim 3, characterized in that The reward function specifically includes: Step b1. Task completion rewards: According to the task objectives, the robot is given a positive reward when it successfully completes the task, and a negative reward when it fails to complete the task. The weight of the positive reward is greater than the weight of the negative reward. Step b2. Collaborative efficiency reward: Collaborative behavior reward: When the robot completes collaborative tasks with other robots, it is given a positive reward; the collaborative tasks include carrying heavy objects together with other robots and avoiding collisions with other robots; if the robot collides with other robots or obstacles, it is given a negative reward; the weight of the negative reward is greater than the weight of the positive reward; Step b3. Resource consumption rewards: According to the robot's energy consumption, a preset negative energy consumption reward is given; if the task completion time is too long, a negative time cost reward is given; among them, the weight of the negative time cost reward is greater than the weight of the negative energy consumption reward; Also includes: Step b4. Position and speed rewards: A gradually increasing positive target approach reward is given based on the distance between the robot's current position and the target position; a positive speed optimization reward is given if the robot moves with higher efficiency without sacrificing safety; the weight of the target approach reward is greater than the weight of the speed optimization reward; Step b5. Environmental perception reward: If the robot successfully avoids colliding with an obstacle, it is given a positive obstacle avoidance reward; when the robot enters the designated work area, it is given a positive work area entry reward; the weight of the work area entry reward is greater than the weight of the obstacle avoidance reward; Step b6. Timing and historical state rewards: If the robot's behavior is temporally coherent and helps complete the task, give positive rewards; Furthermore, when a reward of a specified type reaches a preset score or number of times, the reward weight of that reward type is increased.

5. The method according to claim 1, characterized in that In the step c, the state transfer model is composed of a state transfer probability distribution, and the state transfer probability distribution uses a state transfer matrix in a Markov decision process MDP; constructing the state transfer model further includes the following steps: Step c1. According to the state space and action space, collect the possible actions that the agent may take in each state to obtain state-action pairs; Step c2. For each state-action pair, calculate the probability of the state transferring to each possible next state after taking the action, and obtain the corresponding state transition probability; Step c3. Create a state transfer matrix according to the state-action pair, and fill the state transfer matrix with the state transfer probability to obtain a state transfer model; Step c4. The action strategy output during the training of the neural network model is executed in the state transfer model and the new predicted state generated is input into the state transfer model to obtain an updated state transfer model.

6. The method according to claim 1, characterized in that In step e, the multi-agent collaborative training process specifically includes the following steps: Step e1. Each agent interacts with the environment to form experience data, and saves the experience data to the playback buffer; Step e2. Each agent uses the experience data to learn the state transition model and reward information of the environment, uses the state transition model to generate additional simulation data, and saves the simulation data to the playback buffer; Step e3. The empirical data and the simulated data are both used to train the neural network through a reinforcement learning algorithm; Step e4. Evaluate the performance of the trained multi-agent system to see whether the agents can collaborate effectively and achieve the optimization goal; and adjust the network structure and parameters of the agents based on the evaluation results.

7. The method according to claim 1, characterized in that The hyperparameters of the neural network include: learning rate, batch size, number of iterations, activation function; the parallel learning and collaborative decision-making of multiple agents also include: Divide a large-scale data set into multiple subsets and process the subsets in parallel on multiple computing nodes, each computing node independently training the decision network and evaluation network of the intelligent agent; Use parameter servers to store and synchronously update model parameters so that the model parameters on each computing node remain consistent; Share the state, action, and reward information of each agent with other agents through network communication; Use multi-objective optimization or game theory methods to coordinate the objective functions and strategy choices of different agents, and gradually converge to the optimal collaborative strategy by iteratively updating the strategy of each agent.

8. The method according to claim 1, characterized in that The reinforcement learning algorithm adopts a combination of a policy gradient algorithm and a value decomposition algorithm, wherein the policy gradient algorithm is used for continuous action space and collaborative tasks, and the value decomposition algorithm is used for discrete action space and collaborative tasks.

9. A robot agent reinforcement learning training system for complex scenarios, characterized in that: The system includes a memory, a processor, and a robot agent reinforcement learning training program for complex scenes stored in the memory and executable on the processor. When the robot agent reinforcement learning training program for complex scenes is executed by the processor, the steps of the robot agent reinforcement learning training method for complex scenes as described in any one of claims 1 to 8 are implemented.