Robot control method, device and equipment based on multi-agent reinforcement learning

Abstract

The invention provides a robot control method, device and equipment based on multi-agent reinforcement learning. The method comprises the following steps: adaptively resetting strategy parameters of an initial neural network of each robot in a robot group cooperation system according to a preset rule; inputting the sample observation data of each robot at the current moment and the sample action of each robot at the previous moment into an initial neural network of each robot to obtain a local prediction return value of each robot; calculating a global prediction return value according to the local prediction return value of each robot, and fusing the parameters of the initial neural network of each robot; updating the experience pool of each robot, collecting sample data from the experience pool of each robot, and periodically repeating the training process of each robot until a preset convergence condition is reached; and controlling each robot to complete a cooperation task based on the trained neural network of each robot. The diversity and breadth of exploration behaviors can be improved, and the exploration efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a robot control method, device and equipment based on multi-agent reinforcement learning. Background Art

[0002] In the research and application of multi-agent reinforcement learning, agents need to interact and learn in a shared environment to solve complex decision-making problems. These agents optimize their strategies by interacting with the environment to achieve common or individual goals. Existing multi-agent reinforcement learning methods train multi-agents through random exploration, which often converges quickly during the learning process, resulting in insufficient diversity and breadth of exploration behaviors and insufficient exploration efficiency. In addition, existing methods select training samples in a random manner, which has low sample efficiency and is difficult to adapt to dynamically changing complex application scenarios. Summary of the invention

[0003] The present invention provides a robot control method, device and equipment based on multi-agent reinforcement learning, which is used to solve the problem of insufficient exploration efficiency in the multi-agent reinforcement learning method in the prior art that multi-agents are trained by random exploration.

[0004] The present invention provides a robot control method based on multi-agent reinforcement learning, comprising: Adaptively reset the strategy parameters of the initial neural network of each robot in the robot swarm collaboration system according to preset rules; Inputting the sample observation data of each robot at the current moment and the sample action of each robot at the previous moment into the initial neural network of each robot, and obtaining the local predicted reward value of each robot output by the initial neural network of each robot; Calculating the global predicted reward value of the robot group collaboration system according to the local predicted reward value of each robot, and fusing the parameters of the initial neural network of each robot; Updating the experience pool of each robot, collecting sample data from the experience pool of each robot, and periodically repeating the training process of each robot until a preset convergence condition is reached to obtain a neural network of each robot; Based on the neural network of each robot, each robot in the actual environment is controlled to complete the collaborative task.

[0005] In some embodiments, updating the experience pool of each robot and collecting sample data from the experience pool of each robot includes: Storing the multiple trajectories of each robot in each training cycle into the experience pool of each robot, wherein the experience pool of each robot pre-stores multiple historical trajectories of each robot; A balanced experience sampling strategy is adopted to collect sample trajectories with large differences from reference trajectories from the experience pool of each robot to obtain sample data, wherein the sample data includes sample observation data and sample actions.

[0006] In some embodiments, the balanced experience sampling strategy is used to collect sample trajectories that are significantly different from the reference trajectory from the experience pool of each robot, including: A reference sample trajectory and multiple sample trajectories are randomly selected from the experience pool of each robot: Performing feature extraction on the reference sample trajectory to obtain a feature representation of the reference sample trajectory, and performing feature extraction on the multiple sample trajectories to obtain feature representations of the multiple sample trajectories; Calculating the cosine similarity between the feature representation of each sample trajectory and the feature representation of the reference sample trajectory to obtain the similarity between each sample trajectory and the reference sample trajectory; According to the similarity between each sample trajectory and the reference sample trajectory, a sample trajectory having a greater difference from the reference trajectory is determined from the multiple sample trajectories.

[0007] In some embodiments, the adaptively resetting the strategy parameters of the initial neural network of each robot in the robot swarm collaboration system according to the preset rules includes: Taking each robot as an independent intelligent agent, constructing an initial neural network of each robot; Based on the adaptive strategy reset mechanism, part of the strategy parameters of the initial neural network of each robot are periodically discarded.

[0008] In some embodiments, the calculating the global predicted reward value of the robot group collaboration system according to the local predicted reward value of each robot includes: Acquire the current global state of the robot swarm collaboration system, wherein the current global state includes the environment state of the robot swarm collaboration system, the state of each robot, the interaction information between the robots, and the collaboration task information of the robot swarm collaboration system; Inputting the local predicted reward value of each robot and the current global state of the robot swarm collaborative system into a pre-constructed hybrid network, and calculating a first global predicted reward value of the robot swarm collaborative system according to the local predicted reward value of each robot and the current global state of the robot swarm collaborative system; Correcting the first global prediction reward value to obtain the global prediction reward value; Based on the global predicted reward value and the preset target global reward value, a loss function value is calculated, and according to the loss function value, the parameters of the hybrid network are optimized.

[0009] In some embodiments, the modifying the first global prediction reward value to obtain the global prediction reward value includes: Summing the local predicted reward value of each robot to obtain a second global predicted reward value of the robot group collaboration system; Based on the second global predicted reward value, the first global predicted reward value is corrected to obtain the global predicted reward value.

[0010] In some embodiments, the parameters of the initial neural network of each robot are integrated, including: Determine the weight of the parameters of the initial neural network of each robot according to the contribution of the local predicted reward value of each robot to the global predicted reward value and the difference between the global predicted reward value and a preset target global reward value; Calculating the fusion parameters of each robot according to the parameters of the initial neural network of each robot and the weights of the parameters of the initial neural network of each robot; According to the fusion parameters of each robot, the parameters of the initial neural network of each robot are updated.

[0011] In some embodiments, inputting the sample observation data of each robot at the current moment and the sample action of each robot at the previous moment into the initial neural network of each robot to obtain the local predicted reward value of each robot output by the initial neural network of each robot includes: Acquire sample observation data of each robot at the current moment, wherein the sample observation data includes a sample state of each robot, sample environment information within the observation range of each robot, and sample pose information of a sample object to be processed; The sample observation data of each robot at the current moment and the sample action of each robot at the previous moment are input into the initial neural network of each robot, and the initial neural network of each robot performs feature extraction on the sample observation data of each robot at the current moment and the sample action of each robot at the previous moment to obtain the sample feature representation of each robot, and the predicted action of each robot at the current moment is obtained based on the sample feature representation of each robot, and the local predicted reward value of each robot is calculated based on the reward obtained after each robot takes the predicted action at the current moment.

[0012] In some embodiments, the controlling of a group of robots in an actual environment to complete a collaborative task based on the neural network of each robot includes: Obtaining the observation data of each robot at the current moment and the action at the previous moment; Inputting the observation data of each robot at the current moment and the action of each robot at the previous moment into the neural network of each robot to obtain the action of each robot at the current moment; Based on the current action of each robot, generate a control instruction for each robot; Based on the control instructions of each robot, each robot is controlled to grasp the corresponding target object to complete the collaborative task.

[0013] The present invention also provides a robot control device based on multi-agent reinforcement learning, comprising: A parameter resetting unit, used for adaptively resetting the strategy parameters of the initial neural network of each robot in the robot group collaboration system according to preset rules; A first training unit is used to input the sample observation data of each robot at the current moment and the sample action of each robot at the previous moment in the robot group cooperation system into the initial neural network of each robot, and obtain the local predicted reward value of each robot output by the initial neural network of each robot; A second training unit, configured to calculate a global predicted reward value of the robot group collaboration system according to the local predicted reward value of each robot, and fuse the parameters of the initial neural network of each robot; A third training unit is used to update the experience pool of each robot, collect sample data from the experience pool of each robot, and periodically repeat the training process of each robot until a preset convergence condition is reached to obtain a neural network of each robot; A control unit is used to control each robot in the actual environment to complete a collaborative task based on the neural network of each robot.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a robot control method based on multi-agent reinforcement learning as described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a robot control method based on multi-agent reinforcement learning as described in any one of the above.

[0016] The robot control method, device and equipment based on multi-agent reinforcement learning provided by the present invention adaptively resets the strategy parameters of the initial neural network of each robot in the robot group collaboration system; inputs the sample observation data of each robot at the current moment and the sample action of the previous moment into the initial neural network of each robot to obtain the local predicted reward value of each robot; calculates the global predicted reward value based on the local predicted reward value of each robot, and integrates the parameters of the initial neural network of each robot; updates the experience pool of each robot, collects sample data from the experience pool of each robot, and periodically repeats the training process of each robot until the preset convergence condition is reached; controls each robot to complete the collaborative task based on the trained neural network of each robot; improves the diversity and breadth of exploration behavior, improves the exploration efficiency, and can adapt to dynamically changing complex application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of a robot control method based on multi-agent reinforcement learning provided in an embodiment of the present invention.

[0019] Figure 2 It is one of the flow charts of multi-agent reinforcement learning provided by an embodiment of the present invention.

[0020] Figure 3 This is the second flowchart of multi-agent reinforcement learning provided by an embodiment of the present invention.

[0021] Figure 4 It is a structural schematic diagram of a robot control device based on multi-agent reinforcement learning provided by an embodiment of the present invention.

[0022] Figure 5 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0025] At present, in the research of multi-agent reinforcement learning, improving exploration efficiency and sample efficiency are two core challenges. Existing technical solutions mainly revolve around these two points. In terms of exploration efficiency, researchers have proposed methods based on intrinsic rewards, such as random network distillation, which measures the novelty of states through randomly initialized networks to motivate agents to explore unknown areas. In addition, there are model-based methods that use predictive models to guide exploration, as well as mutual information-based methods that promote exploration by maximizing the mutual information between agent behavior and environmental states.

[0026] In terms of sample efficiency, researchers have developed a variety of strategies to improve the effect of learning from experience replay, such as prioritized experience replay. There are also methods that reset parameters, such as multi-agent reinforcement learning algorithms, which regularly reset the parameters of the agent to prevent overfitting and promote new exploration behaviors. In addition, methods based on representation learning, such as self-prediction representation, improve learning efficiency by predicting the potential state representation of the agent.

[0027] Although these methods have achieved certain success in specific environments, they still face the challenges of insufficient scalability and adaptability when facing large-scale, complex and dynamically changing multi-agent environments. Existing multi-agent reinforcement learning methods have the following defects: (1) Insufficient exploration efficiency: Existing multi-agent reinforcement learning algorithms have certain limitations. The strategies of the agents often converge quickly during the learning process, resulting in insufficient diversity and breadth of exploration behaviors. This is particularly evident when the environment state, observation, and action space are large, and it is difficult for the agent to cover all possible strategy spaces through random exploration. In addition, the strategy generator based on deep neural networks can easily cause the agent to overfit to early experience, thus falling into a local optimal solution in later exploration.

[0028] (2) Low sample efficiency: Mainstream multi-agent reinforcement learning methods often suffer from low sample efficiency when learning from the experience replay buffer. This is because random sampling causes a large number of unimportant or redundant samples to be used for training, which reduces learning efficiency. In addition, these methods often fail to consider the correlation and information content between samples when selecting samples for training, resulting in a slow learning process and prone to overfitting.

[0029] (3) Limited adaptability and scalability: Existing multi-agent reinforcement learning methods often lack sufficient adaptability and scalability when dealing with dynamic changes in complex environments. Especially in scenarios of multi-agent collaboration and competition, agents need to quickly adapt to environmental changes and effectively use experience to learn. However, existing methods often require a large amount of environmental interaction when dealing with large-scale state-action spaces, which not only increases the computational cost but also limits the feasibility of the algorithm in practical applications.

[0030] To this end, the present invention provides a robot control method based on multi-agent reinforcement learning, which improves exploration efficiency and sample efficiency through adaptive strategy resetting and balanced experience sampling based on state representation; through parameter fusion and selective experience replay between agents, the diversity of strategies and the representativeness of samples are enhanced, thereby enabling more effective strategy learning and faster convergence in complex multi-agent environments.

[0031] Figure 1 The following is a flow chart of a robot control method based on multi-agent reinforcement learning provided by an embodiment of the present invention. Figure 1 As shown, a robot control method based on multi-agent reinforcement learning is provided, comprising the following steps: step 110, step 120, step 130, step 140 and step 150. The steps of the method flow are only used as a possible implementation mode of the present invention.

[0032] Step 110: adaptively reset the strategy parameters of the initial neural network of each robot in the robot swarm collaboration system according to preset rules.

[0033] Optionally, the preset rule includes at least one of the following rules: Reset strategy parameters based on the robot’s current environment information (such as location, obstacle distribution, lighting conditions, etc.); Reset strategy parameters according to the type of task the robot currently needs to complete (such as carrying, patrolling, searching, etc.); Reset strategy parameters based on the robot's historical performance data (such as time to complete a task, energy consumption, error rate, etc.).

[0034] Optionally, the strategy parameters of the initial neural network of each robot are adaptively reset periodically, such as resetting the parameters after every n rounds of training are completed.

[0035] Optionally, the strategy parameters of the initial neural network of some robots in the robot swarm collaboration system are selectively adaptively reset.

[0036] In some embodiments, adaptively resetting the strategy parameters of the initial neural network of each robot in the robot swarm collaboration system according to preset rules includes: Take each robot as an independent intelligent agent and construct the initial neural network of each robot; Based on the adaptive strategy reset mechanism, some strategy parameters of the initial neural network of each robot are periodically discarded.

[0037] Among them, the initial neural network includes an initial strategy network and an initial Q value network.

[0038] Among them, the Adaptive Policy Resetting Mechanism (APRM) mechanism is used to dynamically adjust the robot's policy parameters to adapt to environmental changes and task requirements. APRM encourages agents to escape from local optimal solutions and explore new policy spaces by simulating the "forgetting" and "relearning" processes in biological systems. The key technical points of APRM include the strategy for parameter resetting, the choice of resetting timing, and the parameter fusion method, which together constitute a comprehensive solution to prevent policy rigidity and encourage the exploration of new policies.

[0039] It should be noted that the parameter reset of APRM is partial, involving only certain layers of the agent's neural network, while retaining other layers to maintain certain previous learning experience. This partial reset strategy both promotes new exploration behaviors and retains valuable historical information.

[0040] It can be understood that based on the adaptive strategy reset mechanism, periodically discarding some strategy parameters of the initial neural network of each robot can prevent the strategy from converging prematurely and ensure that the robot swarm collaboration system can operate efficiently, stably and safely in a complex and changing environment.

[0041] Step 120: Input the sample observation data of each robot at the current moment and the sample action of each robot at the previous moment into the initial neural network of each robot, and obtain the local predicted reward value of each robot output by the initial neural network of each robot.

[0042] Optionally, a Q-value function of each robot is constructed based on the trajectory of each robot from the starting moment to the current moment and the action of each robot at the current moment, and a local predicted Q-value of each robot is calculated based on the Q-value function of each robot.

[0043] Optionally, the sample observation data includes at least a sample state of each robot, sample environment information within the observation range of each robot, and sample position and posture information of a sample object to be processed.

[0044] In some embodiments, the sample observation data of each robot at the current moment and the sample action of each robot at the previous moment are input into the initial neural network of each robot to obtain the local predicted reward value of each robot output by the initial neural network of each robot, including: Obtaining the sample observation data of each robot at the current moment, the sample observation data including the sample state of each robot, the sample environment information within the observation range of each robot, and the sample pose information of the sample object to be processed; The sample observation data of each robot at the current moment and the sample action of the previous moment are input into the initial neural network of each robot. The initial neural network of each robot performs feature extraction on the sample observation data of each robot at the current moment and the sample action of the previous moment to obtain the sample feature representation of each robot. According to the sample feature representation of each robot, the predicted action of each robot at the current moment is obtained. According to the reward obtained after each robot takes the predicted action at the current moment, the local predicted reward value of each robot is calculated.

[0045] Optionally, the sample state of each robot includes at least the position, posture and motion state of each robot; the sample environmental information within the observation range of each robot includes at least the posture information of other robots within the observation range, obstacle information, ground information, temperature, humidity and light intensity; the sample posture information of the sample object to be processed includes at least the position, posture, size, shape and surface characteristics of the sample object.

[0046] Optionally, based on the Q-value function of each robot and combined with the reward obtained after each robot takes the predicted action at the current moment, the local predicted Q-value of each robot is calculated.

[0047] It can be understood that the initial neural network of each robot determines the predicted action at the current moment based on the sample observation data of each robot at the current moment and the sample action at the previous moment, so as to better understand the changes in the surrounding environment, thereby improving the efficiency, accuracy and adaptability of action prediction.

[0048] Step 130: Calculate the global predicted reward value of the robot-crowd collaboration system based on the local predicted reward value of each robot, and integrate the parameters of the initial neural network of each robot.

[0049] Optionally, based on the local predicted Q value of each robot, the global predicted Q value of the robot crowd collaboration system is calculated, the parameters of the initial neural network of each robot are integrated, and the parameters of the initial neural network of each robot are updated.

[0050] Optionally, a mixing factor of each robot is determined, and during the parameter fusion process, the weights of the parameters of the initial neural network of each robot are controlled based on the mixing factor of each robot.

[0051] Optionally, a multi-agent reinforcement learning algorithm is used to calculate the global predicted reward value of the machine crowd collaboration system, such as the Q-value Mixing (QMIX) algorithm and the Value-Decomposition Networks (VDN) algorithm.

[0052] Among them, the QMIX algorithm aims to find a completely decentralized strategy while maintaining the consistency of the strategy. The QMIX algorithm uses a hybrid network to combine the local Q-value functions of each agent to obtain a global Q-value function.

[0053] Among them, the intelligent agent trained by QMIX not only continuously updates its own strategy through interaction data, but also constantly draws on the experience of other intelligent agents to enhance its exploration of the environment, thereby obtaining more and richer interaction data to train the intelligent agent. To a certain extent, the intelligent agent reset mechanism can interact with the environment to obtain more interaction data, while the representation-based sampling method makes full use of the existing data for strategy optimization. The two are interrelated to ultimately achieve better results.

[0054] Among them, the VDN algorithm achieves efficient strategy learning in a collaborative environment by decomposing the global Q-value function into the sum of multiple local Q-value functions.

[0055] Step 140: update the experience pool of each robot, collect sample data from the experience pool of each robot, and periodically repeat the training process of each robot until a preset convergence condition is reached to obtain the neural network of each robot.

[0056] Optionally, previous experience information is pre-stored in the experience pool of each robot.

[0057] Optionally, the sample observation data, predicted action, local predicted reward value, global predicted reward value and other data of each robot at the current moment are added to the experience pool of each robot.

[0058] Optionally, a sampling method based on state representation (such as a state representation method based on dynamics and mutual simulation) is used to collect sample data from the experience pool of each robot.

[0059] Optionally, the adaptive policy reset mechanism is combined with a state representation-based sampling method to periodically repeat the training process for each robot.

[0060] It can be understood that by adopting a state representation-based sampling method for sampling, learning the dynamic characteristics of the environment and the similarities between states, the agent can have a deeper understanding of the environment, thereby being able to more effectively select informative experiences, improving sample efficiency and the quality of the agent's decision-making.

[0061] Step 150: Control each robot in the actual environment to complete the collaborative task based on the neural network of each robot.

[0062] Optionally, the action of each robot at the current moment is predicted based on the neural network of each robot, and each robot is controlled to take corresponding actions to complete the collaborative task.

[0063] In some embodiments, a group of robots in an actual environment is controlled based on a neural network of each robot to complete a collaborative task, including: Obtain the observation data of each robot at the current moment and the action at the previous moment; Input the observation data of each robot at the current moment and the action of the previous moment into the neural network of each robot to obtain the action of each robot at the current moment; Generate control instructions for each robot based on the current action of each robot; Based on the control instructions of each robot, each robot is controlled to grab the corresponding target object to complete the collaborative task.

[0064] It can be understood that by controlling each robot in the actual environment to complete collaborative tasks based on the neural network of each robot, the robots can make more reasonable decisions when faced with multiple choices, perform tasks more accurately, reduce errors and duplication of work, and improve the collaborative ability of multiple robots, as well as the efficiency and effectiveness of task execution.

[0065] Figure 2 One of the flowcharts of multi-agent reinforcement learning provided in an embodiment of the present invention. Figure 2 (a) is a schematic diagram of the process of discarding agent parameters, which selectively discards some strategy parameters of the initial neural network of each agent; Figure 2(b) is a flow chart of agent parameter fusion, where agent represents the agent. The QMIX algorithm is used as the main algorithm and the VDN algorithm is used as the auxiliary algorithm to calculate the global Q value of each agent based on the local Q value of each agent.

[0066] It should be noted that the observation data of each agent at time t and the action at time t-1 are input into each agent, and the trajectory of each agent at time 0~t and the action at time t are output by each agent. The local Q value of each agent is calculated based on the trajectory of each agent at time 0~t and the action at time t.

[0067] In an embodiment of the present invention, the strategy parameters of the initial neural network of each robot in the robot swarm collaborative system are adaptively reset; the sample observation data of each robot at the current moment and the sample action of the previous moment are input into the initial neural network of each robot to obtain the local predicted reward value of each robot; the global predicted reward value is calculated based on the local predicted reward value of each robot, and the parameters of the initial neural network of each robot are integrated; the experience pool of each robot is updated, sample data is collected from the experience pool of each robot, and the training process of each robot is periodically repeated until the preset convergence condition is reached; each robot is controlled to complete the collaborative task based on the trained neural network of each robot; the diversity and breadth of the exploration behavior are improved, the exploration efficiency is improved, and it can adapt to dynamically changing complex application scenarios, so that the robot swarm can complete the collaborative task efficiently and with high quality.

[0068] In some embodiments, updating the experience pool of each robot and collecting sample data from the experience pool of each robot include: Storing multiple trajectories of each robot in each training cycle into an experience pool of each robot, wherein the experience pool of each robot pre-stores multiple historical trajectories of each robot; A balanced experience sampling strategy is adopted to collect sample trajectories that are significantly different from the reference trajectory from the experience pool of each robot to obtain sample data, which includes sample observation data and sample actions.

[0069] Optionally, a Survey Response Based Experience Sampling (SRBES) strategy is adopted to collect sample trajectories that are significantly different from the reference trajectory from the experience pool of each robot to obtain corresponding sample data.

[0070] Among them, SRBES is a method for dealing with the problem of category imbalance in machine learning. It aims to balance the number of samples in each category by adjusting the distribution of training data, thereby improving the agent's recognition ability and overall performance of minority categories.

[0071] It should be noted that SRBES improves sample efficiency by selecting the most informative experience samples through a compact representation of the learning state. While improving sample efficiency, SRBES also reduces the time complexity during experience selection. By selecting sample data for training through similarity comparison, the intelligent agent can learn from more experience in a shorter time, thereby improving overall learning efficiency.

[0072] It can be understood that by adopting a balanced experience sampling strategy and selecting the most informative and diverse experience samples to improve sample efficiency, the speed of reinforcement learning can be accelerated and the quality of strategy can be improved; by combining the adaptive policy reset mechanism APRM and balanced experience sampling SRBES, the robustness, flexibility, adaptability and scalability of the agent in a complex multi-agent environment can be improved, enabling the agent to learn and adapt to new strategies faster while reducing dependence on environmental interactions, allowing the agent to learn effectively in a large-scale, dynamically changing environment.

[0073] In some embodiments, a balanced experience sampling strategy is adopted to collect sample trajectories with large differences from the reference trajectory from the experience pool of each robot, including: Randomly select a reference sample trajectory and multiple sample trajectories from the experience pool of each robot: Extracting features from a reference sample trajectory to obtain a feature representation of the reference sample trajectory, and extracting features from multiple sample trajectories to obtain feature representations of multiple sample trajectories; Calculate the cosine similarity between the feature representation of each sample trajectory and the feature representation of the reference sample trajectory to obtain the similarity between each sample trajectory and the reference sample trajectory; According to the similarity between each sample trajectory and the reference sample trajectory, a sample trajectory having a greater difference from the reference trajectory is determined from the multiple sample trajectories.

[0074] Optionally, the calculation formula of the cosine similarity between the feature representation of the sample trajectory and the feature representation of the reference sample trajectory is as follows: ; ; in, represents the cosine similarity between the feature representation of the sample trajectory and the feature representation of the reference sample trajectory, Represents the feature representation of the sample trajectory, represents the feature representation of the reference sample trajectory, express The feature representation of the sample trajectory at time instant, express Feature representation of the sample trajectory at time instant.

[0075] It can be understood that by adopting a balanced experience sampling strategy, sample trajectories that differ greatly from the reference trajectory are collected from the experience pool of each robot. While improving the sample efficiency, it also reduces the time complexity during experience selection, enabling the agent to learn from more experience in a shorter time, thereby improving the overall learning efficiency.

[0076] Figure 3 The second flow chart of multi-agent reinforcement learning provided by the embodiment of the present invention. Figure 3 As shown, the experience replay buffer (Replay Buffer) stores multiple sample sequence data, such as Episode 1, Episode 2, Episode 3, ..., Episode K. The cosine similarity between the sample sequence data and the reference sample sequence data is calculated, and Episode 3 is selected for multi-agent training based on the cosine similarity.

[0077] The sample sequence data includes sample state (s) and sample action (a).

[0078] Specifically, based on the dynamic model combined with the state encoder, action encoder and state decoder, feature extraction is performed on multiple sample sequence data to obtain feature representation of the multiple sample sequence data. Based on the feature representation of the multiple sample sequence data, the cosine similarity of any two sample sequence data is calculated, and the target sample sequence data is selected according to the cosine similarity. Based on the target sample sequence data, the QMIX algorithm and the VDN algorithm are used to train different intelligent agents.

[0079] It can be understood that by extracting the feature representation of each sample trajectory and the feature representation of the reference sample trajectory, calculating the similarity between each sample trajectory and the reference sample trajectory, and selecting sample data based on the similarity, it is possible to ensure that the agent can learn from a wide range of dynamic environments and improve the generalization ability of the strategy.

[0080] In some embodiments, calculating the global predicted reward value of the robot crowd collaboration system based on the local predicted reward value of each robot includes: Obtaining the current global state of the robot swarm collaboration system, which includes the environment state of the robot swarm collaboration system, the state of each robot, the interaction information between the robots, and the collaboration task information of the robot swarm collaboration system; Inputting the local predicted reward value of each robot and the current global state of the robot swarm collaboration system into a pre-constructed hybrid network, and calculating a first global predicted reward value of the robot swarm collaboration system according to the local predicted reward value of each robot and the current global state of the robot swarm collaboration system; Correcting the first global prediction reward value to obtain a global prediction reward value; Based on the global predicted reward value and the preset target global reward value, the loss function value is calculated, and the parameters of the hybrid network are optimized according to the loss function value.

[0081] Among them, the environmental state of the robot swarm collaboration system refers to the current condition of the working environment of the robot swarm. The environmental state of the robot swarm collaboration system includes but is not limited to spatial layout, obstacle location, light conditions, temperature, humidity and other factors that may affect the robot's action and perception.

[0082] Among them, the status of each robot includes position, speed, acceleration, battery power, sensor data, etc.; the interaction information between robots includes the quality of the communication link, the content of the transmitted messages, shared sensor data, intentions and plans, etc.; the interaction information is crucial to ensure that the robot group can work together, avoid conflicts and improve efficiency.

[0083] Among them, the collaborative task information of the robot swarm collaboration system includes task objectives, task progress, role allocation, resource requirements, time constraints, etc. The collaborative task information is the core of the robot swarm collaboration and is used to guide the robot swarm to work together to achieve the overall goal.

[0084] Optionally, the first global predicted reward value is modified based on additional factors, such as risk assessment, resource constraints, task priority changes, etc.

[0085] The hybrid network is trained based on the local sample reward value of each robot, the sample global state of the robot swarm collaboration system, and the first global reward value label of the robot swarm collaboration system.

[0086] It can be understood that by inputting the local predicted reward value of each robot and the current global state of the robot swarm collaboration system into a pre-constructed hybrid network, the first global predicted reward value of the robot swarm collaboration system is calculated according to the local predicted reward value of each robot and the current global state of the robot swarm collaboration system; the first global predicted reward value is corrected to obtain the global predicted reward value, which improves the accuracy of the global predicted reward value, thereby enabling the robot to make decisions more intelligently, improve the efficiency and success rate of task execution, and reduce potential risks and conflicts.

[0087] In some embodiments, the first global prediction reward value is modified to obtain the global prediction reward value, including: The local predicted reward values ​​of each robot are summed to obtain the second global predicted reward value of the robot group collaboration system; Based on the second global prediction reward value, the first global prediction reward value is corrected to obtain a global prediction reward value.

[0088] It can be understood that by summing the local predicted reward values ​​of each robot, the second global predicted reward value of the robot swarm collaboration system is obtained. Based on the second global predicted reward value, the first global predicted reward value is corrected, so that the system can dynamically adapt to environmental changes and changes in robot states, providing a more reliable basis for decision-making, and helping to balance individual interests and overall interests when making decisions, ensuring that the robot swarm collaboration system can meet the local needs of each robot as much as possible while pursuing overall optimization.

[0089] In some embodiments, the parameters of the initial neural network of each robot are integrated, including: Determine the weight of the parameters of the initial neural network of each robot according to the contribution of the local predicted reward value of each robot to the global predicted reward value, and the difference between the global predicted reward value and the preset target global reward value; Calculating the fusion parameters of each robot according to the parameters of the initial neural network of each robot and the weights of the parameters of the initial neural network of each robot; According to the fusion parameters of each robot, the parameters of the initial neural network of each robot are updated.

[0090] Optionally, the calculation formula of the fusion parameter of each robot is as follows: ; in, represents the fusion parameters of each robot, represents the parameters of the initial neural network of each robot, represents the parameters of the initial neural network of other robots, Represents the mixing factor for each robot.

[0091] It can be understood that the SRBES mechanism is closely linked to parameter fusion and collaboration between agents. Through the parameter fusion mechanism, agents can share and mix parameters, which enhances the strategy diversity and adaptability between agents. Agents can not only retain their own learning experience, but also share and utilize the team's experience by fusing parameters with other agents, which can promote team collaboration and experience sharing, improve the exploration efficiency of agents, and ensure that agents can learn from a wide range of experiences, avoiding premature convergence to suboptimal strategies, and significantly improving the adaptability of agents to environmental changes and the generalization of strategies.

[0092] The following is a description of a robot control device based on multi-agent reinforcement learning provided in an embodiment of the present invention. The robot control device based on multi-agent reinforcement learning described below and the robot control method based on multi-agent reinforcement learning described above can be referenced to each other.

[0093] Figure 4 A schematic diagram of the structure of a robot control device based on multi-agent reinforcement learning provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the robot control device 400 based on multi-agent reinforcement learning includes: A parameter resetting unit 410, used for periodically adaptively resetting the strategy parameters of the initial neural network of each robot in the robot swarm collaboration system; The first training unit 420 is used to input the sample observation data of each robot at the current moment and the sample action of each robot at the previous moment in the robot group cooperation system into the initial neural network of each robot, and obtain the local predicted reward value of each robot output by the initial neural network of each robot; A second training unit 430 is used to calculate the global predicted reward value of the robot-crowd collaboration system according to the local predicted reward value of each robot, and fuse the parameters of the initial neural network of each robot; The third training unit 440 is used to update the experience pool of each robot, collect sample data from the experience pool of each robot, and periodically repeat the training process of each robot until a preset convergence condition is reached to obtain a neural network of each robot; The control unit 450 is used to control each robot in the actual environment to complete the collaborative task based on the neural network of each robot.

[0094] Optionally, updating the experience pool of each robot and collecting sample data from the experience pool of each robot include: Storing multiple trajectories of each robot in each training cycle into an experience pool of each robot, wherein the experience pool of each robot pre-stores multiple historical trajectories of each robot; A balanced experience sampling strategy is adopted to collect sample trajectories that are significantly different from the reference trajectory from the experience pool of each robot to obtain sample data, which includes sample observation data and sample actions.

[0095] Optionally, a balanced experience sampling strategy is adopted to collect sample trajectories that are significantly different from the reference trajectory from the experience pool of each robot, including: Randomly select a reference sample trajectory and multiple sample trajectories from the experience pool of each robot: Extracting features from a reference sample trajectory to obtain a feature representation of the reference sample trajectory, and extracting features from multiple sample trajectories to obtain feature representations of multiple sample trajectories; Calculate the cosine similarity between the feature representation of each sample trajectory and the feature representation of the reference sample trajectory to obtain the similarity between each sample trajectory and the reference sample trajectory; According to the similarity between each sample trajectory and the reference sample trajectory, a sample trajectory having a greater difference from the reference trajectory is determined from the multiple sample trajectories.

[0096] Optionally, adaptively resetting the strategy parameters of the initial neural network of each robot in the robot swarm collaboration system according to preset rules includes: Take each robot as an independent intelligent agent and construct the initial neural network of each robot; Based on the adaptive strategy reset mechanism, some strategy parameters of the initial neural network of each robot are periodically discarded.

[0097] Optionally, the global predicted reward value of the robot crowd collaboration system is calculated based on the local predicted reward value of each robot, including: Obtaining the current global state of the robot swarm collaboration system, which includes the environment state of the robot swarm collaboration system, the state of each robot, the interaction information between the robots, and the collaboration task information of the robot swarm collaboration system; Inputting the local predicted reward value of each robot and the current global state of the robot swarm collaboration system into a pre-constructed hybrid network, and calculating a first global predicted reward value of the robot swarm collaboration system according to the local predicted reward value of each robot and the current global state of the robot swarm collaboration system; Correcting the first global prediction reward value to obtain a global prediction reward value; Based on the global predicted reward value and the preset target global reward value, the loss function value is calculated, and the parameters of the hybrid network are optimized according to the loss function value.

[0098] Optionally, the first global prediction reward value is modified to obtain a global prediction reward value, including: The local predicted reward values ​​of each robot are summed to obtain the second global predicted reward value of the robot group collaboration system; Based on the second global prediction reward value, the first global prediction reward value is corrected to obtain a global prediction reward value.

[0099] Optionally, the parameters of the initial neural network of each robot are integrated, including: Determine the weight of the parameters of the initial neural network of each robot according to the contribution of the local predicted reward value of each robot to the global predicted reward value, and the difference between the global predicted reward value and the preset target global reward value; Calculating the fusion parameters of each robot according to the parameters of the initial neural network of each robot and the weights of the parameters of the initial neural network of each robot; According to the fusion parameters of each robot, the parameters of the initial neural network of each robot are updated.

[0100] Optionally, the sample observation data of each robot at the current moment and the sample action of each robot at the previous moment are input into the initial neural network of each robot to obtain the local predicted reward value of each robot output by the initial neural network of each robot, including: Obtaining the sample observation data of each robot at the current moment, the sample observation data including the sample state of each robot, the sample environment information within the observation range of each robot, and the sample pose information of the sample object to be processed; The sample observation data of each robot at the current moment and the sample action of the previous moment are input into the initial neural network of each robot. The initial neural network of each robot performs feature extraction on the sample observation data of each robot at the current moment and the sample action of the previous moment to obtain the sample feature representation of each robot. According to the sample feature representation of each robot, the predicted action of each robot at the current moment is obtained. According to the reward obtained after each robot takes the predicted action at the current moment, the local predicted reward value of each robot is calculated.

[0101] Optionally, a robot group in an actual environment is controlled based on the neural network of each robot to complete a collaborative task, including: Obtain the observation data of each robot at the current moment and the action at the previous moment; Input the observation data of each robot at the current moment and the action of the previous moment into the neural network of each robot to obtain the action of each robot at the current moment; Generate control instructions for each robot based on the current action of each robot; Based on the control instructions of each robot, each robot is controlled to grab the corresponding target object to complete the collaborative task.

[0102] It should be noted here that the robot control device based on multi-agent reinforcement learning provided in an embodiment of the present invention can implement all the method steps implemented in the above-mentioned robot control method embodiment based on multi-agent reinforcement learning, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0103] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the robot control method based on multi-agent reinforcement learning, the method comprising: adaptively resetting the strategy parameters of the initial neural network of each robot in the robot group collaboration system according to preset rules; inputting the sample observation data of each robot at the current moment and the sample action of the previous moment into the initial neural network of each robot to obtain the local predicted reward value of each robot output by the initial neural network of each robot; calculating the global predicted reward value of the robot group collaboration system according to the local predicted reward value of each robot, and integrating the parameters of the initial neural network of each robot; updating the experience pool of each robot, collecting sample data from the experience pool of each robot, and periodically repeating the training process of each robot until the preset convergence condition is reached to obtain the neural network of each robot; and controlling each robot in the actual environment to complete the collaborative task based on the neural network of each robot.

[0104] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0105] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the robot control method based on multi-agent reinforcement learning provided by the above-mentioned methods, the method comprising: adaptively resetting the strategy parameters of the initial neural network of each robot in the robot group collaborative system according to preset rules; inputting the sample observation data of each robot at the current moment and the sample action of each robot at the previous moment into the initial neural network of each robot, and obtaining the local predicted reward value of each robot output by the initial neural network of each robot; calculating the global predicted reward value of the robot group collaborative system based on the local predicted reward value of each robot, and integrating the parameters of the initial neural network of each robot; updating the experience pool of each robot, collecting sample data from the experience pool of each robot, and periodically repeating the training process of each robot until the preset convergence condition is reached to obtain the neural network of each robot; and controlling each robot in the actual environment to complete the collaborative task based on the neural network of each robot.

[0106] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot control method based on multi-agent reinforcement learning, characterized in that: include: Adaptively reset the strategy parameters of the initial neural network of each robot in the robot swarm collaboration system according to preset rules; Inputting the sample observation data of each robot at the current moment and the sample action of each robot at the previous moment into the initial neural network of each robot, and obtaining the local predicted reward value of each robot output by the initial neural network of each robot; Calculating the global predicted reward value of the robot group collaboration system according to the local predicted reward value of each robot, and fusing the parameters of the initial neural network of each robot; Updating the experience pool of each robot, collecting sample data from the experience pool of each robot, and periodically repeating the training process of each robot until a preset convergence condition is reached to obtain a neural network of each robot; Based on the neural network of each robot, each robot in the actual environment is controlled to complete the collaborative task.

2. The robot control method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The updating of the experience pool of each robot and collecting sample data from the experience pool of each robot includes: Storing the multiple trajectories of each robot in each training cycle into the experience pool of each robot, wherein the experience pool of each robot pre-stores multiple historical trajectories of each robot; A balanced experience sampling strategy is adopted to collect sample trajectories with large differences from reference trajectories from the experience pool of each robot to obtain sample data, wherein the sample data includes sample observation data and sample actions.

3. The robot control method based on multi-agent reinforcement learning according to claim 2 is characterized in that: The balanced experience sampling strategy is adopted to collect sample trajectories with large differences from the reference trajectory from the experience pool of each robot, including: A reference sample trajectory and multiple sample trajectories are randomly selected from the experience pool of each robot: Performing feature extraction on the reference sample trajectory to obtain a feature representation of the reference sample trajectory, and performing feature extraction on the multiple sample trajectories to obtain feature representations of the multiple sample trajectories; Calculating the cosine similarity between the feature representation of each sample trajectory and the feature representation of the reference sample trajectory to obtain the similarity between each sample trajectory and the reference sample trajectory; According to the similarity between each sample trajectory and the reference sample trajectory, a sample trajectory having a greater difference from the reference trajectory is determined from the multiple sample trajectories.

4. The robot control method based on multi-agent reinforcement learning according to claim 1, characterized in that: The method of adaptively resetting the strategy parameters of the initial neural network of each robot in the robot group collaboration system according to the preset rules includes: Taking each robot as an independent intelligent agent, constructing an initial neural network of each robot; Based on the adaptive strategy reset mechanism, part of the strategy parameters of the initial neural network of each robot are periodically discarded.

5. The robot control method based on multi-agent reinforcement learning according to claim 1, characterized in that: The calculating the global predicted reward value of the robot group collaboration system according to the local predicted reward value of each robot includes: Acquire the current global state of the robot swarm collaboration system, wherein the current global state includes the environment state of the robot swarm collaboration system, the state of each robot, the interaction information between the robots, and the collaboration task information of the robot swarm collaboration system; Inputting the local predicted reward value of each robot and the current global state of the robot swarm collaborative system into a pre-constructed hybrid network, and calculating a first global predicted reward value of the robot swarm collaborative system according to the local predicted reward value of each robot and the current global state of the robot swarm collaborative system; Correcting the first global prediction reward value to obtain the global prediction reward value; Based on the global predicted reward value and the preset target global reward value, a loss function value is calculated, and according to the loss function value, the parameters of the hybrid network are optimized.

6. The robot control method based on multi-agent reinforcement learning according to claim 5 is characterized in that: The step of correcting the first global prediction reward value to obtain the global prediction reward value includes: Summing the local predicted reward value of each robot to obtain a second global predicted reward value of the robot group collaboration system; Based on the second global predicted reward value, the first global predicted reward value is corrected to obtain the global predicted reward value.

7. The robot control method based on multi-agent reinforcement learning according to claim 1, characterized in that: The parameters of the initial neural network of each robot are integrated, including: Determine the weight of the parameters of the initial neural network of each robot according to the contribution of the local predicted reward value of each robot to the global predicted reward value and the difference between the global predicted reward value and a preset target global reward value; Calculating the fusion parameters of each robot according to the parameters of the initial neural network of each robot and the weights of the parameters of the initial neural network of each robot; According to the fusion parameters of each robot, the parameters of the initial neural network of each robot are updated.

8. The robot control method based on multi-agent reinforcement learning according to claim 1, characterized in that: The step of inputting the sample observation data of each robot at the current moment and the sample action of each robot at the previous moment into the initial neural network of each robot to obtain the local predicted reward value of each robot output by the initial neural network of each robot includes: Acquire sample observation data of each robot at the current moment, wherein the sample observation data includes a sample state of each robot, sample environment information within the observation range of each robot, and sample pose information of a sample object to be processed; The sample observation data of each robot at the current moment and the sample action of each robot at the previous moment are input into the initial neural network of each robot, and the initial neural network of each robot performs feature extraction on the sample observation data of each robot at the current moment and the sample action of each robot at the previous moment to obtain the sample feature representation of each robot, and the predicted action of each robot at the current moment is obtained based on the sample feature representation of each robot, and the local predicted reward value of each robot is calculated based on the reward obtained after each robot takes the predicted action at the current moment.

9. The robot control method based on multi-agent reinforcement learning according to any one of claims 2 to 8, characterized in that: The neural network based on each robot controls the robot group in the actual environment to complete the collaborative task, including: Obtaining the observation data of each robot at the current moment and the action at the previous moment; Inputting the observation data of each robot at the current moment and the action of each robot at the previous moment into the neural network of each robot to obtain the action of each robot at the current moment; Based on the current action of each robot, generate a control instruction for each robot; Based on the control instructions of each robot, each robot is controlled to grasp the corresponding target object to complete the collaborative task.

10. A robot control device based on multi-agent reinforcement learning, characterized in that: include: A parameter resetting unit, used for adaptively resetting the strategy parameters of the initial neural network of each robot in the robot group collaboration system according to preset rules; A first training unit is used to input the sample observation data of each robot at the current moment and the sample action of each robot at the previous moment in the robot group cooperation system into the initial neural network of each robot, and obtain the local predicted reward value of each robot output by the initial neural network of each robot; A second training unit, configured to calculate a global predicted reward value of the robot group collaboration system according to the local predicted reward value of each robot, and fuse the parameters of the initial neural network of each robot; A third training unit is used to update the experience pool of each robot, collect sample data from the experience pool of each robot, and periodically repeat the training process of each robot until a preset convergence condition is reached to obtain a neural network of each robot; A control unit is used to control each robot in the actual environment to complete a collaborative task based on the neural network of each robot.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the robot control method based on multi-agent reinforcement learning as described in any one of claims 1 to 9 is implemented.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the robot control method based on multi-agent reinforcement learning as described in any one of claims 1 to 9 is implemented.

Images