A single-body robot dynamic manipulation method based on multi-agent reinforcement learning

Abstract

The application provides a kind of single robot dynamic manipulation method based on multi-agent reinforcement learning, it is related to single robot technical field.The method of the application constructs the dynamic manipulation of single arm four-legged robot as a collaborative multi-agent reinforcement learning MARL problem.Specifically, the method is based on multi-agent proximal policy optimization algorithm MAPPO, four legs are regarded as four independent isomorphic agents, and efficient motion coordination is realized by sharing the same set of network parameters, while the robot arm is modeled as another independent heterogeneous agent, using another set of parameters to focus on manipulation tasks. Through this cooperative modeling method of heterogeneous agents, the close coordination of movement and manipulation can be achieved in a unified reinforcement learning framework, thereby significantly improving the efficiency and robustness of the four-legged robot with arm in unstructured environments.

Description

Technical Field

[0001] This invention relates to the field of single-agent robot technology, and in particular to a dynamic manipulation method for single-agent robots based on multi-agent reinforcement learning. Background Technology

[0002] With the continuous advancement of intelligent control and machine learning technologies, the mobility of quadruped robots in dynamic environments has been significantly enhanced, particularly in terms of motion stability and terrain adaptability in unstructured terrain. In recent years, legged robot systems integrating mobile platforms and manipulators have gradually become a research focus in the fields of service robots and special-purpose robots due to their high mobility and operational flexibility during task execution. Currently, the mainstream technical approaches for control strategies of quadruped robots with manipulators mainly include the following two categories:

[0003] (1) Layered decoupled control architecture: This method divides the robot system into two relatively independent modules: motion (lower limbs) and manipulation (upper limbs), and designs control strategies for each module and trains and optimizes them independently. Some methods attempt to achieve overall control and coordination through an upper-level system integration framework. Although this type of method shows certain practicality in specific tasks, the separation framework during the training process leads to a lack of deep collaboration and decoupling between the motion and manipulation units, making it difficult for the robot to cope with complex tasks that require a high degree of coordination.

[0004] (2) Integrated end-to-end learning framework: This approach typically utilizes imitation learning or single-agent reinforcement learning to construct an integrated mapping mechanism from environmental perception to joint control, thereby achieving end-to-end control strategy learning. Furthermore, some studies have explored combining traditional trajectory optimization or diffusion-based optimization with deep reinforcement learning to improve control interpretability and tracking capabilities. However, due to the strong dynamic coupling between movement and manipulation in robots, simple end-to-end learning (such as single-agent end-to-end learning) often fails to adequately model the complex collaborative interactions within the system. Therefore, the control accuracy and generalization performance of this type of method are severely limited in complex task scenarios.

[0005] In summary, while existing learning methods have made some progress in improving the motion and manipulation capabilities of quadruped robots with arms, they still have the following shortcomings: First, the modular approach of hierarchical control leads to insufficient collaboration between the upper and lower body, making it difficult to meet the needs of coordinated full-body control in complex environments. Second, end-to-end methods generally suffer from low training efficiency and poor policy transferability when dealing with high-dimensional coupled control systems. Third, existing methods are still insufficient in their generalization ability to handle multiple tasks and scenarios, making it difficult to meet the requirements of high robustness and adaptability in practical applications. Therefore, existing technologies have not effectively addressed the core issues of whole-body coordinated control for quadruped robots with arms, restricting their ability to achieve efficient and stable dynamic manipulation in multiple tasks and scenarios. Based on this, there is an urgent need to propose a new control architecture to achieve unified optimization and efficient collaboration of the whole-body motion and manipulation of quadruped robots with arms, overcome the limitations of existing technologies, and adapt to increasingly complex practical application scenarios. Summary of the Invention

[0006] To address the shortcomings of existing quadruped robots with arms in terms of movement and maneuver coordination, this invention provides a dynamic manipulation method for a single robot based on multi-agent reinforcement learning. This overcomes the limitations of existing technologies that often employ modular hierarchical planning and control or model the entire robot as a single agent for dynamic grasping.

[0007] A method for dynamic manipulation of a single robot based on multi-agent reinforcement learning, specifically including the following steps:

[0008] Step 1: Model a single quadruped robot with an arm and provide a formal description;

[0009] Modeling a single quadruped robot with an arm as a cooperative multi-agent problem is described as a partially observable distributed Markov decision process, consisting of tuples. Definition; where S is the state space, A is the action space, P is the state transition distribution, R is the shared reward, Z is the observation space, O is the observation function, N is the number of agents, γ is the discount factor, and T is the time range; at each time step t, each agent Select an action within action space A to generate a combined action. ;in For the action space of the policy output of the Nth agent at time step t, the environment transitions to a new state according to the state transition distribution P and provides a shared reward; each agent receives an observation from O. And maintain an observation-action history record.

[0010] Step 2: Design the state-aware module;

[0011] The state perception module is used to collect state information of various parts of the robot. The state information is divided into local observations, shared observations, and global observations. The local observations are state information perceived independently by each agent, while the shared observations are global information shared by all agents. The global observations are the overall state information of the robot received by the global critic network Critic, including the motor position and speed of each limb, previous and current actions, foot contact state, gait sorting triangle timing guidance controller, projected gravity, high-level control commands generated by the upper-level strategy, body posture and speed, end effector state, visual information, and external disturbance information.

[0012] Step 3: Design the strategy reasoning module, the Actor network, and the Critic network;

[0013] The policy reasoning module divides the quadruped robot with an arm into a quadrupedal motion system and a robotic arm manipulation system. The quadrupedal motion system consists of four homogeneous agents, which receive local cascaded observations from the four leg agents through a first Actor network that shares the same set of parameters. The robotic arm manipulation system, on the other hand, is an independent heterogeneous agent that uses a second Actor network with a different set of parameters to process the robotic arm agent's exclusive manipulation observations. This enables collaborative reasoning of motion and manipulation within a unified reinforcement learning framework. Specifically, it is based on the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm and is implemented through a centralized value function evaluation and a distributed policy execution mechanism.

[0014] For a quadrupedal locomotion system, the input local cascaded observations include: leg joint movement positions. ,speed Actions at time step t-1 and the action of time step t Gait sequencing triangular timing guidance controller Projected gravity Command value fuselage attitude Aircraft speed And agent one-hot encoding ;

[0015] Both the first Actor network and the second Actor network consist of a normalization layer, a multilayer perceptron (MLP) layer, a gated recurrent unit (GRU) layer, and a normalization layer. An offset phase is introduced at the output of the first Actor network and the second Actor network to correct the difference between the target joint position and the actual joint position.

[0016] For robotic arm manipulation systems, color-depth RGB-D images from the end effector's perspective are encoded using PointNet and used as one of the inputs to the robotic arm agent. The state inputs of the robotic arm agent include: leg joint motion positions. ,speed Actions at time step t-1 and the action of time step t End effector status The color-depth RGB-D image encoded by PointNet is the visual information. External disturbance information High-level control commands generated from upper-level strategies Projected gravity fuselage attitude Aircraft speed And agent one-hot encoding ;

[0017] The first Actor network outputs torque or position commands for the three joints of each leg; the second Actor network outputs control commands for the six joints of the robotic arm and opening / closing commands for the end effector.

[0018] The Critic network adopts the same MLP / GRU network structure as the Actor network, replacing the last normalization layer with a PopArt layer. It takes global observations as input, including motor position, motor speed, previous action, current action, plantar contact state, gait sequencing triangular timing guidance controller, projected gravity, high-level control commands generated by upper-level strategies, fuselage attitude, fuselage speed, end effector state, visual information, and external disturbance information. The Critic network outputs continuous state values.

[0019] Step 4: Establish a hierarchical decoupling reward mechanism;

[0020] Specifically, end effector velocity tracking rewards and end effector pose tracking rewards were introduced;

[0021] The end effector speed tracking reward formula is as follows:

[0022] (1);

[0023] Among them, speed error This indicates the error between the current speed and the commanded speed. , This refers to the current linear or angular velocity of the robot base along the x-axis or y-axis. The target linear velocity or angular velocity given by the command. It is a hyperparameter;

[0024] The end effector pose tracking reward calculates the current pose of the end effector in the base coordinate system in real time, including the position LPY and end pose RPY in the spherical coordinate system, and the geometric error between it and the target pose given by the task command. A negative reward is applied to this geometric error, namely the robot arm command tracking error penalty, which is as follows:

[0025] (2);

[0026] in, and These are the weights of the LPY and RPY errors read from the configuration respectively. For LPY normalization error, , This is the current end effector LPY value. It is the target LPY value given by the instruction. These are the respective instruction ranges of LPY. For RPY normalization error, ;

[0027] The layered decoupling reward mechanism also integrates a series of auxiliary rewards and penalties, including:

[0028] (1) Motion smoothness penalty: The formula is as follows:

[0029] (3);

[0030] (4);

[0031] (5);

[0032] (6);

[0033] in, It is the angular velocity of the i-th joint. It is the angular acceleration of the i-th joint. It is the action instruction output by the j-th executor at time step t. It is the total number of joint degrees of freedom. These are the weighting coefficients in the reward function;

[0034] (2) Energy consumption penalty: Penalize the motor output torque to guide all agents to learn energy-saving gaits and actions with lower energy consumption. The formula is as follows:

[0035] (7);

[0036] in, It is the motor torque applied to the i-th joint;

[0037] (3) Behavioral constraints and safety penalties: The formula is as follows:

[0038] (8);

[0039] (9);

[0040] (10);

[0041] in, It is a collection of key body parts that need to be avoided from collisions. It is the contact force at part k. It is a preset contact force threshold. It is an activation function that takes the value 1 when the condition is true and 0 otherwise. θ and θ are the current roll angle and pitch angle of the aircraft, respectively. and It is the attitude angle expected by the command; These are the weighting coefficients in the reward function;

[0042] (4) Base motion tracking reward: The formula is as follows:

[0043] (11);

[0044] (12);

[0045] (13);

[0046] Among them, v xy It is the current linear velocity vector of the base in the xy plane. It is the desired linear velocity vector. It is a hyperparameter that controls reward sensitivity; It is the current angular velocity of the base around the z-axis. It is the desired angular velocity. The corresponding sensitivity parameters; These are the weighting coefficients in the reward function.

[0047] Step 5: A two-stage training process enables dynamic manipulation;

[0048] By establishing a multi-agent reinforcement learning framework in the CTDE paradigm, the state perception module, policy reasoning module, and hierarchical decoupled reward mechanism are integrated in a unified manner. Under this architecture, a two-stage training process is adopted. In the first stage, only the quadrupedal movement system is trained to obtain a stable basic walking policy. In the second stage, local observation and proprietary reward of the robotic arm agent are introduced on the basis of retaining the basic walking policy, so that the quadrupedal and the robotic arm can achieve collaborative optimization under a unified framework, complete stable movement and precise grasping tasks in complex environments, and finally form a deployable and efficient dynamic manipulation system.

[0049] In multi-agent proximal policy optimization, each agent Have an independent strategy , where o i Let a represent the local observation of the i-th agent. i Let represent the action of the i-th agent; in a paradigm of centralized training and distributed execution, a shared global state-value function is used. A unified evaluation is conducted, and the advantage function is calculated based on this to guide policy updates; the core optimization objective function of MAPPO is: (14);

[0050] in, The probability ratio between the old and new strategies. It is the old policy before the update, and θ is the set of parameters of the policy network; It represents the expectation over time step t; clip(•) is the clipping function in the PPO algorithm; Based on global state value function The estimated advantage function, s t ε represents the current global state; ε is the pruning threshold. In multi-agent scenarios, the policy parameters of the agents are updated independently, but they all rely on the shared global state value function for advantage estimation. Therefore, the joint optimization objective is written as the sum of the optimization objectives of all agents to ensure stable and efficient collaborative learning within a unified framework.

[0051] (15);

[0052] Where N is the total number of agents.

[0053] On the other hand, a dynamic manipulation method for a single robot based on multi-agent reinforcement learning is implemented through the following system: including: a state perception module, a policy reasoning module, and a PD control module;

[0054] The state perception module receives image data from an RGB-D camera mounted on the end effector, and the PointNet network encodes the visual information of the target pose. Simultaneously, it combines sensor information from various robot parts, including joint motor positions, speeds, motion history, body posture and speed, projected gravity, gait sequencing triangular timing guidance controller, high-level control commands generated by upper-level strategies, end effector state, foot contact state, and external disturbance information, to fuse multi-source data and output encoded local observations, shared observations, and global observations. The local observations are input to the Actor networks of the four leg agents of the quadrupedal motion system and the robotic arm agent of the robotic arm manipulation system, respectively. Shared observations are used for collaborative information exchange between agents, while global observations are input to a unified Critic network for state value assessment.

[0055] The policy reasoning module generates leg joint commands and robotic arm joint and end effector grasping commands based on the observation input of each agent using the MAPPO algorithm. The Actor network is responsible for action output, and the Critic network evaluates and guides policy updates based on the global state value function.

[0056] The PD control module compares the joint torque commands output by the Actor network in the strategy reasoning module with the actual joint state, performs error correction and dynamic compensation through proportional-derivative control law, generates stable motor drive signals, and sends them down to the quadruped, robotic arm and end effector, thereby ensuring the execution accuracy and stability of the robot's coordinated movement and dynamic grasping in complex environments.

[0057] Thirdly, this application proposes an electronic device, comprising: one or more processors, and a memory for storing instructions, which, when executed by the one or more processors, cause the one or more processors to perform the aforementioned method for dynamic manipulation of a single robot based on multi-agent reinforcement learning.

[0058] Fourthly, this application proposes a computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform the aforementioned method for dynamic manipulation of a single robot based on multi-agent reinforcement learning.

[0059] Fifthly, this application proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the aforementioned method for dynamic manipulation of a single robot based on multi-agent reinforcement learning.

[0060] The beneficial effects of adopting the above technical solution are as follows:

[0061] This invention provides a method for dynamic manipulation of a single robot based on multi-agent reinforcement learning, which has the following advantages:

[0062] (1) Multi-agent collaborative architecture improves training efficiency and control performance;

[0063] This invention employs a cooperative heterogeneous multi-agent reinforcement learning framework, modeling each limb (quadrupes and robotic arm) of a quadruped robot as an independent agent, and optimizing the policy through a centralized training and distributed execution paradigm. This architecture effectively improves the coordination between limbs and significantly accelerates the convergence speed of the policy. Experimental results show that, compared with the traditional single-agent proximal policy optimization (PPO) baseline, this invention achieves faster reward growth in the early stages of training and reaches a higher asymptotic reward value after convergence, especially in leg control tasks where the improvement is more significant.

[0064] (2) The reward mechanism is dense, adaptive, and stable;

[0065] The reward signal of this invention features density (continuous feedback throughout the process) and adaptability (weights are adjusted according to the task stage), supplemented by sparse success rewards, force penalties, relaxation penalties, and other stabilizing terms to improve the convergence and robustness of the training process. Furthermore, abstract grasping metrics and realistic grasping posture metrics are used for evaluation in both simulated and real-world environments to ensure the strategy performs well under different evaluation systems.

[0066] (3) Domain randomization and transferability;

[0067] To bridge the performance gap between simulation and reality, this invention introduces domain randomization during training. This includes randomization of physical parameters during environment initialization and randomization of dynamic perturbations at each time step. Furthermore, the actuator network captures the non-ideal relationship between PD control error and actual torque during training. This mechanism enhances the policy's adaptability to different physical conditions, enabling the trained policy to possess good real-world transfer capabilities.

[0068] (4) Deployment performance and robustness verification;

[0069] The controller of this invention closely follows the reference trajectory in both phase and amplitude, which is significantly better than the traditional PPO baseline and avoids overshoot and phase lag. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of the process of a single robot dynamic manipulation method based on multi-agent reinforcement learning in an embodiment of the present invention;

[0071] Figure 2 This is a training logic diagram for a single intelligent agent in an embodiment of the present invention;

[0072] Figure 3 This is a diagram of the training network structure for a single agent in an embodiment of the present invention. Detailed Implementation

[0073] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope or methods of the invention.

[0074] Example 1: This paper considers a finite-time Markov Decision Process (MDP), defined by a tuple M = (S, A, P, r, γ, T). S represents the state space. Represents a sequence of actions in a finite action space. , Represents the state transition distribution. Represents the reward function, Let represent the discount factor, and T represent the time range. At each time step t, the policy π is... Select an action from [location]. Once the sampling enters the next state, the AI ​​will immediately receive a reward. The agent continues to perform actions until it enters a terminated state or time interval T is reached. Reinforcement learning aims to learn policies for decision-making problems by maximizing discounted rewards. Furthermore, multi-agent proximal policy optimization is a reinforcement learning algorithm specifically designed for multi-agent environments. It extends the single-agent proximal policy optimization, enabling multiple agents to stably learn their respective policies in a shared environment.

[0075] In practical training, MAPPO typically maintains an independent policy network for each agent while using a centralized value function network to evaluate the state of the entire system. The policy network updates action selection policies through gradient ascent, while the value network fits the global state reward through regression to calculate the advantage function. This centralized evaluation-distributed execution architecture enables MAPPO to maintain stability and efficiency in complex cooperative or adversarial tasks. Thanks to this design, MAPPO exhibits good convergence speed, policy coordination, and generalization ability in multi-agent reinforcement learning tasks. It is widely used in fields such as robot collaboration, intelligent transportation, and online games, becoming one of the mainstream methods in multi-agent reinforcement learning. Compared to traditional methods, MAPPO is easier to implement, more stable to train, and has stronger adaptability and robustness to environmental changes or increases in the number of agents.

[0076] A method for dynamic manipulation of a single robot based on multi-agent reinforcement learning is proposed. Employing a MARL architecture with a centralized training and distributed execution (CTDE) paradigm, it achieves motion and manipulation capabilities for a quadruped robot with an arm in complex environments. The system models each limb (arm and leg) of the quadruped robot as an independent agent within a cooperative heterogeneous agent system. All agents possess independent actor policy networks that output actions, while sharing a common global critic for estimating the state-value function. (e.g.) Figure 2 This framework bridges the gap between dynamic manipulation learning for a single quadruped robot with an arm and multi-agent cooperative learning. Experimental results verify the effectiveness of the cooperative multi-agent learning strategy in improving the motion manipulation capabilities of a single quadruped robot with an arm. In terms of system architecture, the whole-body control system for the quadruped robot's mobile grasping function mainly consists of a policy inference module and a state perception module. The modules interact with each other through standardized signal interfaces, forming a closed-loop control system. Specifically, it includes the following steps:

[0077] Step 1: Model a single quadruped robot with an arm and provide a formal description;

[0078] Modeling a single quadruped robot with an arm as a cooperative multi-agent problem is described as a decentralized-partially observable Markov Decision Process (dec-POMDP), consisting of tuples. Definition; where S is the state space, A is the action space, P is the state transition distribution, R is the shared reward, Z is the observation space, O is the observation function, N is the number of agents, γ is the discount factor, and T is the time range; at each time step t, each agent Select an action within action space A to generate a combined action. ;in For the action space of the policy output of the Nth agent at time step t, the environment transitions to a new state according to the state transition distribution P and provides a shared reward; each agent receives an observation from O. and maintain an observation-action history record;

[0079] This invention proposes a novel MARL framework, which aims to learn a strategy that maximizes expected cumulative reward. By leveraging interlimb coordination to enhance the locomotion capabilities of a single robot, and employing the CTDE paradigm, the system maintains stability and consistency in dynamic environments. Specifically, the policy reasoning module learns policies (π) through a dual-network structure. θ ) and state value function ( This approach, combined with a time-based reward mechanism, effectively improves policy convergence speed and grasp success rate. The value function helps reduce training variability and handles additional global observations. Compared to traditional methods, this architecture achieves superior final performance, faster convergence, and enhanced deployment robustness, which is particularly effective for complex robot control tasks requiring cooperative learning and operational flexibility.

[0080] Step 2: Design the state-aware module;

[0081] The state perception module is used to collect state information of various parts of the robot and encode the state information into the input of the agent's policy. To adapt to the multi-agent architecture, the state information is divided into: local observations, shared observations, and global observations. The local observations are state information independently perceived by each agent, which includes a quadruped and a robotic arm. The shared observations are global information shared by all agents, including the robot's body state, timing guidance controller, etc. The global observations are the overall state information of the robot received by the global critic network Critic, including the motor position and speed of each limb, previous and current actions, gait sorting triangle timing guidance controller, projected gravity, high-level control commands generated by the upper-level policy, body posture and speed, foot contact state, end effector state, and external disturbance information, which are used for state value function estimation and policy optimization.

[0082] Step 3: Design the strategy reasoning module, the Actor network, and the Critic network;

[0083] The strategy reasoning module divides the quadruped robot with an arm into a quadrupedal motion system and a robotic arm manipulation system (e.g., ...). Figure 1 The quadrupedal locomotion system consists of four homogeneous agents. It receives local cascaded observations from the four leg agents in the quadrupedal locomotion system through a first Actor network that shares the same set of parameters to achieve efficient coordination. The robotic arm manipulation system, on the other hand, is an independent heterogeneous agent. It uses a second Actor network with a different set of parameters to process the robotic arm agent's exclusive manipulation observations. This enables collaborative reasoning of motion and manipulation within a unified reinforcement learning framework. Specifically, it is based on the multi-agent proximal policy optimization algorithm MAPPO and is implemented through a centralized value function evaluation and a distributed policy execution mechanism.

[0084] For a quadrupedal locomotion system, the input local cascaded observations include: leg joint movement positions. ,speed Actions at time step t-1 and the action of time step t Gait sequencing triangular timing guidance controller Projected gravity Command value fuselage attitude Aircraft speed And agent one-hot encoding (ID);

[0085] Both the first Actor network and the second Actor network consist of a normalization layer, a multilayer perceptron (MLP) layer, a gated recurrent unit (GRU) layer, and a normalization layer (e.g., Figure 3 In addition, a deviation phase is introduced at the output of the first Actor network and the second Actor network to correct the difference between the target joint position and the actual joint position. This deviation term is used to improve control accuracy, and it is particularly stable when there are execution errors or physical delays.

[0086] For robotic arm manipulation systems, to enhance the strategy input of the robotic arm, this invention utilizes a color-depth RGB-D image from the end effector's perspective, encoded using PointNet, and serves as one of the inputs to the robotic arm agent. The visual input encoded with PointNet possesses order invariance and spatial representation capabilities. The state input of the robotic arm agent includes: leg joint movement position. ,speed Actions at time step t-1 and the action of time step t End effector status (Including spatial position, quaternions, velocity, and the Euclidean distance and quaternion angle difference between the end effector and the target pose), and visual information in the color-depth RGB-D image encoded by PointNet. External disturbance information High-level control commands generated from upper-level strategies Projected gravity fuselage attitude Aircraft speed And agent one-hot encoding ;

[0087] The first Actor network outputs torque or position commands for the three joints of each leg; the second Actor network outputs control commands for the six joints of the robotic arm and opening / closing commands for the end effector.

[0088] The Critic network employs the same MLP / GRU network structure as the Actor network, replacing the last normalization layer with a PopArt layer that integrates adaptive and denormalized linear layers. It takes global observations as input, including the motor positions and speeds of each limb, previous actions, current actions, plantar contact states, gait ordering triangulation timing guidance controller, projected gravity, higher-level control commands generated by the upper-level policy, and fuselage speed. The Critic network outputs continuous state values, which are used to calculate the advantage function, thereby guiding policy updates. It outputs a stable value prediction with accurate scaling (e.g., ...). Figure 3 );

[0089] Step 4: Establish a hierarchical decoupling reward mechanism;

[0090] To improve the learning efficiency and performance of multi-agent cooperative strategies, this invention designs a unified, hierarchical, and decoupled reward function mechanism. The core of this mechanism lies in the fact that all Actor policies are evaluated using a unified Critic network. Simultaneously, error-based dense rewards are used to precisely quantify the deviation between the robot's current state and the desired instruction, providing continuous and effective gradient signals for policy optimization. Specifically, for dynamic manipulation tasks of robotic arms, end-effector velocity tracking rewards and end-effector pose tracking rewards are introduced.

[0091] The end effector speed tracking reward formula is as follows:

[0092] (1);

[0093] Among them, speed error This indicates the error between the current speed and the commanded speed. , This refers to the current linear or angular velocity of the robot base along the x-axis or y-axis. The target linear velocity or angular velocity given by the command. It is a hyperparameter that controls the sensitivity of the reward function to error.

[0094] The end effector pose tracking reward calculates the current pose of the end effector in the base coordinate system in real time, including the position LPY and end pose RPY in the spherical coordinate system, and the geometric error between this pose and the target pose given by the task command. A negative reward, i.e., a robotic arm command tracking error penalty, is applied to this geometric error to incentivize the robotic arm agent to achieve accurate and rapid target pose approach on the moving base. The robotic arm command tracking error penalty is as follows:

[0095] (2);

[0096] in, and These are the weights of the LPY and RPY errors read from the configuration respectively. For LPY normalization error, , This is the current end effector LPY value. It is the target LPY value given by the instruction. These are the respective instruction ranges of LPY, used for normalization to make errors in different dimensions comparable. For RPY normalization error, This design clearly defines the goal of the dynamic manipulation task of "movement-grasping," guiding the policy network to learn the decoupling and coordination between the base motion and the upper-level robotic arm manipulation, thereby significantly improving the grasping accuracy and task success rate in complex dynamic scenarios. Furthermore, to ensure the physical realism, energy efficiency, and smoothness of the robot's motion, the LPY values ​​typically refer to the three attitude angles (Roll, Pitch, Yaw) of the end effector. L represents Roll (roll angle); P (Pitch): pitch angle, representing the rotation of the end effector about the horizontal axis; Y (Yaw): yaw angle, representing the rotation of the end effector about the vertical axis. RPY is a commonly used attitude representation method (Euler angles) in robotics: R (Roll): rotation about the x-axis; P (Pitch): rotation about the y-axis; Y (Yaw): rotation about the z-axis.

[0097] The layered decoupling reward mechanism also integrates a series of auxiliary rewards and penalties, including:

[0098] (1) Motion smoothness penalty: By imposing penalties on joint angular velocity, angular acceleration and rate of change of motion, the robot is suppressed from generating high-frequency, jittery, and ineffective movements, making its behavior smoother and more stable. The formula is as follows:

[0099] (3);

[0100] (4);

[0101] (5);

[0102] (6);

[0103] in, It is the angular velocity of the i-th joint. It is the angular acceleration of the i-th joint. It is the action instruction output by the j-th executor at time step t. It is the total number of joint degrees of freedom. These are the weighting coefficients in the reward function;

[0104] (2) Energy consumption penalty: Penalize the motor output torque to guide all agents to learn energy-saving gaits and actions with lower energy consumption. The formula is as follows:

[0105] (7);

[0106] in, It is the motor torque applied to the i-th joint;

[0107] (3) Behavioral constraints and safety penalties: For example, imposing severe penalties on behaviors such as collisions between the robot and the environment, or deviations from safe orientation, to ensure the robot's safety and stability in the physical world. The formula is as follows:

[0108] (8);

[0109] (9);

[0110] (10);

[0111] in, It is a collection of key body parts that need to be avoided from collisions. It is the contact force at part k. It is a preset contact force threshold. It is an activation function that takes the value 1 when the condition is true and 0 otherwise. θ and θ are the current roll angle and pitch angle of the aircraft, respectively. and It is the attitude angle expected by the command; These are the weighting coefficients in the reward function;

[0112] (4) Base Motion Tracking Reward: By rewarding the robot base for accurate tracking of the desired linear and angular velocity commands (base_lin, base_ang), the mobility and controllability of the mobile platform are ensured. The formula is as follows:

[0113] (11);

[0114] (12);

[0115] (13);

[0116] Among them, v xy It is the current linear velocity vector of the base in the xy plane. It is the desired linear velocity vector. It is a hyperparameter that controls reward sensitivity; It is the current angular velocity of the base around the z-axis. It is the desired angular velocity. The corresponding sensitivity parameters; These are the weighting coefficients in the reward function. This invention decomposes complex dynamic manipulation tasks into quantifiable sub-objectives through a multi-objective, hierarchically decoupled reward function system. This system not only clarifies the core task through pose tracking rewards but also regulates the robot's behavior through a series of auxiliary penalty terms, ultimately effectively improving the policy convergence speed, task success rate, and efficiency in exploring and utilizing the potential workspace.

[0117] Step 5: A two-stage training process enables dynamic manipulation;

[0118] By establishing a multi-agent reinforcement learning framework within the CTDE paradigm, a unified integration of the state perception module, policy reasoning module, and hierarchical decoupled reward mechanism is achieved. The state perception module collects and encodes local, shared, and global observation information. The policy reasoning module consists of a homogeneous Actor network for the quadrupedal locomotion system and a heterogeneous Actor network for the robotic arm manipulation system, outputting leg joint commands and robotic arm joint and end effector commands, respectively. All Actor policies are evaluated using a unified Critic network, while the reward mechanism achieves dynamic balance through a combination of core task rewards (end-efficiency velocity and pose tracking, successful grasping) and auxiliary constraint rewards (smoothness, energy consumption, safety, base tracking) combined with time scheduling weights. Under this architecture, a two-stage training process is adopted. The first stage trains only the quadrupedal locomotion system to obtain stable basic walking policies, reducing the difficulty of directly learning complex dynamic manipulation. The second stage, while retaining the basic walking policies, introduces local observations and proprietary rewards from the robotic arm agent, enabling the quadrupedal locomotion system and robotic arm to achieve collaborative optimization within a unified framework. This allows for stable movement and precise grasping tasks in complex environments, ultimately forming a deployable and efficient dynamic manipulation system, such as... Figure 2 As shown.

[0119] The core idea of ​​MAPPO is to improve overall learning efficiency and stability by maintaining independent policy updates for each agent while utilizing a centralized training mechanism. Compared to traditional multi-agent policy gradient methods, MAPPO introduces a centralized value function evaluation, allowing each agent to refer to global information when updating its policy, thereby better coordinating behavior and avoiding conflicts or inconsistencies. It still uses the pruning objective function of PPO to limit the magnitude of policy updates and prevent drastic policy changes during training. This mechanism is particularly important in multi-agent scenarios because a policy change in one agent can affect the learning process of other agents. In multi-agent proximal policy optimization, each agent... Have an independent strategy , where o i Let a represent the local observation of the i-th agent. i Let represent the action of the i-th agent; in a paradigm of centralized training and distributed execution, a shared global state-value function is used. A unified evaluation is conducted, and the advantage function is calculated based on this to guide policy updates; the core optimization objective function of MAPPO is: (14);

[0120] in, The probability ratio between the old and new strategies. It is the old policy before the update, and θ is the set of parameters of the policy network; It represents the expectation over time step t; clip(•) is the clipping function in the PPO algorithm; Based on global state value function The estimated advantage function, s t ε represents the current global state; ε is the pruning threshold used to limit the magnitude of policy updates. In multi-agent scenarios, the policy parameters of the agents are updated independently, but all rely on a shared global state value function for advantage estimation. Therefore, the joint optimization objective is written as the sum of the optimization objectives of all agents to ensure stable and efficient collaborative learning within a unified framework.

[0121] (15);

[0122] Where N is the total number of agents, and each agent i maintains its policy parameter θ i While updating independently, advantage estimation is performed based on the global state value function. The calculation depends on the global state s t This enables stable and efficient multi-agent collaborative learning within a unified reinforcement learning framework.

[0123] On the other hand, a dynamic manipulation method for a single robot based on multi-agent reinforcement learning is implemented through the following system: including a state perception module, a policy reasoning module, and a PD control module; each functional module is connected through a standardized signal interface to form the following signal flow:

[0124] The state perception module receives image data from an RGB-D camera mounted on the end effector, and encodes the visual information of the target pose using a PointNet network. Simultaneously, it integrates sensor information from various robot parts, including joint motor positions, velocities, motion history, body posture and velocity, projected gravity, gait sequencing triangular timing guidance controller, high-level control commands generated by upper-level strategies, end effector state, foot contact state, and external disturbance information. This multi-source data is fused to output encoded local observations, shared observations, and global observations. The local observations are input to the Actor networks of the four leg agents of the quadrupedal motion system and the robotic arm agent of the robotic arm manipulation system, respectively, serving as state inputs to the policy reasoning module. Shared observations are used for collaborative information exchange between agents, while global observations are input to a unified Critic network for state value evaluation.

[0125] The policy reasoning module generates leg joint commands and robotic arm joint and end effector grasping commands based on the observation input of each agent using the MAPPO algorithm. The Actor network is responsible for action output, and the Critic network evaluates and guides policy updates based on the global state value function.

[0126] The PD control module compares the joint torque commands output by the Actor network in the strategy reasoning module with the actual joint state, performs error correction and dynamic compensation through proportional-derivative control law, generates stable motor drive signals, and sends them down to the quadruped, robotic arm and end effector, thereby ensuring the execution accuracy and stability of the robot's coordinated movement and dynamic grasping in complex environments.

[0127] Example 2:

[0128] In the simulation, the system was deployed on an onboard microcomputer equipped with an Intel Core i7-13620H processor and an NVIDIA GeForce RTX 4060 Mobile graphics card. The system was built on the Unitree Go2 quadruped robot, equipped with an ARX5 robotic arm, a Robotiq gripper as the end effector controller, and an Intel RealSense D415 camera mounted on the wrist. To bridge the gap between simulation and real-world performance, this invention employed domain randomization and actor networks. Domain randomization refers to randomizing various parameters to train robust policies under different conditions. Based on the application's temporal sequence, all domain randomized parameters can be divided into two categories. The first category includes physical parameters randomized during environment initialization, such as link mass and inertia, joint damping and friction, ground friction, and gravity. The second category involves parameters randomized at each simulation step, including motion delays, torque noise, and external disturbances. These randomizations enhance the policy's robustness and adaptability to different physical conditions. The training process of the actor network captures the non-ideal relationship between PD error and realized torque, which helps to improve the transferability and control accuracy of the policy in real physical systems.

[0129] The simulation experiments and verification on the real platform are as follows:

[0130] The simulation experiments included: ground / tabletop object grasping based on the Visual Barrier Challenge Dataset (VBC) and random object grasping based on the Yale-CMU-Berkeley Object and Model Set (YCB). Extensive experiments on a dynamic manipulation environment created by Isaac Gym and a real-world platform (Unitree quadruped robot Unitree Go2 + ARX5 ultralight force-controlled robotic arm) demonstrated the invention's high precision and robustness. Especially under complex terrain and dynamic disturbance scenarios, the system maintained stable grasping performance, verifying the invention's broad applicability in practical applications.

[0131] This invention proposes a cooperative heterogeneous multi-agent reinforcement learning framework for control tasks of quadruped robots with arms. Based on a centralized training and distributed execution paradigm, it achieves coordinated movement and precise manipulation capabilities in complex environments. This method has been extensively validated on the Isaac Gym simulation platform and in the real world using Unitree Go2 + ARX5, demonstrating the following key technical advantages:

[0132] (1) Collaborative Multi-Agent Learning Architecture. The control problem of a single quadruped robot with an arm is reformulated as a multi-agent collaborative control problem, and a heterogeneous agent system is constructed. Each agent has an independent policy network and is responsible for local action reasoning; the global commentator network shares global observation information for value evaluation. Through shared experience and joint optimization, this invention achieves rapid policy convergence and cross-limb collaborative enhancement.

[0133] (2) Multi-layered observation structure and shared learning mechanism. A three-layered structure is constructed, consisting of local observations (joint states, movement history), shared observations (body states, timing guidance controllers, agent one-hot encoding), and global observations (for Critic network input). Gait coordination is achieved through a timing guide, and policy independence is ensured through the agent's one-hot encoded ID. Shared learning strategies enhance cross-limb coordination and policy generalization.

[0134] (3) Domain randomization and the "Sim-to-Real transfer" mechanism. Domain randomization is introduced during training, including randomization of physical parameters (mass, friction, damping, etc.) during the environment initialization phase and dynamic disturbances (action delay, torque noise, external interference, etc.) at each time step. The actuator network captures the non-ideal relationship between PD control error and actual torque during training, improving the control accuracy and transfer capability of the strategy in real physical systems. Experimental results show that the strategy maintains stable performance under different terrain and disturbance conditions and has strong adaptability.

[0135] This invention has undergone extensive experimental verification on simulators and real platforms, and the evaluation metrics include:

[0136] Convergence time: The number of training iterations is significantly less than that of the baseline method;

[0137] Motion smoothness: Controls the continuity of motion sequences, avoiding abrupt changes;

[0138] Aircraft stability: Less attitude fluctuation and smoother movement;

[0139] Limb coordination: The phase difference between the joints in the gait cycle is more consistent, resulting in stronger coordination.

[0140] In gait robustness testing, the robot recovered to its normal state in just one gait cycle after being subjected to external disturbances, demonstrating rapid recovery capability and high stability.

[0141] Example 3:

[0142] This embodiment proposes an electronic device, including: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the aforementioned method for dynamic manipulation of a single robot based on multi-agent reinforcement learning.

[0143] The electronic device may be a mobile phone, computer, or tablet computer, etc., and includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements a single-robot dynamic manipulation method based on multi-agent reinforcement learning as described in the embodiments. It is understood that the electronic device may also include an input / output (I / O) interface and communication components.

[0144] The processor is used to execute all or part of the steps in the multi-agent reinforcement learning-based dynamic manipulation method for a single robot as described in the above embodiments. The memory is used to store various types of data, which may include, for example, instructions for any application or method in an electronic device, as well as application-related data.

[0145] The processor can be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic components, and is used to execute the single-robot dynamic manipulation method based on multi-agent reinforcement learning described in the above embodiments.

[0146] Example 4:

[0147] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0148] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the single robot dynamic manipulation method based on multi-agent reinforcement learning described in the various embodiments of this application.

[0149] The aforementioned storage media include: flash memory, hard disk, multimedia card, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory, random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, APP (Application) application store, and other media capable of storing program verification codes. These media store computer programs, and when executed by a processor, they can implement the various steps of the aforementioned method for dynamic manipulation of a single robot based on multi-agent reinforcement learning.

[0150] Example 5:

[0151] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the aforementioned method for dynamic manipulation of a single robot based on multi-agent reinforcement learning.

[0152] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0153] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0154] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of the methods disclosed herein and their equivalents, then the intent of this disclosure also includes such modifications and variations.

Claims

1. A method for dynamic manipulation of a single robot based on multi-agent reinforcement learning, characterized in that, Includes the following steps: Step 1: Model a single quadruped robot with an arm and provide a formal description; Step 2: Design the state-aware module; The state perception module is used to collect state information of various parts of the robot; Step 3: Design the strategy reasoning module, the Actor network, and the Critic network; The policy reasoning module divides the quadruped robot with an arm into a quadrupedal motion system and a robotic arm manipulation system. The quadrupedal motion system consists of four homogeneous agents, which receive local cascaded observations from the four leg agents through a first Actor network that shares the same set of parameters. The robotic arm manipulation system, on the other hand, is an independent heterogeneous agent that uses a second Actor network with a different set of parameters to process the robotic arm agent's exclusive manipulation observations. This enables collaborative reasoning of motion and manipulation within a unified reinforcement learning framework. Specifically, it is based on the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm and is implemented through a centralized value function evaluation and a distributed policy execution mechanism. Step 4: Establish a hierarchical decoupling reward mechanism, specifically introducing end effector velocity tracking rewards and end effector pose tracking rewards; Step 5: A two-stage training process enables dynamic manipulation; By establishing a multi-agent reinforcement learning framework in the CTDE paradigm, the state perception module, policy reasoning module, and hierarchical decoupled reward mechanism are integrated in a unified manner. Under this architecture, a two-stage training process is adopted. In the first stage, only the quadrupedal motion system is trained to obtain a stable basic walking policy. In the second stage, local observation and proprietary reward of the robotic arm agent are introduced on the basis of retaining the basic walking policy, so that the quadrupedal and the robotic arm can achieve collaborative optimization under a unified framework and realize dynamic manipulation of a single robot.

2. The method for dynamic manipulation of a single robot based on multi-agent reinforcement learning according to claim 1, characterized in that, Step 1 specifically involves: modeling a single quadruped robot with an arm as a cooperative multi-agent problem, describing it as a partially observable distributed Markov decision process, consisting of tuples. Definition; where S is the state space, A is the action space, P is the state transition distribution, R is the shared reward, Z is the observation space, O is the observation function, N is the number of agents, γ is the discount factor, and T is the time range; at each time step t, each agent Select an action within action space A to generate a combined action. ;in For the action space of the policy output of the Nth agent at time step t, the environment transitions to a new state according to the state transition distribution P and provides a shared reward; each agent receives an observation from O. And maintain an observation-action history record.

3. The method for dynamic manipulation of a single robot based on multi-agent reinforcement learning according to claim 1, characterized in that, The state information in step 2 is divided into: local observations, shared observations, and global observations. The local observations are state information perceived independently by each agent, while the shared observations are global information shared by all agents. The global observations are the overall state information of the robot received by the global critic network Critic, including the motor position and speed of each limb, previous and current actions, foot contact state, gait sequencing triangular timing guidance controller, projected gravity, high-level control commands generated by the upper-level strategy, body posture and speed, end effector state, visual information, and external disturbance information.

4. The method for dynamic manipulation of a single robot based on multi-agent reinforcement learning according to claim 1, characterized in that, Step 3 specifically involves: For a quadrupedal locomotion system, the input local cascaded observations include: leg joint movement positions. ,speed Actions at time step t-1 and the action of time step t Gait sequencing triangular timing guidance controller Projected gravity Command value fuselage attitude Aircraft speed And agent one-hot encoding ; Both the first Actor network and the second Actor network consist of a normalization layer, a multilayer perceptron (MLP) layer, a gated recurrent unit (GRU) layer, and a normalization layer. An offset phase is introduced at the output of the first Actor network and the second Actor network to correct the difference between the target joint position and the actual joint position. For robotic arm manipulation systems, color-depth RGB-D images from the end effector's perspective are encoded using PointNet and used as one of the inputs to the robotic arm agent. The state inputs of the robotic arm agent include: leg joint motion positions. ,speed Actions at time step t-1 and the action of time step t End effector status The color-depth RGB-D image encoded by PointNet is the visual information. External disturbance information High-level control commands generated from upper-level strategies Projected gravity fuselage attitude Aircraft speed And agent one-hot encoding ; The first Actor network outputs torque or position commands for the three joints of each leg; the second Actor network outputs control commands for the six joints of the robotic arm and opening / closing commands for the end effector. The Critic network adopts the same MLP / GRU network structure as the Actor network, replacing the last normalization layer with a PopArt layer. It takes global observations as input, including motor position, motor speed, previous action, current action, plantar contact state, gait sequencing triangular timing guidance controller, projected gravity, high-level control commands generated by upper-level strategies, fuselage attitude, fuselage speed, end effector state, visual information, and external disturbance information. The Critic network outputs continuous state values.

5. The method for dynamic manipulation of a single robot based on multi-agent reinforcement learning according to claim 2, characterized in that, The end effector speed tracking reward formula described in step 4 is as follows: （1）； Among them, speed error This indicates the error between the current speed and the commanded speed. , This refers to the current linear or angular velocity of the robot base along the x-axis or y-axis. The target linear velocity or angular velocity given by the command. It is a hyperparameter; The end effector pose tracking reward calculates the current pose of the end effector in the base coordinate system in real time, including the position LPY and end pose RPY in the spherical coordinate system, and the geometric error between it and the target pose given by the task command. A negative reward is applied to this geometric error, namely the robot arm command tracking error penalty, which is as follows: （2）； in, and These are the weights of the LPY and RPY errors read from the configuration respectively. For LPY normalization error, , This is the current end effector LPY value. It is the target LPY value given by the instruction. Each of the LPY instruction ranges is unique. For RPY normalization error, ; The layered decoupling reward mechanism also integrates a series of auxiliary rewards and penalties, including: (1) Motion smoothness penalty: The formula is as follows: （3）； （4）； （5）； （6）； in, It is the angular velocity of the i-th joint. It is the angular acceleration of the i-th joint. It is the action instruction output by the j-th executor at time step t. It is the total number of joint degrees of freedom. These are the weighting coefficients in the reward function; (2) Energy consumption penalty: Penalize the motor output torque to guide all agents to learn energy-saving gaits and actions with lower energy consumption. The formula is as follows: （7）； in, It is the motor torque applied to the i-th joint; (3) Behavioral constraints and safety penalties: The formula is as follows: （8）； （9）； （10）； in, It is a collection of key body parts that need to be avoided from collisions. It is the contact force at part k. It is a preset contact force threshold. It is an activation function that takes the value 1 when the condition is true and 0 otherwise. θ and θ are the current roll angle and pitch angle of the aircraft, respectively. and It is the attitude angle expected by the command; These are the weighting coefficients in the reward function; (4) Base motion tracking reward: The formula is as follows: （11）； （12）； （13）； Among them, v xy It is the current linear velocity vector of the base in the xy plane. It is the desired linear velocity vector. It is a hyperparameter that controls reward sensitivity; It is the current angular velocity of the base around the z-axis. It is the desired angular velocity. The corresponding sensitivity parameters; These are the weighting coefficients in the reward function.

6. The method for dynamic manipulation of a single robot based on multi-agent reinforcement learning according to claim 4, characterized in that, Step 5 specifically involves: In multi-agent proximal policy optimization, each agent Have an independent strategy , where o i Let a represent the local observation of the i-th agent. i Let represent the action of the i-th agent; in a paradigm of centralized training and distributed execution, a shared global state-value function is used. A unified evaluation is conducted, and the advantage function is calculated based on this to guide policy updates; the core optimization objective function of MAPPO is: (14); in, The probability ratio between the old and new strategies. It is the old policy before the update, and θ is the set of parameters of the policy network; It represents the expectation over time step t; clip(•) is the clipping function in the PPO algorithm; Based on global state value function The estimated advantage function, s t ε represents the current global state; ε is the pruning threshold. In multi-agent scenarios, the policy parameters of the agents are updated independently, but they all rely on the shared global state value function for advantage estimation. Therefore, the joint optimization objective is written as the sum of the optimization objectives of all agents to ensure stable and efficient collaborative learning within a unified framework. （15）； Where N is the total number of agents.

7. The method for dynamic manipulation of a single robot based on multi-agent reinforcement learning according to claim 6, implemented through the following system, characterized in that: include: State awareness module, strategy reasoning module, PD control module; The state perception module receives image data from an RGB-D camera mounted on the end effector, and the PointNet network encodes the visual information of the target pose. Simultaneously, it combines sensor information from various robot parts, including joint motor positions, speeds, motion history, body posture and speed, projected gravity, gait sequencing triangular timing guidance controller, high-level control commands generated by upper-level strategies, end effector state, foot contact state, and external disturbance information, to fuse multi-source data and output encoded local observations, shared observations, and global observations. The local observations are input to the Actor networks of the four leg agents of the quadrupedal motion system and the robotic arm agent of the robotic arm manipulation system, respectively. Shared observations are used for collaborative information exchange between agents, while global observations are input to a unified Critic network for state value assessment. The policy reasoning module generates leg joint commands and robotic arm joint and end effector grasping commands based on the observation input of each agent using the MAPPO algorithm. The Actor network is responsible for action output, and the Critic network evaluates and guides policy updates based on the global state value function. The PD control module compares the joint torque commands output by the Actor network in the strategy reasoning module with the actual joint state, performs error correction and dynamic compensation through proportional-derivative control law, generates stable motor drive signals, and sends them down to the quadruped, robotic arm and end effector, thereby ensuring the execution accuracy and stability of the robot's coordinated movement and dynamic grasping in complex environments.

8. An electronic device, characterized in that, include: One or more processors, and a memory for storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the single-robot dynamic manipulation method based on multi-agent reinforcement learning as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed, cause the processor to perform the single-robot dynamic manipulation method based on multi-agent reinforcement learning as described in any one of claims 1-7.

10. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the single-robot dynamic manipulation method based on multi-agent reinforcement learning as described in any one of claims 1-7.

Images