A double-layer intelligent agent optimization method, device and equipment for a multi-robot arm and a storage medium

Abstract

The application provides a double-layer intelligent agent optimization method and device for a multi-robot arm, equipment and a storage medium. By introducing an RRT-Connect algorithm and a B-spline trajectory smoothing optimization in a high layer, global path planning is performed on environment information of a carrying task, a kinematic chain model is used to convert a smoothed centroid trajectory into a reference trajectory sequence of each robot arm end, the reference trajectory is used as a bottom-layer distributed real-time control guide signal driven by a MAPPO algorithm, a multi-layer reward function is combined to comprehensively feed back task completion, motion smoothness and coordination, and each robot arm intelligent agent is guided to cooperatively adjust joint actions in a tracking process, so that seamless connection of global planning and local control is realized, and the smoothness and cooperation efficiency of carrying motion are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent agent optimization, and in particular to a two-layer intelligent agent optimization method, apparatus, device, and storage medium for multi-robotic arms. Background Technology

[0002] Collaborative handling of large or heavy-duty objects by multiple robotic arms is an important research direction in the field of robotics, with broad application prospects in industrial automation, intelligent logistics, and aerospace assembly. Compared to single-arm operation, multi-arm collaboration can overcome the limitations of a single robotic arm in terms of load capacity, workspace, and operational flexibility, enabling stable handling of large, heavy-duty, or irregularly shaped objects. However, achieving efficient, stable, and intelligent multi-arm collaborative handling still faces many technical challenges. Existing technologies mainly suffer from the following shortcomings:

[0003] 1. While traditional model-based control methods (such as PID control and sliding mode control) can achieve precise control under ideal conditions, their performance heavily depends on accurate system dynamics models. In complex, highly coupled systems such as multi-arm cooperative systems, it is difficult to establish accurate models, and the systems are susceptible to uncertainties such as load changes and external disturbances, resulting in poor robustness and environmental adaptability of these methods.

[0004] 2. Multi-Agent Reinforcement Learning (MARL), as a data-driven method, does not rely on precise models and theoretically can learn strategies to adapt to complex dynamic environments through interaction with the environment. However, when directly applied to multi-arm cooperative handling, it has some shortcomings: low learning efficiency and difficulty in convergence. The huge state-action space leads to an extremely large number of samples required for training, and simultaneous learning by multiple agents can easily cause training instability and difficulty in convergence; lack of global guidance and poor motion quality. MARL is usually based on local observation decisions and lacks global task planning, which can easily lead to uneven, inefficient, and even jittery motion trajectories, affecting the stability of handling. The fundamental reason is that exploring in a huge, unstructured joint action space, without macroscopic, physically constrained path guidance, makes it extremely difficult for reinforcement learning agents to learn complex strategies that balance global optimization and local smoothness.

[0005] 3. Hierarchical reinforcement learning (HRL) attempts to improve the efficiency of MARL through task decomposition, but its hierarchical structure and subtask division often rely on expert prior knowledge, lacking generality and adaptability to dynamic environments. For specific tasks such as multi-arm cooperative handling, which have extremely high requirements for smoothness, real-time performance, and high-precision coordination, existing HRL research is still insufficient.

[0006] Furthermore, path planning algorithms (such as RRT and its variants) can effectively generate collision-free geometric paths, but they mainly solve static path planning problems and cannot directly handle the dynamic control, real-time adaptation, and learning optimization of cooperative strategies in multi-arm cooperative systems.

[0007] In view of the above, this application is hereby submitted. Summary of the Invention

[0008] This invention discloses a two-layer intelligent agent optimization method, device, equipment, and storage medium for multi-robotic arms, aiming to solve the problems of disconnect between global path planning and local intelligent agent control, uneven motion, and low collaborative efficiency in existing multi-robotic arm collaborative handling tasks.

[0009] The first embodiment of the present invention provides a two-layer intelligent agent optimization method for multi-robotic arms, comprising:

[0010] The environmental information of the multi-robotic arm collaborative handling task is obtained, and the high-level task planner is called to process the environmental information. The high-level task planner uses the RRT-Connect algorithm to explore the initial path, combines B-splines to optimize the trajectory smoothing of the initial path, and converts the object's centroid trajectory into a reference trajectory sequence for each robotic arm end effector through a kinematic chain model. The environmental information includes the initial pose of the target object, the target pose, and the environmental map.

[0011] The reference trajectory sequence is used as a guide signal input to the underlying motion actuator, wherein the underlying motion actuator consists of multiple independent robotic arm agents. The MAPPO algorithm is invoked to perform distributed real-time control on each robotic arm agent based on the reference trajectory sequence, and decisions are made by combining the local observations of each robotic arm agent to generate control commands. The guide signal is a part of the local observations of each robotic arm agent.

[0012] The execution effect of the control command is obtained, and feedback is given to the underlying motion actuator through a preset multi-layer reward function based on the execution effect, so as to guide each robotic arm agent to learn the collaborative control strategy, thereby enabling multiple robotic arms to effectively track the reference trajectory sequence and smooth the movement of each robotic arm during the tracking process. The multi-layer reward function includes task completion reward, smoothness reward and coordination reward, and the guidance signal is the input of the task completion reward.

[0013] Preferably, the step of converting the trajectory of the object's center of mass into a reference trajectory sequence for each robotic arm end effector using a kinematic chain model specifically involves:

[0014] Based on the predefined grasping relationship between each robotic arm end effector and the object's center of mass, the desired world coordinate system pose of each end effector is calculated using the following kinematic transformation relationship as the reference trajectory sequence, wherein the motion chain from the workspace to the object's center point is represented as:

[0015]

[0016] The kinematic chain from the center point of the object to the i-th gripping point of the robotic arm is represented as:

[0017]

[0018] in, These are the x, y, and z coordinates of the object's center of mass in the world coordinate system. These are the roll angle, pitch angle, and yaw angle of the object's center of mass, respectively. These are the x, y, and z coordinate offsets of the i-th robotic arm grasping point relative to the object's center of mass, respectively. These represent the angular offsets of the i-th robotic arm grasping point relative to the object's center of mass. Represents the translation transformation function. Represents the rotation transformation function;

[0019] Preferably, the state space of each robotic arm agent's local observation is as follows:

[0020]

[0021] in, Represents robotic arm The pose of the capture point includes the x, y, and z coordinates and the roll, pitch, and yaw angles; Represents the i-th robotic arm The speed of the grab point; Configure the current angles of the 6 joints of the i-th robotic arm; The current pose of the center of mass of the object being moved; The current velocity of the object's center of mass; The center position of the obstacle in the environment. The dimensions are those of a square obstacle.

[0022] Preferably, the multi-layer reward function includes high-level rewards and low-level rewards;

[0023] Senior management rewards include:

[0024] Task completion reward ),in, The current position of the object's center of mass. For the target location, This is the position reward coefficient. This is the distance attenuation parameter;

[0025] Smoothness of object motion reward , ,in, Let t be the velocity vector of the object's center of mass at time t, and T be the total number of time steps. Let be the acceleration vector of the object's center of mass at time t. This refers to the speed deviation between the end effector of the robotic arm and the center of the object being transported. This refers to the acceleration deviation between the end effector of the robotic arm and the center of the object being transported.

[0026] The base rewards include:

[0027] Grab point distance stability bonus ,in, It is the first The distance from the point where the robotic arm grasps the object to its center of mass. It is the average distance from all grab points to the centroid;

[0028] Force balance reward ,in, Indicates the first The force applied by the robotic arm, This is the average of the forces exerted by all robotic arms.

[0029] Joint movement smoothness reward , ,in, Indicates the first The first robotic arm Each joint in time angular velocity, Indicates the first The first robotic arm Each joint in time The angular acceleration is N, where N is the number of robotic arms and M is the number of joints in a single robotic arm.

[0030] Preferably, the step of guiding each robotic arm agent to learn a collaborative control strategy, thereby enabling multiple robotic arms to effectively track the reference trajectory sequence, specifically involves:

[0031] Within each time step t, each agent Obtain its local observations and current reference pose It selects actions through its Actor network, where each agent's architecture is an Actor-Critic architecture;

[0032] The joint action of all agents The data is sent to the physics engine for calculation, which updates the states of the robotic arm and the object, and returns the joint observation for the next time step. and the joint reward calculated based on the reward function ;

[0033] empirical tuples The data is stored in an experience replay buffer, and data is periodically sampled from the buffer to update the parameters of the Actor and Critic networks. The learning process is repeated to enable each agent to gradually learn to track the reference trajectory and achieve effective cooperation.

[0034] Preferably, it further includes:

[0035] Each robotic arm agent's Actor network relies only on its local observations. To make decisions, each robotic arm agent is equipped with an independent policy network. The policy of the Actor network is evaluated by learning the global Critic network and based on the global state. Learning center value function .

[0036] Policy network for each agent It consists of three hidden layers, which will enable local observation. The mapping is represented by an action probability distribution. The output layer uses a tanh activation function to ensure that the joint angle adjustment is within the constraints of the robotic arm's operation. The objective function for optimizing the Actor network in the MAPPO algorithm is:

[0037]

[0038] The loss function minimized by the Critic network is:

[0039]

[0040] in, This represents the local observation state of the i-th agent. This represents the parameters of the i-th agent (Actor) network. This represents the action output by the i-th agent. This represents the global state information at time t. The parameters representing the Critic network, This indicates the global state of the Critic network. The value estimate, This represents the probability ratio between the new strategy and the old strategy. Here, T represents the estimated value of the dominance function, and T represents the total number of time steps. The instant reward at time t, This is the discount factor.

[0041] A second embodiment of the present invention provides a two-layer intelligent agent optimization device for multi-robotic arms, comprising:

[0042] The high-level task processing unit is used to acquire environmental information of the multi-robotic arm collaborative handling task, and call the high-level task planner to process the environmental information. The high-level task planner uses the RRT-Connect algorithm to explore the initial path, combines B-splines to optimize the trajectory smoothing of the initial path, and converts the trajectory of the object's centroid into a reference trajectory sequence of each robotic arm end effector through a kinematic chain model. The environmental information includes the initial pose of the target object, the target pose, and the environmental map.

[0043] The underlying task processing unit is used to input the reference trajectory sequence as a guidance signal into the underlying motion actuator, wherein the underlying motion actuator is composed of multiple independent robotic arm agents. The MAPPO algorithm is invoked to perform distributed real-time control on each robotic arm agent based on the reference trajectory sequence, and decisions are made by combining the local observations of each robotic arm agent to generate control commands. The reference trajectory sequence is used to compare with the actual trajectory, and the deviation is used to guide the agent to learn. The guidance signal is a part of the local observations of each robotic arm agent.

[0044] The feedback unit is used to obtain the execution effect of the control command and provide feedback to the underlying motion actuator based on the execution effect through a preset multi-layer reward function, so as to guide each of the robotic arm agents to learn the cooperative control strategy, thereby enabling multiple robotic arms to effectively track the reference trajectory sequence and smooth the movement of each robotic arm during the tracking process. The multi-layer reward function includes task completion reward, smoothness reward and coordination reward, and the guidance signal is the input of the task completion reward.

[0045] The third embodiment of the present invention provides a two-layer intelligent agent optimization device for multiple robotic arms, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement a two-layer intelligent agent optimization method for multiple robotic arms as described in any of the above embodiments.

[0046] The fourth embodiment of the present invention provides a computer-readable storage medium storing a computer program, which can be executed by a processor of the device in which the computer-readable storage medium is located, to implement a two-layer intelligent agent optimization method for multiple robotic arms as described in any of the above claims.

[0047] Based on the two-layer intelligent agent optimization method, device, equipment, and storage medium for multi-robotic arms provided by this invention, global path planning is performed on the environmental information of the handling task by introducing the RRT-Connect algorithm and B-spline trajectory smoothing optimization at a high level. The smoothed centroid trajectory is converted into a reference trajectory sequence for the end effector of each robotic arm using a kinematic chain model. This reference trajectory is then used as the bottom-level distributed real-time control guidance signal driven by the MAPPO algorithm. Combined with a multi-layer reward function, the task completion degree, motion smoothness, and coordination are comprehensively fed back, guiding each robotic arm intelligent agent to coordinately adjust joint movements during the tracking process. This achieves seamless connection between global planning and local control, significantly improving the smoothness and coordination efficiency of the handling motion. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating a two-layer intelligent agent optimization method for multiple robotic arms provided in the first embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the kinematics chain of the multi-arm cooperative handling system provided by the present invention;

[0050] Figure 3 This is a schematic diagram of a dual-arm collaborative handling task scenario provided by the present invention;

[0051] Figure 4 This is a schematic diagram comparing the trajectory of the robotic arm end effector and the trajectory of the target object's center of mass using the MAPPO method.

[0052] Figure 5 This is a schematic diagram of the trajectory of the end effector of the robotic arm and the trajectory of the centroid of the target object based on H-MAPPO, provided by the present invention.

[0053] Figure 6 This is a schematic diagram of the angular velocity and angular acceleration curves of the robotic arm joints compared to the MAPPO method;

[0054] Figure 7 This is a schematic diagram of the angular velocity and angular acceleration curves of the H-MAPPO robotic arm joints provided by the present invention;

[0055] Figure 8 These are schematic diagrams illustrating the reward and loss curves for the two methods;

[0056] Figure 9 This is a schematic diagram of a module of a dual-layer intelligent agent optimization device for multiple robotic arms provided in the second embodiment of the present invention. Detailed Implementation

[0057] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0058] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0059] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0060] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0061] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0062] The terms "first" and "second" used in the embodiments are merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permissible. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein.

[0063] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0064] The first embodiment of the present invention provides a two-layer intelligent agent optimization method for multi-robotic arms, which can be executed by an optimization device (or system), specifically by one or more processors within the optimization device, to at least implement the following steps:

[0065] S101, Obtain environmental information for the multi-robotic arm collaborative handling task, and call the high-level task planner to process the environmental information. The high-level task planner uses the RRT-Connect algorithm for initial path exploration (in other optional implementations, path exploration can also use other graph search algorithms such as A* and D*, which are not specifically limited here, but these schemes are all within the protection scope of this invention), combines B-splines to perform trajectory smoothing optimization on the initial path, and converts the object's centroid trajectory into a reference trajectory sequence for each robotic arm end effector through a kinematic chain model. The environmental information includes the initial pose of the target object, the target pose, and the environmental map.

[0066] In this embodiment, the optimization device can be a desktop computer, workstation, server or other terminal with data processing capabilities, which can establish communication with multiple robotic arms. The optimization device can be equipped with a corresponding operating system and application software, and the functions required in this embodiment can be realized through the combination of the operating system and application software. The reference trajectory sequence is used to compare with the actual trajectory, and the deviation is used to guide the intelligent agent to learn.

[0067] It should be noted that during the implementation of multi-robotic arm collaborative handling tasks, the system first acquires complete environmental information, including the initial pose of the target object, the target pose, and an environmental map. After receiving this environmental information, the high-level task planner uses the RRT-Connect algorithm to explore the initial path in the configuration space. This algorithm quickly finds a feasible geometric path from the initial pose to the target pose by simultaneously growing two random trees from the start and end points and attempting to connect them. Since the path generated by the RRT-Connect algorithm is usually a polyline form connected by line segments, it has non-smooth inflection points. Therefore, the system further uses the B-spline interpolation method to optimize the trajectory smoothing of the initial path, transforming the discrete path points into a time-parameterized continuous and smooth trajectory.

[0068] After obtaining a smooth trajectory of the object's center of mass, the system converts this trajectory into a sequence of reference trajectories for each robotic arm's end effector using a kinematic chain model. This process is based on a predefined grasping relationship between each robotic arm's end effector and the object's center of mass, and establishes a complete kinematic mapping relationship using a homogeneous coordinate transformation matrix.

[0069] Please combine Figure 2 World coordinate system {X O ,Y O Z O}, serving as the global reference coordinate system, the object's centroid coordinate system {X C ,Y C Z C}, with the center of mass of the object being transported as the origin, the coordinate system of the robot arm base {x1i ,y 1i ,z 1i}, the base coordinate system of each robotic arm, and the end effector coordinate system of the robotic arm {x ei ,y ei ,z ei}, coordinate system of robotic arm end effector, key transformation relationships Transformation vector from the world coordinate system to the object's center of mass. Transformation vector from the object's center of mass to the i-th gripping point of the robotic arm. The transformation vector from the end effector of the robotic arm to the base. The orange ellipse represents the object being moved. The two robotic arms grasp the object from different positions. Each robotic arm has its own base and end effector. All coordinate systems are connected by a kinematic chain.

[0070] Kinematic chain from workspace to object center point The homogeneous transformation matrix represents it as follows:

[0071]

[0072] in, These are the x, y, and z coordinates of the object's center of mass in the world coordinate system. These are the roll angle, pitch angle, and yaw angle of the object's center of mass, respectively. Based on the pose information of the object's center of mass, the system further calculates the kinematic transformation relationship from the object's center point to the i-th robotic arm gripping point (the kinematic chain from the object's center point to the gripping point). This transformation is expressed by the following matrix:

[0073]

[0074] In this transformation, These are the x, y, and z coordinate offsets of the i-th robotic arm grasping point relative to the object's center of mass, respectively. These are the angular offsets of the i-th robotic arm gripping point relative to the object's center of mass. These parameters are predetermined based on the object's geometric characteristics and gripping strategy.

[0075] To establish a complete kinematic chain, the system also needs to determine the kinematic relationship between the robot arm's end effector and the base. Through a similar transformation, the kinematic chain from the robot arm's end effector to the base can be obtained. :

[0076]

[0077] By cascading and combining these kinematic transformations, the system can establish a complete kinematic mapping from the world coordinate system to the end effectors of each robotic arm. Given the trajectory of the object's center of mass... At this time, the system can calculate the expected pose trajectory of each end effector in the world coordinate system by inverse kinematic chain calculation. This trajectory sequence is the reference trajectory that each robotic arm needs to track. The entire kinematic mapping process ensures that the end effectors of each robotic arm maintain a constant relative position to the center of mass of the object, thereby achieving stable collaborative grasping and handling operations.

[0078] S102, the reference trajectory sequence is used as a guide signal to be input into the underlying motion actuator, wherein the underlying motion actuator is composed of multiple independent robotic arm agents, the MAPPO algorithm is called to perform distributed real-time control on each robotic arm agent based on the reference trajectory sequence, and the decision is made by combining the local observations of each robotic arm agent to generate control commands.

[0079] It should be noted that after the high-level task planner generates the reference trajectory sequence, the system uses these trajectory sequences as guiding signals to input into the low-level motion actuator for distributed control. The low-level motion actuator adopts a multi-agent reinforcement learning framework, consisting of N independent robotic arm agents, each controlling one robotic arm. The system calls the MAPPO (Multi-Agent Proximal Policy Optimization) algorithm to perform distributed real-time control on each robotic arm agent. This algorithm combines the stability advantages of the PPO algorithm in a single-agent environment with the collaborative learning capabilities of a multi-agent system.

[0080] During the implementation of the MAPPO algorithm, each robotic arm agent needs to not only track the reference trajectory sequence provided by the high-level planner, but also... Furthermore, real-time decision-making must be based on the robot's own local observation information. The system defines a comprehensive state space for each agent, which includes multi-dimensional data such as the robot's own state, the target object's state, and environmental information. Specifically, the local observation state space of the i-th robot agent is defined as:

[0081]

[0082] In this state space definition Represents robotic arm The pose of the capture point includes the x, y, and z coordinates and the roll, pitch, and yaw angles;

[0083] Speed ​​information through The vector representation contains the instantaneous velocity components of the i-th robotic arm grasping point in the three coordinate axes, providing dynamic feedback information for the motion control of the intelligent agent.

[0084] The internal configuration of the robotic arm is determined by the joint angle vector. The six-dimensional vector corresponds to the current angle configuration of each joint in a standard six-DOF industrial robotic arm (such as the UR5), directly reflecting the internal motion state of the arm. To achieve effective cooperative control, each agent also needs to perceive the state information of the object being transported, including the current pose of the object's center of mass. and center of mass velocity This information enables each agent to understand the overall execution status of the transport task and adjust its own control strategy accordingly.

[0085] Considering the complexity of the environment in practical applications, the system also includes environmental obstacle information in the state space. This represents the coordinates of the center position of an obstacle in the environment, while This describes the size parameters of the square obstacle, providing the agent with the necessary information for obstacle avoidance decisions. Based on this complete state space definition, each agent operates through its independent policy network. Map the current observed state to specific control actions, where This represents the parameters of the policy network for the i-th agent. Indicates the output control commands.

[0086] S103, obtain the execution effect of the control command, and provide feedback to the underlying motion actuator based on the execution effect through a preset multi-layer reward function, so as to guide each of the robotic arm agents to learn the collaborative control strategy, thereby enabling multiple robotic arms to effectively track the reference trajectory sequence and smooth the movement of each robotic arm during the tracking process. The multi-layer reward function includes task completion reward, smoothness reward, and coordination reward.

[0087] It should be noted that after the underlying motion actuators generate control commands, the system needs to continuously acquire the execution effects of these control commands to evaluate the motion state and collaborative performance of each robotic arm through feedback from the actual system. Based on the execution effects, the system provides real-time feedback to the underlying motion actuators through a preset multi-layered reward function. This feedback guides each robotic arm agent to learn collaborative control strategies. Furthermore, the multi-layered reward function is divided into two levels: high-level rewards and low-level rewards, corresponding to the global task objective and the optimization requirements of local motion quality, respectively.

[0088] High-level rewards primarily focus on the overall task completion and the global characteristics of the object's motion. The task completion reward is defined using an exponential decay function:

[0089] )

[0090] in, The current position of the object's center of mass. For the target location, The positional reward coefficient is used to adjust the absolute size of the reward. The distance decay parameter controls the sensitivity of reward to changes in distance. This effectively ensures that the agent receives a higher reward as the object gets closer to the target position, thus effectively guiding cooperative transport in the correct direction.

[0091] To ensure smooth object movement during handling, a smoothness reward is configured, which is achieved by penalizing drastic changes in the object's movement.

[0092] , ,in, Let t be the velocity vector of the object's center of mass at time t, and T be the total number of time steps. Let be the acceleration vector of the object's center of mass at time t. This refers to the speed deviation between the end effector of the robotic arm and the center of the object being transported. In this embodiment, the acceleration deviation between the end effector of the robotic arm and the center of the object being transported is considered. Improve motion smoothness by penalizing rapid changes in the velocity of the object's center of mass (i.e., the jerkiness of the center of mass's motion). The smoothness of motion is further improved by penalizing rapid changes in the acceleration of the object's center of mass.

[0093] The underlying reward focuses on the coordination between the robotic arms and the smoothness of individual arm movements. The gripping point distance stability reward is used to maintain a stable gripping posture for each robotic arm relative to the object's center of mass.

[0094]

[0095] in, It is the first The distance from the point where the robotic arm grasps the object to its center of mass. It is the average distance from all gripping points to the center of mass, and N is the total number of robotic arms. This reward encourages the robotic arms to maintain a stable relative position by minimizing the variance of the distances between gripping points.

[0096] To ensure that the forces exerted by each robotic arm on the object remain balanced, the system is designed with a force balance reward:

[0097]

[0098] in, Indicates the first The force applied by the robotic arm, It is the average force of all robotic arms, which can effectively avoid system instability caused by a single robotic arm bearing too much load;

[0099] Joint-level motion smoothness is achieved through the following reward function:

[0100]

[0101]

[0102] in, Indicates the first The first robotic arm Each joint in time angular velocity, Indicates the first The first robotic arm Each joint in time The angular acceleration is given by N, the number of robotic arms, and M, the number of joints in a single robotic arm. In this embodiment, smoothness of robotic arm movement is promoted by penalizing abrupt changes in joint motion.

[0103] In guiding each robotic arm agent to learn the cooperative control strategy, the system employs a cyclic learning mechanism. Within each time step t, each agent i first acquires its local observations. and current reference pose Then through its Actor network Select the corresponding action Each agent adopts an Actor-Critic architecture, where the Actor network is responsible for policy decision-making and the Critic network is responsible for value evaluation.

[0104] The system coordinates the actions of all intelligent agents. The data is sent to the physics engine for dynamic calculations. The physics engine updates the states of the robotic arm and the object based on the current state and actions, and returns the joint observations for the next time step, along with the joint reward calculated using a multi-layered reward function. The system will then provide the complete experience tuple. Stored in the experience replay buffer, where This represents the reference trajectory information for each robotic arm at time t. By periodically sampling historical experience data from the buffer, the system uses the MAPPO algorithm to update the Actor and Critic network parameters of each agent. Through a cyclic learning process, each agent gradually learns to accurately track the reference trajectory. Simultaneously, guided by multi-layered reward functions, effective cooperative control is achieved, ultimately reaching the overall goal of smoothly transporting the target object.

[0105] In one possible implementation of the present invention, it further includes:

[0106] In the specific implementation of the MAPPO algorithm, the system adopts a distributed decision-making but centralized training architecture. Each robotic arm agent's Actor network strictly adheres to the principle of local observation, relying only on its own perceived local state information. Decision-making ensures the distributed nature and scalability of the system during actual deployment. Each robotic arm agent is equipped with an independent policy network. ,in This represents the parameter vector of the i-th agent network, which is continuously updated during the training process to optimize the control strategy.

[0107] To achieve effective collaborative learning while maintaining decision-making independence, the system designs a shared global Critic network to evaluate the quality of each Actor's network policies. This design allows each agent to gain an understanding of the overall system state by learning the global Critic network, thereby making more coordinated decisions. The global Critic network is based on the complete system state. Learning center value function , This represents the parameters of the Critic network, and the function can accurately evaluate the long-term cumulative reward expectation of the system under the current global state.

[0108] In terms of the specific design of the network architecture, the policy network of each agent adopts a deep neural network structure, consisting of three hidden layers. Each hidden layer contains an appropriate number of neurons to ensure sufficient expressive power. The input layer of the network receives local observation states. After nonlinear transformation through three hidden layers, the local observation information is mapped to an action probability distribution. To ensure that the output control action conforms to the physical constraints of the robotic arm, the output layer uses the tanh activation function to restrict the network output to the interval [-1,1]. Then, appropriate scaling and offset operations are used to ensure that the joint angle adjustment is always within the operating limits of the robotic arm.

[0109] The MAPPO algorithm updates the policies of each agent by optimizing a specific objective function:

[0110]

[0111] In this objective function, This is the probability ratio of the new strategy to the old strategy, reflecting the magnitude of the strategy update. The estimated value of the advantage function

[0112] Used to measure the state Take action below The degree of superiority or inferiority relative to the average level, For the clipping parameters, the clipping operation limits the probability ratio to [ To prevent overly aggressive policy updates, the value is set to 0.2 within a certain range.

[0113] In contrast to the policy optimization of the Actor network, the Critic network learns an accurate value function by minimizing the temporal difference error. The loss function of the Critic network is defined as:

[0114]

[0115] In this loss function, This shows the Critic network's current global state. The value estimate, The instantaneous reward obtained by the agent at time t. The discount factor is used to balance the importance of immediate rewards and future rewards, and is typically set between 0.95 and 0.99. The expression represents the value estimate of the next state. This constitutes the objective estimate of the true value of the current state. During actual training, the system employs an experience replay mechanism to stabilize the learning process. Experience tuples collected at each time step are stored in a shared experience buffer. During training, batch data is sampled from the buffer to update the network parameters. The Actor network improves the policy by maximizing the aforementioned objective function, enabling the agent to adapt to environmental changes and optimize cooperative behavior while following the reference trajectory. The Critic network provides a more accurate policy evaluation by minimizing the value estimation error, offering reliable guidance for updating the Actor network.

[0116] By combining the Actor-Critic architecture and the MAPPO algorithm, each robotic arm agent can learn a high-quality cooperative control strategy while maintaining decision-making independence. It can accurately track the reference trajectory provided by the high-level planner and perform local optimization based on real-time feedback during execution, ultimately achieving smooth, stable, and coordinated multi-robotic arm cooperative handling control. In other optional implementation methods, other multi-agent reinforcement learning algorithms such as MADDPG and QMIX can also be used at the bottom layer. No specific limitation is made here, but these schemes are all within the protection scope of this invention.

[0117] To verify the effectiveness of the method of the present invention (denoted as H-MAPPO), simulation experiments were conducted and compared with the benchmark end-to-end multi-agent reinforcement learning method (MAPPO).

[0118] Experimental setup:

[0119] Simulation platform: CoppeliaSimEdu4.5.

[0120] Programming and Frameworks: Python language, using PyTorch to implement neural network models, and combining OpenAIGym to define the reinforcement learning environment interface. Communication with the simulator is achieved through the CoppeliaSimRemoteAPI.

[0121] Hardware: Training and testing were conducted on a personal computer equipped with an Intel Core i5-11400H CPU and an NVIDIA RTX 3050 Laptop GPU.

[0122] Task scenario (reference) Figure 3 This experiment simulates the coordinated movement of a large wooden bench slat (1580x600x40mm, 15.1kg) using two UR5 robotic arms. The initial pose of the object (position (0.42, 0.0, 0.26) meters and orientation (0.0, 0.0, -90.0) degrees in world coordinates) and the target pose (in this comparative experiment, the target pose may be the same as the initial pose to test the ability to stably grasp and maintain the orientation, or a moving target may be set) are both included. Obstacles may be present in the environment.

[0123] Comparison method: End-to-end training was performed using a benchmark MAPPO algorithm with the same network structure and hyperparameter settings as the underlying controller in the H-MAPPO method of this invention.

[0124] Evaluation metrics: (1) Task success rate (not directly given in the attached figure, but as an implicit evaluation); (2) Trajectory smoothness (by visually observing the motion trajectory of the end of the robotic arm and the center of mass of the object); (3) Joint motion stability (by recording and plotting the joint angular velocity and angular acceleration curves); (4) Training convergence speed (by plotting the average round reward curve during the training process).

[0125] Experimental results:

[0126] Trajectory Comparison: Figure 4 The diagram shows the motion trajectories of the two robotic arm end effectors (red and blue lines) and the object's center of mass (green line) during task execution after training using the benchmark MAPPO method. It can be seen that the trajectories exhibit significant twists and turns and discontinuities, indicating unstable motion. In contrast, Figure 5 The diagram shows the corresponding trajectories generated after training using the H-MAPPO method of this invention. These trajectories are significantly smoother, more direct, and more coherent, demonstrating better motion quality.

[0127] Comparison of joint movements: Figure 6The curves showing the angular velocity and angular acceleration of each joint (e.g., joints 1-6) of the two robotic arms as a function of time under the baseline MAPPO method are presented. It can be seen that the curves contain many sharp peaks and violent fluctuations, indicating frequent acceleration and deceleration of joint movements and significant impact. Figure 7 The curves corresponding to the H-MAPPO method of this invention are shown. The peak values ​​of these curves are significantly reduced, and the fluctuations are smoother, indicating that the joint movements are more stable and gentler.

[0128] Training performance comparison: Figure 8 The diagram illustrates the variation of the average epoch reward during training for the H-MAPPO method (e.g., the red curve in the figure) and the baseline MAPPO method (e.g., the blue curve in the figure) as a function of the number of training epochs (or time steps). It can be seen that the reward curve of the H-MAPPO method rises faster and eventually converges to a higher reward level than the baseline MAPPO method. This indicates that the method of the present invention has higher learning efficiency and better final performance.

[0129] Conclusion: The above comparative experimental results strongly demonstrate that the proposed two-layer multi-agent reinforcement learning method (H-MAPPO) can significantly improve the smoothness and stability of motion, while improving training efficiency and final performance, compared with the traditional end-to-end MARL method (MAPPO), in multi-robotic arm collaborative handling of large objects.

[0130] Please see Figure 9 The second embodiment of the present invention provides a two-layer intelligent agent optimization device for multiple robotic arms, comprising:

[0131] High-level task processing unit 201 is used to acquire environmental information of a multi-robotic arm collaborative handling task, and to call a high-level task planner to process the environmental information. The high-level task planner uses the RRT-Connect algorithm to explore the initial path, combines B-splines to optimize the trajectory smoothing of the initial path, and converts the trajectory of the object's centroid into a reference trajectory sequence for each robotic arm end effector through a kinematic chain model. The environmental information includes the initial pose of the target object, the target pose, and the environmental map.

[0132] The underlying task processing unit 202 is used to input the reference trajectory sequence as a guidance signal into the underlying motion actuator, wherein the underlying motion actuator is composed of multiple independent robotic arm agents, and calls the MAPPO algorithm to perform distributed real-time control on each robotic arm agent based on the reference trajectory sequence, and makes decisions by combining the local observations of each robotic arm agent to generate control commands, wherein the reference trajectory sequence is used to compare with the actual trajectory, and the deviation is used to guide the agent to learn;

[0133] Feedback unit 203 is used to obtain the execution effect of the control command and provide feedback to the underlying motion actuator based on the execution effect through a preset multi-layer reward function, so as to guide each of the robotic arm agents to learn the collaborative control strategy, thereby enabling multiple robotic arms to effectively track the reference trajectory sequence and smooth the movement of each robotic arm during the tracking process. The multi-layer reward function includes task completion reward, smoothness reward, and coordination reward.

[0134] The third embodiment of the present invention provides a two-layer intelligent agent optimization device for multiple robotic arms, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement a two-layer intelligent agent optimization method for multiple robotic arms as described in any of the above embodiments.

[0135] The fourth embodiment of the present invention provides a computer-readable storage medium storing a computer program, which can be executed by a processor of the device in which the computer-readable storage medium is located, to implement a two-layer intelligent agent optimization method for multiple robotic arms as described in any of the above claims.

[0136] Based on the two-layer intelligent agent optimization method, device, equipment, and storage medium for multi-robotic arms provided by this invention, global path planning is performed on the environmental information of the handling task by introducing the RRT-Connect algorithm and B-spline trajectory smoothing optimization at a high level. The smoothed centroid trajectory is converted into a reference trajectory sequence for the end effector of each robotic arm using a kinematic chain model. This reference trajectory is then used as the bottom-level distributed real-time control guidance signal driven by the MAPPO algorithm. Combined with a multi-layer reward function, the task completion degree, motion smoothness, and coordination are comprehensively fed back, guiding each robotic arm intelligent agent to coordinately adjust joint movements during the tracking process. This achieves seamless connection between global planning and local control, significantly improving the smoothness and coordination efficiency of the handling motion.

[0137] Exemplary examples show that the computer program described in the third and fourth embodiments of the present invention can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in implementing a two-layer intelligent agent optimization device for multi-robotic arms. For example, the apparatus described in the second embodiment of the present invention.

[0138] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the aforementioned two-layer intelligent agent optimization method for multi-robotic arms, connecting various parts of the method through various interfaces and lines.

[0139] The memory can be used to store the computer program and / or modules. The processor, by running or executing the computer program and / or modules stored in the memory, and by calling the data stored in the memory, implements various functions of a two-layer intelligent agent optimization method for multi-robotic arms. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, text conversion function, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, text message data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0140] If the implemented module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0141] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0142] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A double-layered agent optimization method for multi-robot arms, characterized in that, The method comprises the following steps: acquiring environment information of a multi-robot cooperative carrying task, and calling a high-level task planner to process the environment information, wherein the high-level task planner adopts an RRT-Connect algorithm to perform initial path exploration, combines a B-spline to perform trajectory smoothing optimization on the initial path, and converts an object centroid trajectory into a reference trajectory sequence of each robot end effector through a kinematic chain model, and the environment information comprises an initial pose, a target pose and an environment map of a target object; inputting the reference trajectory sequence as a guide signal into a bottom action executor, wherein the bottom action executor is composed of multiple independent robot agents, calling a MAPPO algorithm to perform distributed real-time control on each robot agent based on the reference trajectory sequence, and combining self-local observation of each robot agent to make decisions and generate control instructions, wherein the guide signal is part of the self-local observation of each robot agent; acquiring an execution effect of the control instructions, and feeding back the bottom action executor through a preset multi-layer reward function based on the execution effect to guide each robot agent to learn a cooperative control strategy, so that the multi-robot effectively tracks the reference trajectory sequence and smoothes the motion of each robot in the tracking process, wherein the multi-layer reward function comprises a task completion reward, a smoothness reward and a coordination reward, the guide signal is an input of the task completion reward, and the multi-layer reward function comprises a high-level reward and a bottom reward; the high-level reward comprises: Task completion reward wherein, is the current position of the object's center of mass, is the target position, is the position reward coefficient, is the distance decay parameter; Object motion smoothness reward , wherein, is the object centroid velocity vector at time t, is the object centroid velocity vector at time t-1, T is the total number of time steps, is the object centroid acceleration vector at time t, is the object centroid acceleration vector at time t-1, is the velocity deviation between the end of the robot arm and the center of the object being carried, is the acceleration deviation between the end of the robot arm and the center of the object being carried. the bottom reward comprises: Grab point distance stability bonus ,in, It is the first The distance from the point where the robotic arm grasps the object to its center of mass. It is the average distance from all grab points to the centroid; Force balance reward wherein, represents the force exerted by the i-th robot arm, is the average of all robot arm forces;​ Articulation smoothness reward , wherein, denotes the angular velocity of the joint of the robot arm at time , denotes the angular acceleration of the joint of the robot arm at time , N is the number of robots, and M is the number of joints of a single robot.

2. The dual-layer agent optimization method for multi-robot arms according to claim 1, wherein, the conversion of the object centroid trajectory into the reference trajectory sequence of each robot end effector through the kinematic chain model is specifically: based on a pre-defined grasping relationship of each robot end effector relative to the object centroid, the expected world coordinate system pose of each end effector is calculated as the reference trajectory sequence using the following kinematic transformation relationship, wherein a motion chain from a work space to an object center point is represented as: a motion chain from the object center point to a grasping point of the i-th robot is represented as: wherein, respectively the x, y, z coordinates of the object's center of mass in the world coordinate system, respectively the roll, pitch, yaw angles of the object's center of mass, respectively the x, y, z coordinate offsets of the i-th robotic arm grasp point relative to the object's center of mass, respectively the angle offsets of the i-th robotic arm grasp point relative to the object's center of mass, denotes a translation transformation function, denotes a rotation transformation function.

3. The dual-layer agent optimization method for multiple robotic arms of claim 1, wherein, a state space of the self-local observation of each robot agent is: wherein, represents a robot arm poses of the grasp points, containing x, y, z coordinate positions and roll, pitch, yaw angles; represents the i-th robot arm velocities of the grasp points; current angles of the 6 joints of the i-th robot arm; current pose of the object mass center; current velocity of the object mass center; center positions of the obstacle objects in the environment, dimensions of the square obstacle objects.

4. A double-layered agent optimization device for a multi-robot, characterized by, a double-agent optimization method for a multi-robot is used to implement the method according to any one of claims 1 to 3, comprising: a high-level task processing unit is configured to acquire environment information of a multi-robot cooperative carrying task, and call a high-level task planner to process the environment information, wherein the high-level task planner adopts an RRT-Connect algorithm to perform initial path exploration, combines a B-spline to perform trajectory smoothing optimization on the initial path, and converts an object centroid trajectory into a reference trajectory sequence of each robot end effector through a kinematic chain model, and the environment information comprises an initial pose, a target pose and an environment map of a target object; A bottom layer task processing unit is configured to input the reference trajectory sequence as a guide signal into a bottom layer action executor, wherein the bottom layer action executor is composed of a plurality of independent mechanical arm agents, a MAPPO algorithm is called to perform distributed real-time control on each mechanical arm agent based on the reference trajectory sequence, and a control instruction is generated by combining the local observation of each mechanical arm agent to make a decision. A feedback unit is configured to obtain an execution effect of the control instruction, and feed back the bottom layer action executor based on the execution effect through a preset multi-layer reward function to guide each mechanical arm agent to learn a collaborative control strategy, so that the multiple mechanical arms effectively track the reference trajectory sequence and smooth the motion of each mechanical arm in the tracking process, wherein the multi-layer reward function includes a task completion reward, a smoothness reward, and a coordination reward, and the guide signal is an input of the task completion reward.

5. A double-layered agent optimization device for multi-robot arms, characterized by, A computer program is stored in a memory, and the computer program can be executed by a processor to implement the double-layer agent optimization method for multiple mechanical arms according to any one of claims 1 to 3.

6. A computer readable storage medium characterized by, A computer program is stored in a computer readable storage medium, and the computer program can be executed by a processor of a device where the computer readable storage medium is located to implement the double-layer agent optimization method for multiple mechanical arms according to any one of claims 1 to 3.

Images