Double-layer intelligent agent optimization method, device and equipment for multiple mechanical arms and storage medium

Abstract

The invention provides a double-layer intelligent agent optimization method, device and equipment for multiple mechanical arms and a storage medium. An RRT-Connect algorithm and B spline track smooth optimization are introduced into a high layer, and global path planning is conducted on environment information of a carrying task; the smoothed centroid trajectory is converted into a reference trajectory sequence at the tail end of each mechanical arm by using a kinematics chain model, then the reference trajectory is used as a bottom layer distributed real-time control guide signal driven by an MAPPO algorithm, and comprehensive feedback is performed on task completion degree, motion smoothness and coordination in combination with a multi-layer reward function; and all the mechanical arm intelligent bodies are guided to cooperatively adjust joint actions in the tracking process, so that seamless connection of global planning and local control is achieved, and the smoothness and cooperative efficiency of carrying motion are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent agent optimization, and in particular to a two-layer intelligent agent optimization method, apparatus, device, and storage medium for multi-robotic arms. Background Technology

[0002] Collaborative handling of large or heavy-duty objects by multiple robotic arms is an important research direction in the field of robotics, with broad application prospects in industrial automation, intelligent logistics, and aerospace assembly. Compared to single-arm operation, multi-arm collaboration can overcome the limitations of a single robotic arm in terms of load capacity, workspace, and operational flexibility, enabling stable handling of large, heavy-duty, or irregularly shaped objects. However, achieving efficient, stable, and intelligent multi-arm collaborative handling still faces many technical challenges. Existing technologies mainly suffer from the following shortcomings: 1. While traditional model-based control methods (such as PID control and sliding mode control) can achieve precise control under ideal conditions, their performance heavily depends on accurate system dynamics models. In complex, highly coupled systems such as multi-arm cooperative systems, it is difficult to establish accurate models, and the systems are susceptible to uncertainties such as load changes and external disturbances, resulting in poor robustness and environmental adaptability of these methods.

[0003] 2. Multi-Agent Reinforcement Learning (MARL), as a data-driven method, does not rely on precise models and theoretically can learn strategies to adapt to complex dynamic environments through interaction with the environment. However, when directly applied to multi-arm cooperative handling, it has some shortcomings: low learning efficiency and difficulty in convergence. The huge state-action space leads to an extremely large number of samples required for training, and simultaneous learning by multiple agents can easily cause training instability and difficulty in convergence; lack of global guidance and poor motion quality. MARL is usually based on local observation decisions and lacks global task planning, which can easily lead to uneven, inefficient, and even jittery motion trajectories, affecting the stability of handling. The fundamental reason is that exploring in a huge, unstructured joint action space, without macroscopic, physically constrained path guidance, makes it extremely difficult for reinforcement learning agents to learn complex strategies that balance global optimization and local smoothness. 3. Hierarchical reinforcement learning (HRL) attempts to improve the efficiency of MARL through task decomposition, but its hierarchical structure and subtask division often rely on expert prior knowledge, lacking generality and adaptability to dynamic environments. For specific tasks such as multi-arm cooperative handling, which have extremely high requirements for smoothness, real-time performance, and high-precision coordination, existing HRL research is still insufficient.

[0004] In addition, path planning algorithms such as RRT and its variants can effectively generate collision-free geometric paths, but they mainly solve static path planning problems and cannot directly handle the problems of dynamic control, real-time adaptation and optimization of collaborative strategies of multiple arms.

[0005] In view of this, the present application is proposed. SUMMARY

[0006] The application discloses a double-layer agent optimization method, device and equipment for multiple mechanical arms and a storage medium, and aims to solve the problems of disconnection between global path planning and local agent control, non-smooth motion and low collaborative efficiency in existing multiple mechanical arm collaborative handling tasks.

[0007] The first embodiment of the application provides a double-layer agent optimization method for multiple mechanical arms, comprising: obtaining environment information of a multiple mechanical arm collaborative handling task, and calling a high-level task planner to process the environment information, wherein the high-level task planner performs initial path exploration by using an RRT-Connect algorithm, performs trajectory smoothing optimization on the initial path in combination with a B-spline, and converts an object centroid trajectory into a reference trajectory sequence of each mechanical arm end effector through a kinematic chain model, and the environment information comprises an initial pose, a target pose and an environment map of a target object; inputting the reference trajectory sequence as a guide signal into a bottom layer action executor, wherein the bottom layer action executor is composed of multiple independent mechanical arm agents, calling a MAPPO algorithm to perform distributed real-time control on each mechanical arm agent based on the reference trajectory sequence, and combining self-local observation of each mechanical arm agent to make decisions and generate control instructions, wherein the guide signal is part of the self-local observation of each mechanical arm agent; obtaining an execution effect of the control instructions, and feeding back the bottom layer action executor through a preset multi-layer reward function based on the execution effect, so as to guide each mechanical arm agent to learn a collaborative control strategy, thereby enabling the multiple mechanical arms to effectively track the reference trajectory sequence and smooth the motion of each mechanical arm in the tracking process, wherein the multi-layer reward function comprises a task completion reward, a smoothness reward and a coordination reward, and the guide signal is an input of the task completion reward.

[0008] Preferably, the conversion of the object centroid trajectory into the reference trajectory sequence of each mechanical arm end effector through the kinematic chain model is specifically: Based on the predefined grasping relationship of each end effector of the robot arm relative to the object centroid, the desired world coordinate system pose of each end effector is calculated as the reference trajectory sequence using the following kinematics transformation relationship, wherein the motion chain from the workspace to the object center point is represented as:

[0009] The motion chain from the object center point to the i-th robot arm grasping point is represented as:

[0010] wherein, are the x, y, z coordinates of the object centroid in the world coordinate system, respectively, are the roll angle, pitch angle, yaw angle of the object centroid, respectively, are the x, y, z coordinate offset of the i-th robot arm grasping point relative to the object centroid, respectively, are the angle offset of the i-th robot arm grasping point relative to the object centroid, respectively, represents a translation transformation function, represents a rotation transformation function; Preferably, the state space of the self-local observation of each robot arm agent is:

[0011] wherein, represents the pose of the robot arm grasping point, including x, y, z coordinate position and roll angle, pitch angle, yaw angle; represents the velocity of the i-th robot arm grasping point; is the current angle configuration of the 6 joints of the i-th robot arm; is the current pose of the object centroid; is the current velocity of the object centroid; is the center position of the obstacle object in the environment, is the size of the square obstacle object.

[0012] Preferably, the multi-layer reward function includes a high-level reward and a low-level reward; The high-level reward includes: task completion reward , wherein, is the current position of the object centroid, is the target position, is the position reward coefficient, is the distance decay parameter; object motion smoothness reward , wherein, is the object's center of mass velocity vector at time t, T is the total number of time steps, is the object's center of mass acceleration vector at time t, is the velocity deviation between the end of the robot arm and the center of the object, is the acceleration deviation between the end of the robot arm and the center of the object; The bottom layer reward includes: Grabbing point distance stability reward wherein, is the distance from the th robot arm grabbing point to the object's center of mass, is the average of all grabbing point to center of mass distances; Force balance reward wherein, represents the force exerted by the th robot arm, is the average of all robot arm forces; Joint motion smoothness reward , wherein, represents the angular velocity of the th joint of the th robot arm at time , represents the angular acceleration of the th joint of the th robot arm at time , N is the number of robot arms, and M is the number of joints of a single robot arm.

[0013] Preferably, the guiding causes each of the robot arm agents to learn a collaborative control strategy, so that the multiple robot arms effectively track the reference trajectory sequence, specifically: At each time step t, each agent obtains its local observation and the current reference pose , and selects an action through its Actor network, wherein the architecture of each agent is an Actor-Critic architecture; The joint action of all agents is sent to the physics engine for calculation, thereby updating the state of the robot arms and the object, and returning the joint observation at the next time and the joint reward calculated according to the reward function; The experience tuple is stored in the experience replay buffer, and data is periodically sampled from the buffer to update the parameters of the Actor and Critic networks, and the learning is repeated to enable the agents to gradually learn to track the reference trajectory and achieve effective collaboration.

[0014] Preferably, further comprising: The Actor network of each robotic arm agent only depends on its local observation Make decisions, each robotic arm agent is equipped with an independent policy network , evaluate the policy of the Actor network by learning the global Critic network, and according to the global state Learn the central value function .

[0015] The policy network of each agent Composed of three hidden layers, mapping the local observation Into an action probability distribution, the output layer uses a tanh activation function to ensure that the joint angle adjustment is within the operating limit of the robotic arm, and the optimization objective function of the Actor network of the MAPPO algorithm is:

[0016] The minimum loss function of the Critic network is:

[0017] Where, represents the local observation state of the i-th agent, represents the parameters of the i-th agent Actor network, represents the action output by the i-th agent, represents the global state information at time t, represents the parameters of the Critic network, represents the value estimate of the Critic network for the global state , is the probability ratio of the new policy to the old policy, is the advantage function estimate value, T is the total time step, is the immediate reward at time t, is the discount factor.

[0018] The second embodiment of the application provides a double-layer agent optimization device for multiple robotic arms, comprising: A high-level task processing unit is configured to acquire environment information of a cooperative carrying task of multiple robotic arms, and call a high-level task planner to process the environment information, wherein the high-level task planner performs initial path exploration using an RRT-Connect algorithm, performs trajectory smoothing optimization on the initial path in combination with a B-spline, and converts an object centroid trajectory into a reference trajectory sequence of each robotic arm end effector through a kinematic chain model, and the environment information includes an initial pose, a target pose and an environment map of a target object. a bottom layer task processing unit configured to input the reference trajectory sequence as a guide signal into a bottom layer action executor, wherein the bottom layer action executor is composed of a plurality of independent mechanical arm agents, a MAPPO algorithm is invoked to perform distributed real-time control on each of the mechanical arm agents based on the reference trajectory sequence, and a control instruction is generated by combining the self-local observation of each of the mechanical arm agents to make a decision, wherein the reference trajectory sequence is used to compare with an actual trajectory, and a deviation thereof is used to guide agent learning, and the guide signal is part of the self-local observation of each of the mechanical arm agents; a feedback unit configured to obtain an execution effect of the control instruction, and feed back the bottom layer action executor based on the execution effect through a preset multi-layer reward function to guide each of the mechanical arm agents to learn a collaborative control strategy, so that the multi-mechanical arms effectively track the reference trajectory sequence and smooth the motion of each mechanical arm in the tracking process, wherein the multi-layer reward function includes a task completion reward, a smoothness reward, and a coordination reward, and the guide signal is an input of the task completion reward.

[0019] A third embodiment of the present application provides a double-layer agent optimization device for multi-mechanical arms, including a memory and a processor, the memory stores a computer program, and the computer program can be executed by the processor to implement the double-layer agent optimization method for multi-mechanical arms according to any one of the above.

[0020] A fourth embodiment of the present application provides a computer readable storage medium storing a computer program, and the computer program can be executed by a processor of a device where the computer readable storage medium is located to implement the double-layer agent optimization method for multi-mechanical arms according to any one of the above.

[0021] Based on the double-layer agent optimization method, device, equipment and storage medium for multi-mechanical arms provided by the present application, the RRT-Connect algorithm and B-spline trajectory smoothing optimization are introduced in the high layer, the global path planning is performed on the environment information of the carrying task, the kinematic chain model is used to convert the smoothed centroid trajectory into the reference trajectory sequence of each mechanical arm end, the reference trajectory is used as the bottom layer distributed real-time control guide signal driven by the MAPPO algorithm, the multi-layer reward function is used to comprehensively feed back the task completion degree, the motion smoothness and the coordination, each mechanical arm agent is guided to cooperatively adjust the joint action in the tracking process, so that the seamless connection of global planning and local control is realized, and the smoothness and the coordination efficiency of the carrying motion are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flowchart of a double-layer agent optimization method for multi-mechanical arms provided by the first embodiment of the present application; Figure 2 is a kinematic diagram of a multi-arm cooperative carrying system provided by the present application; Figure 3 is a schematic diagram of a dual-arm cooperative carrying task scene provided by the present application; Figure 4 is a schematic diagram of the trajectory of the end effector of a robotic arm and the trajectory of the center of mass of a target object of a comparative method MAPPO; Figure 5 is a schematic diagram of the trajectory of the end effector of a robotic arm and the trajectory of the center of mass of a target object based on H-MAPPO provided by the present application; Figure 6 is a schematic diagram of the angular velocity and angular acceleration curve of the joints of a robotic arm of the comparative method MAPPO; Figure 7 is a schematic diagram of the angular velocity and angular acceleration curve of the joints of a robotic arm of H-MAPPO provided by the present application; Figure 8 is a schematic diagram of the reward and loss curve of the two methods; Figure 9 is a schematic diagram of a module of a dual-layer intelligent agent optimization device for multiple robotic arms provided by the second embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to better understand the technical solutions of the present application, the embodiments of the present application will be described in detail below with reference to the drawings.

[0024] It should be clear that the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0025] The terms used in the embodiments of the present application are only for the purpose of describing the specific embodiments, and are not intended to limit the present application. The singular forms "a", "an" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0026] It should be understood that the term "and / or" used herein is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0027] Depending on context, the word "if" as used herein can be interpreted to mean "when" or "while" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]."

[0028] The "first / second" mentioned in the embodiments are only to distinguish similar objects, and do not represent a specific order of the objects. Understandably, the "first / second" can be interchanged in a specific order or sequence as permitted. It should be understood that the objects distinguished by the "first / second" can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0029] The specific embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0030] The first embodiment of the present application provides a double-layer intelligent agent optimization method for multiple robot arms, which can be executed by an optimization device (or system), in particular, by one or more processors in the optimization device, to at least implement the following steps: S101, obtaining environment information of a multiple robot arm cooperative carrying task, calling a high-level task planner to process the environment information, wherein the high-level task planner uses an RRT-Connect algorithm to perform initial path exploration (in other optional implementation manners, path exploration can also use other graph search algorithms such as A*, D* and the like, which are not specifically limited here, but these schemes are within the protection scope of the present application), combines a B-spline to perform trajectory smoothing optimization on the initial path, and converts an object centroid trajectory into a reference trajectory sequence of each robot arm end effector through a kinematic chain model, the environment information including an initial pose, a target pose and an environment map of a target object; In the present embodiment, the optimization device can be a desktop computer, a workstation, a server or the like terminal having data processing capability, which can establish communication with multiple robot arms. The optimization device can be installed with a corresponding operating system and application software, and the functions required by the present embodiment can be realized through the combination of the operating system and the application software. The reference trajectory sequence is used for comparison with an actual trajectory, and the deviation is used to guide intelligent agent learning. It should be noted that in the implementation process of the multi-robot arm cooperative carrying task, the system first acquires complete environment information including the initial pose of the target object, the target pose and the environment map. After receiving these environment information, the high-level task planner uses the RRT-Connect algorithm to explore the initial path in the configuration space. This algorithm quickly finds a feasible geometric path from the initial pose to the target pose by simultaneously growing two random trees from the starting point and the ending point and trying to connect them. Since the path generated by the RRT-Connect algorithm is usually a polyline connected by line segments, there are inflection points that are not smooth. The system further uses the B-spline interpolation method to optimize the trajectory smoothing of the initial path, and converts the discrete path points into a continuous and smooth trajectory parameterized by time.

[0031] After obtaining the smooth object centroid trajectory, the system converts the object centroid trajectory into a sequence of reference trajectories of each robot arm end effector through the kinematic chain model. This process is based on the pre-defined grasping relationship of each robot arm end effector relative to the object centroid, and uses the homogeneous coordinate transformation matrix to establish the complete kinematic mapping relationship.

[0032] Please refer to Figure 2 , the world coordinate system {X O ,Y O ,Z O} as the global reference coordinate system, the object centroid coordinate system {X C ,Y C ,Z C} with the centroid of the object to be carried as the origin, the robot arm base coordinate system {x 1i ,y 1i ,z 1i}, the base coordinate system of each robot arm, the robot arm end coordinate system {x ei ,y ei ,z ei}, the robot arm end effector coordinate system, the key transformation relationship : transformation vector from world coordinate system to object centroid : transformation vector from object centroid to i-th robot arm grasping point : transformation vector from robot arm end to base, the orange ellipse represents the object to be carried, the two robot arms grasp the object from different positions, each robot arm has its own base and end effector, and all coordinate systems are connected through the kinematic chain. Kinematic chain from workspace to object center point It is expressed by a homogeneous transformation matrix as follows:

[0033] wherein, are the x, y, z coordinates of the object centroid in the world coordinate system, respectively, respectively, are the roll angle, pitch angle, and yaw angle of the object's center of mass, based on the pose information of the object's center of mass, the system further calculates the kinematic transformation relationship from the object's center point to the i-th robotic arm's grasping point (the kinematic chain from the object's center point to the grasping point ), which is expressed by the following matrix:

[0034] In this transformation, respectively, are the x, y, z coordinate offset amounts of the i-th robotic arm's grasping point relative to the object's center of mass, respectively, are the angle offset amounts of the i-th robotic arm's grasping point relative to the object's center of mass, which are pre-determined according to the object's geometric characteristics and grasping strategy.

[0035] In order to establish a complete kinematic chain, the system also needs to determine the kinematic relationship from the robotic arm's end effector to the base. Through a similar transformation method, the kinematic chain from the robotic arm's end effector to the base can be obtained :

[0036] By cascading and combining these kinematic transformation relationships, the system can establish a complete kinematic mapping from the world coordinate system to each robotic arm's end effector. When the trajectory of the object's center of mass is given , the system can calculate the expected pose trajectory of each end effector in the world coordinate system through inverse kinematic calculation, and this trajectory sequence is the reference trajectory that each robotic arm needs to track . The entire kinematic mapping process ensures that each robotic arm's end effector can maintain a constant relative position relationship with the object's center of mass, thereby achieving stable cooperative grasping and carrying operations.

[0037] S102, input the reference trajectory sequence as a guide signal to the underlying action executor, wherein the underlying action executor is composed of multiple independent robotic arm agents, invoke the MAPPO algorithm to perform distributed real-time control on each robotic arm agent based on the reference trajectory sequence, and combine the local observation of each robotic arm agent to make decisions and generate control instructions; It should be noted that after the high-level task planner generates the reference trajectory sequence, the system inputs these trajectory sequences as guide signals to the bottom layer action executor for distributed control. The bottom layer action executor adopts a multi-agent reinforcement learning framework, which is composed of N independent robot arm agents, each of which corresponds to the control of a robot arm. The system calls the MAPPO (Multi-Agent Proximal Policy Optimization) algorithm to perform distributed real-time control on each robot arm agent, which combines the stability advantages of PPO algorithm in single-agent environment and the collaborative learning ability of multi-agent system.

[0038] In the implementation process of the MAPPO algorithm, each robot arm agent not only needs to track the reference trajectory sequence provided by the high-level planner , but also must make real-time decisions based on its own local observation information. The system defines a comprehensive state space for each agent, which includes the state of the robot arm itself, the state of the target object, and environmental information, among other multidimensional data. Specifically, the local observation state space of the i-th robot arm agent is defined as:

[0039] In this state space definition, represents the pose of the robot arm grasp point, including x, y, z coordinate positions and roll, pitch, yaw angles; The velocity information is represented by vector, which contains the instantaneous velocity components of the i-th robot arm grasp point in three coordinate axis directions, providing dynamic feedback information for the motion control of the agent.

[0040] The internal configuration state of the robot arm is described by the joint angle vector , which is a six-dimensional vector corresponding to the current angle configuration of each joint of a standard six-degree-of-freedom industrial robot arm (such as UR5), directly reflecting the internal motion state of the robot arm. In order to achieve effective collaborative control, each agent also needs to perceive the state information of the object being transported, including the current pose and centroid velocity of the object's centroid, which enables agents to understand the execution status of the overall transportation task and adjust their control strategies accordingly; Considering the complexity of the environment in practical applications, the system also includes environmental obstacle information in the state space. represents the center position coordinates of the environmental obstacles, while describes the size parameters of the square obstacles, which provide the necessary obstacle avoidance decision basis for the agent. Based on such a complete state space definition, each agent makes real-time decisions through its independent policy network mapping the current observation state into a specific control action, wherein denotes the parameters of the i-th agent policy network, denotes the output control command.

[0041] S103, obtaining the execution effect of the control command, and feeding back the bottom layer action executor based on the execution effect through a preset multi-layer reward function to guide each of the robot arm agents to learn a collaborative control strategy, so that the multiple robot arms effectively track the reference trajectory sequence and smooth the motion of each robot arm in the tracking process.

[0042] It should be noted that after the bottom layer action executor generates the control command, the system needs to continuously obtain the execution effect of the control command, and evaluate the motion state and collaborative performance of each robot arm through the feedback of the actual system. The system feeds back the bottom layer action executor in real time based on the execution effect through a preset multi-layer reward function, wherein the feedback is used to guide each robot arm agent to learn a collaborative control strategy. Further, the multi-layer reward function is divided into two levels of high-level reward and bottom-level reward, corresponding to the optimization requirements of global task target and local motion quality respectively.

[0043] The high-level reward mainly focuses on the completion of the overall task and the global characteristics of object motion. The task completion reward is defined in the form of an exponential decay function as follows: ) wherein, is the current position of the object centroid, is the target position, is the position reward coefficient for adjusting the absolute size of the reward, is the distance decay parameter to control the sensitivity of the reward to the distance, which can effectively make the reward obtained by the agent higher when the object is closer to the target position, thereby effectively guiding the collaborative transportation in the correct direction In order to ensure the smoothness of the object motion during the transportation process, an object motion smoothness reward is configured, which is realized by punishing the violent changes in the object motion as follows: , wherein, is the velocity vector of the object centroid at time t, and T is the total number of time steps, is the acceleration vector of the object centroid at time t, is the velocity deviation between the end of the robot arm and the center of the transported object, is the acceleration deviation between the end of the robot arm and the center of the transported object, and in the present embodiment, Improve motion smoothness by penalizing rapid changes in the velocity of the object's center of mass (i.e., the jerkiness of the center of mass's motion). The smoothness of motion is further improved by penalizing rapid changes in the acceleration of the object's center of mass.

[0044] The underlying reward focuses on the coordination between the robotic arms and the smoothness of individual arm movements. The gripping point distance stability reward is used to maintain a stable gripping posture for each robotic arm relative to the object's center of mass.

[0045] in, It is the first The distance from the point where the robotic arm grasps the object to its center of mass. It is the average distance from all gripping points to the center of mass, and N is the total number of robotic arms. This reward encourages the robotic arms to maintain a stable relative position by minimizing the variance of the distances between gripping points.

[0046] To ensure that the forces exerted by each robotic arm on the object remain balanced, the system is designed with a force balance reward:

[0047] in, Indicates the first The force applied by the robotic arm, It is the average force of all robotic arms, which can effectively avoid system instability caused by a single robotic arm bearing too much load; Joint-level motion smoothness is achieved through the following reward function:

[0048]

[0049] in, Indicates the first The first robotic arm Each joint in time angular velocity, Indicates the first The first robotic arm Each joint in time The angular acceleration is given by N, the number of robotic arms, and M, the number of joints in a single robotic arm. In this embodiment, smoothness of robotic arm movement is promoted by penalizing abrupt changes in joint motion.

[0050] In guiding each robotic arm agent to learn the cooperative control strategy, the system employs a cyclic learning mechanism. Within each time step t, each agent i first acquires its local observations. and current reference pose Then through its Actor network Select the corresponding action Each agent adopts an Actor-Critic architecture, where the Actor network is responsible for policy decision-making and the Critic network is responsible for value evaluation.

[0051] The system coordinates the actions of all intelligent agents. The data is sent to the physics engine for dynamic calculations. The physics engine updates the states of the robotic arm and the object based on the current state and actions, and returns the joint observations for the next time step, along with the joint reward calculated using a multi-layered reward function. The system will then provide the complete experience tuple. Stored in the experience replay buffer, where This represents the reference trajectory information for each robotic arm at time t. By periodically sampling historical experience data from the buffer, the system uses the MAPPO algorithm to update the Actor and Critic network parameters of each agent. Through a cyclic learning process, each agent gradually learns to accurately track the reference trajectory. Simultaneously, guided by multi-layered reward functions, effective cooperative control is achieved, ultimately reaching the overall goal of smoothly transporting the target object. In one possible implementation of the present invention, it further includes: In the specific implementation of the MAPPO algorithm, the system adopts a distributed decision-making but centralized training architecture. Each robotic arm agent's Actor network strictly adheres to the principle of local observation, relying only on its own perceived local state information. Decision-making ensures the distributed nature and scalability of the system during actual deployment. Each robotic arm agent is equipped with an independent policy network. ,in This represents the parameter vector of the i-th agent network, which is continuously updated during the training process to optimize the control strategy.

[0052] To achieve effective collaborative learning while maintaining decision-making independence, the system designs a shared global Critic network to evaluate the quality of each Actor's network policies. This design allows each agent to gain an understanding of the overall system state by learning the global Critic network, thereby making more coordinated decisions. The global Critic network is based on the complete system state. Learning center value function , This represents the parameters of the Critic network, and the function can accurately evaluate the long-term cumulative reward expectation of the system under the current global state.

[0053] In terms of the specific design of the network architecture, the policy network of each agent adopts a deep neural network structure, consisting of three hidden layers. Each hidden layer contains an appropriate number of neurons to ensure sufficient expressive power. The input layer of the network receives local observation states. After nonlinear transformation through three hidden layers, the local observation information is mapped to an action probability distribution. To ensure that the output control action conforms to the physical constraints of the robotic arm, the output layer uses the tanh activation function to restrict the network output to the interval [-1,1]. Then, appropriate scaling and offset operations are used to ensure that the joint angle adjustment is always within the operating limits of the robotic arm.

[0054] The MAPPO algorithm updates the policies of each agent by optimizing a specific objective function:

[0055] In this objective function, This is the probability ratio of the new strategy to the old strategy, reflecting the magnitude of the strategy update. The estimated value of the advantage function Used to measure the state Take action below The degree of superiority or inferiority relative to the average level, For the clipping parameters, the clipping operation limits the probability ratio to [ To prevent overly aggressive policy updates, the value is set to 0.2 within a certain range.

[0056] In contrast to the policy optimization of the Actor network, the Critic network learns an accurate value function by minimizing the temporal difference error. The loss function of the Critic network is defined as:

[0057] In this loss function, This shows the Critic network's current global state. The value estimate, The instantaneous reward obtained by the agent at time t. The discount factor is used to balance the importance of immediate rewards and future rewards, and is typically set between 0.95 and 0.99. The expression represents the value estimate of the next state. The target estimate constitutes the true value of the current state, and in the actual training process, the system uses an experience replay mechanism to stabilize the learning process. The experience tuples collected at each time step are stored in a shared experience buffer, and when training, a batch of data is sampled from the buffer to update the network parameters. The actor network improves the policy by maximizing the above objective function, so that the agent can adapt to environmental changes and optimize collaborative behavior while following the reference trajectory. The critic network provides more accurate policy evaluation by minimizing the value estimation error, providing a reliable guidance signal for the update of the actor network.

[0058] Through the combination of the Actor-Critic architecture and the MAPPO algorithm, each robotic arm agent can learn a high-quality collaborative control strategy while maintaining decision independence. It can accurately track the reference trajectory provided by the high-level planner and perform local optimization based on real-time feedback during execution, ultimately achieving smooth, stable, and coordinated multi-robot arm collaborative control. In other optional implementation methods, the underlying can also use other multi-agent reinforcement learning algorithms such as MADDPG, QMIX, etc. Here we do not make specific limitations, but these solutions are within the scope of the present invention.

[0059] To verify the effectiveness of the method (denoted as H-MAPPO), a simulation experiment was conducted and compared with the baseline end-to-end multi-agent reinforcement learning method (MAPPO).

[0060] Experimental setup: Simulation platform: CoppeliaSim Edu 4.5.

[0061] Programming and framework: Python language, using PyTorch to implement neural network models, combined with OpenAI Gym to define reinforcement learning environment interfaces. Communicate with the simulator through CoppeliaSim Remote API.

[0062] Hardware: Training and testing were performed on a personal computer equipped with an Intel Core i5-11400H CPU and an NVIDIA RTX 3050 Laptop GPU.

[0063] Task scenario (refer to Figure 3): Simulate two arms cooperatively moving a large bench plank (size 1580x600x40mm, mass 15.1kg). Use two UR5 robots. Set the initial pose of the object (position (0.42, 0.0, 0.26) meters, attitude (0.0, 0.0, -90.0) degrees in the world coordinate system) and the target pose (in this comparative experiment, the target pose can be the same as the initial pose, aiming to test the ability to stabilize the grasping and holding attitude, or set a moving target). Obstacles can be set in the environment.

[0064] Comparative method: Train the baseline MAPPO algorithm end-to-end using the same network structure and hyperparameter settings as the underlying controller in the H-MAPPO method of the present application.

[0065] Evaluation indicators: (1) Task success rate (not directly given in the figures, but as an implicit evaluation); (2) Trajectory smoothness (observe the motion trajectories of the robot end and object centroid through visualization); (3) Joint motion stability (record and plot the joint angular velocity and angular acceleration curves); (4) Training convergence speed (plot the average episode reward curve during training).

[0066] Experimental results: Trajectory comparison: Figure 4 Show the motion trajectories of the two robot ends (red and blue lines) and the object centroid (green line) when executing the task after training using the baseline MAPPO method. It can be seen that the trajectories are obviously jagged and discontinuous, indicating unstable motion. In contrast, Figure 5 Show the corresponding trajectories produced after training using the H-MAPPO method of the present application. These trajectories are obviously smoother, more direct, and coherent, showing better motion quality.

[0067] Joint motion comparison: Figure 6 Show the angular velocity and angular acceleration curves of each joint (e.g., joints 1-6) of the two robots over time under the baseline MAPPO method. It can be seen that the curves have many sharp peaks and violent fluctuations, indicating frequent acceleration and deceleration of joint motion, with large impacts. Figure 7 Then show the corresponding curves under the H-MAPPO method of the present application. The peak values of these curves are significantly reduced, and the fluctuations are more gentle, indicating more stable and gentle joint motion.

[0068] Training performance comparison: Figure 8The average round reward of the H-MAPPO method (for example, the red curve in the figure) and the benchmark MAPPO method (for example, the blue curve in the figure) during the training process is shown. It can be seen that the reward curve of the H-MAPPO method rises faster and finally converges to a higher reward level than the benchmark MAPPO method. This indicates that the method has higher learning efficiency and better final performance.

[0069] Conclusion: The above comparative experiment results strongly prove that the double-layer multi-agent reinforcement learning method (H-MAPPO) proposed in the application can significantly improve the smoothness and stability of movement, while improving the training efficiency and final performance in the task of multi-robot cooperative handling of large objects compared with the traditional end-to-end MARL method (MAPPO).

[0070] Please refer to Figure 9 The second embodiment of the application provides a double-layer agent optimization device for multi-robot, comprising: A high-level task processing unit 201 is configured to acquire environment information of a multi-robot cooperative handling task, and call a high-level task planner to process the environment information, wherein the high-level task planner uses an RRT-Connect algorithm to perform initial path exploration, combines a B-spline to perform trajectory smoothing optimization on the initial path, and converts an object centroid trajectory into a reference trajectory sequence of each robot end effector through a kinematic chain model, and the environment information includes an initial pose, a target pose and an environment map of a target object. A bottom-level task processing unit 202 is configured to input the reference trajectory sequence as a guide signal into a bottom-level action executor, wherein the bottom-level action executor is composed of multiple independent robot agents, and a MAPPO algorithm is called to perform distributed real-time control on each robot agent based on the reference trajectory sequence, and each robot agent makes decisions based on its own local observation to generate control instructions, wherein the reference trajectory sequence is used for comparison with an actual trajectory, and the deviation thereof is used to guide agent learning. A feedback unit 203 is configured to acquire the execution effect of the control instructions, and feedback the bottom-level action executor based on the execution effect through a preset multi-layer reward function to guide each robot agent to learn a cooperative control strategy, so that the multi-robot effectively tracks the reference trajectory sequence and smoothes the movement of each robot during tracking, wherein the multi-layer reward function includes a task completion reward, a smoothness reward, and a coordination reward.

[0071] The third embodiment of the present application provides a double-layer agent optimization device for multiple robot arms, comprising a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the double-layer agent optimization method for multiple robot arms according to any one of the above.

[0072] The fourth embodiment of the present application provides a computer readable storage medium storing a computer program, and the computer program can be executed by a processor of a device where the computer readable storage medium is located to implement the double-layer agent optimization method for multiple robot arms according to any one of the above.

[0073] Based on the double-layer agent optimization method, device, equipment and storage medium for multiple robot arms provided by the present application, the RRT-Connect algorithm and B-spline trajectory smoothing optimization are introduced in the upper layer, the global path planning of the environment information of the carrying task is performed, the kinematic chain model is used to convert the smoothed centroid trajectory into a reference trajectory sequence of each robot arm end, the reference trajectory is used as the bottom layer distributed real-time control guide signal driven by the MAPPO algorithm, the task completion degree, motion smoothness and coordination are comprehensively fed back by combining the multi-layer reward function, and each robot arm agent is guided to cooperatively adjust the joint action in the tracking process, so that the seamless connection of global planning and local control is realized, and the smoothness and coordination efficiency of the carrying motion are significantly improved.

[0074] Exemplarily, the computer program in the third and fourth embodiments of the present application can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present application. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the double-layer agent optimization device for multiple robot arms. For example, the device in the second embodiment of the present application.

[0075] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or can also be any conventional processor. The processor is a control center of the method for optimizing a dual-layer intelligent agent for a multi-robot arm, and is connected to each part of the method for optimizing a dual-layer intelligent agent for a multi-robot arm through various interfaces and lines.

[0076] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the method for optimizing a dual-layer intelligent agent for a multi-robot arm by running or executing the computer program and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required for a function (such as a sound playing function, a text conversion function, etc.), etc.; and the data storage area can store data created according to the use of the mobile phone (such as audio data, text message data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0077] The implemented modules, if implemented in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods of the present application can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the contents included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0078] It should be noted that the above-described device embodiments are only schematic, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. In addition, the connection relationship between the modules in the device embodiment provided by the present application indicates that there is a communication connection between them, which can be realized as one or more communication buses or signal lines. Those skilled in the art can understand and implement it without creative labor.

[0079] The above is only a preferred embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed by the present application can be easily thought of by those skilled in the art, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A double-layered agent optimization method for multi-robot arms, characterized in that, The method comprises the following steps: Obtain environment information of a multi-robot cooperative carrying task, and call a high-level task planner to process the environment information, wherein the high-level task planner uses an RRT-Connect algorithm to perform initial path exploration, combines a B-spline to perform trajectory smoothing optimization on the initial path, and converts an object centroid trajectory into a reference trajectory sequence of each robot end effector through a kinematic chain model, and the environment information comprises an initial pose, a target pose and an environment map of a target object; Input the reference trajectory sequence as a guide signal into a bottom action executor, wherein the bottom action executor is composed of multiple independent robot agents, call a MAPPO algorithm to perform distributed real-time control on each robot agent based on the reference trajectory sequence, and combine the local observation of each robot agent to make decisions and generate control instructions, wherein the guide signal is part of the local observation of each robot agent; Obtain the execution effect of the control instructions, and feed back the bottom action executor through a preset multi-layer reward function based on the execution effect to guide each robot agent to learn a cooperative control strategy, so that the multi-robot effectively tracks the reference trajectory sequence and smoothes the motion of each robot in the tracking process, wherein the multi-layer reward function comprises a task completion reward, a smoothness reward and a coordination reward, and the guide signal is the input of the task completion reward.

2. The bi-layered agent optimization method for multi-robot arms according to claim 1, wherein, The conversion of the object centroid trajectory into the reference trajectory sequence of each robot end effector through the kinematic chain model is specifically: Based on the pre-defined grasping relationship of each robot end effector relative to the object centroid, the expected world coordinate system pose of each end effector is calculated as the reference trajectory sequence using the following kinematic transformation relationship, wherein the motion chain from the workspace to the object center point is represented as: The motion chain from the object center point to the i-th robot grasping point is represented as: wherein, respectively the x, y, z coordinates of the object's center of mass in the world coordinate system, respectively the roll, pitch, yaw angles of the object's center of mass, respectively the x, y, z coordinate offsets of the i-th robotic arm grasp point relative to the object's center of mass, respectively the angle offsets of the i-th robotic arm grasp point relative to the object's center of mass, denotes a translation transformation function, denotes a rotation transformation function.

3. The dual-layer agent optimization method for multiple robotic arms of claim 1, wherein, The state space of the local observation of each robot agent is: wherein, represents a robot arm poses of the grasp points, containing x, y, z coordinate positions and roll, pitch, yaw angles; represents the i-th robot arm velocities of the grasp points; current angles of the 6 joints of the i-th robot arm; current pose of the object mass center; current velocity of the object mass center; center positions of the obstacle objects in the environment, dimensions of the square obstacle objects.

4. The bi-layered agent optimization method for multiple robotic arms of claim 1, wherein, The multi-layer reward function comprises a high-level reward and a bottom reward; The high-level reward comprises: Task completion reward ), wherein, is the current position of the object's center of mass, is the target position, is the position reward coefficient, is the distance decay parameter; Object motion smoothness reward , wherein, is the object centroid velocity vector at time t, T is the total number of time steps, is the object centroid acceleration vector at time t, is the velocity deviation between the end of the robot arm and the center of the object being carried, is the acceleration deviation between the end of the robot arm and the center of the object being carried; The bottom reward comprises: Grab point distance stability reward wherein, is the distance of the th robotic arm grab point to the object centroid, is the average of all the grab point to centroid distances; Force balance reward wherein, represents the force exerted by the i-th robot arm, is the mean of all robot arm forces;​ Articulation smoothness reward , ,wherein, denotes the angular velocity of the j-th joint of the i-th robot arm at time t, denotes the angular acceleration of the j-th joint of the i-th robot arm at time t, N is the number of robots, and M is the number of joints of a single robot arm.​​​​ 5. The bi-layered agent optimization method for multiple robotic arms of claim 1, wherein, The guiding each robot agent to learn a cooperative control strategy, so that the multi-robot effectively tracks the reference trajectory sequence, is specifically: At each time step t, each agent obtains its local observation and current reference pose selects an action through its Actor network, where the architecture of each agent is an Actor-Critic architecture; The joint action of all agents is sent to the physics engine computation, which in turn updates the state of the robot and the objects, and returns the joint observation at the next time step and the joint reward computed from the reward function ; The experience tuples are stored into an experience replay buffer, data is periodically sampled from the buffer, the parameters of the Actor and Critic networks are updated, and the learning is iterated to make the agents learn to track the reference trajectories and achieve effective coordination gradually.

6. The bi-layered agent optimization method for multi-robot arms according to claim 5, wherein, Further comprising: The Actor network of each robotic arm agent only relies on its local observation Each robotic arm agent is equipped with an independent policy network to make decisions The policy of the Actor network is evaluated by learning a global Critic network, and the global state Learning the central value function ; Policy network of each agent consists of three hidden layers, which map local observations into action probability distributions, with a tanh activation function in the output layer to ensure joint angle adjustments are within the manipulator’s operating limits. The objective function for the actor network of the MAPPO algorithm is: The minimization loss function of the Critic network is: wherein, represents the local observation state of the i-th agent, represents the parameters of the i-th agent Actor network, represents the action output by the i-th agent, represents the global state information at time t, represents the parameters of the Critic network, represents the value estimate of the Critic network for the global state at time t, is the probability ratio of the new policy to the old policy, is the advantage function estimate, and T is the total number of time steps, is the immediate reward at time t, is the discount factor.

7. A double-layered agent optimization device for multi-robot arms, characterized by, The method comprises the following steps: A high-level task processing unit is configured to obtain environment information of a multi-robot cooperative carrying task, and call a high-level task planner to process the environment information, wherein the high-level task planner uses an RRT-Connect algorithm to perform initial path exploration, combines a B-spline to perform trajectory smoothing optimization on the initial path, and converts an object centroid trajectory into a reference trajectory sequence of each robot end effector through a kinematic chain model, and the environment information comprises an initial pose, a target pose and an environment map of a target object; A bottom layer task processing unit is configured to input the reference trajectory sequence as a guide signal into a bottom layer action executor, wherein the bottom layer action executor is composed of a plurality of independent mechanical arm agents, a MAPPO algorithm is called to perform distributed real-time control on each mechanical arm agent based on the reference trajectory sequence, and a control instruction is generated by combining the local observation of each mechanical arm agent to make a decision. A feedback unit is configured to obtain an execution effect of the control instruction, and feed back the bottom layer action executor based on the execution effect through a preset multi-layer reward function to guide each mechanical arm agent to learn a collaborative control strategy, so that the multiple mechanical arms effectively track the reference trajectory sequence and smooth the motion of each mechanical arm in the tracking process, wherein the multi-layer reward function includes a task completion reward, a smoothness reward, and a coordination reward, and the guide signal is an input of the task completion reward.

8. A double-layered agent optimization device for a multi-robot, characterized by, A computer program is stored in a memory, and the computer program can be executed by a processor to implement the double-layer agent optimization method for multiple mechanical arms according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program is stored in a computer readable storage medium, and the computer program can be executed by a processor of a device where the computer readable storage medium is located to implement the double-layer agent optimization method for multiple mechanical arms according to any one of claims 1 to 6.

Images