Multi-arm coordination action planning method based on improved reinforcement learning algorithm

Abstract

The invention discloses a multi-arm coordination action planning method based on an improved reinforcement learning algorithm, which comprises the following steps of: establishing a six-degree-of-freedom mechanical arm kinematic model by adopting a DH method, and modeling a multi-arm coordination reinforcement learning process of a mechanical arm into a Markov decision process; a reward function is designed according to the relative position of the mechanical arm in the moving process, the distance from the target point and the distance from the obstacle; an attention mechanism, priority track experience playback and a course learning strategy are organically fused, a traditional MADDPG framework is systematically improved, and an ATP-MADDPG framework is provided, so that the ATP-MADDPG framework has the capabilities of adaptive feature extraction, efficient utilization of high-value collaborative experience and staged learning of complex tasks; according to the scheme, on the basis of the proposed improved ATP-MADDPG algorithm, the algorithm is applied to the multi-arm cooperative motion planning problem, and the core challenges of strategy instability, low cooperation efficiency, poor convergence in a complex environment and the like caused by high-dimensional state coupling are effectively solved.

Description

Technical Field

[0001] This invention belongs to the field of robotic arm motion planning, specifically relating to a multi-arm collaborative motion planning method based on an improved reinforcement learning algorithm. Background Technology

[0002] In recent years, robotic arms have become an indispensable component of modern automation due to their high-precision positioning, strong load-bearing capacity, and stable post-maintenance operation. With advancements in intelligent manufacturing and artificial intelligence technologies, robotic arms have been widely applied in industrial production, medical assistance, space exploration, hazardous operations, and precision assembly, performing exceptionally well in tasks requiring high repeatability and stability. However, single-arm robotic arms suffer from physical and perceptual limitations, such as a limited number of end effectors, restricted workspace, insufficient sensors, and limited dexterity. To overcome these limitations, multi-arm collaborative robot systems have emerged. Compared to single-arm robotic arms, multi-arm collaboration effectively expands the task space, enhances adaptability to complex environments, and demonstrates higher efficiency and scalability in both theory and practice.

[0003] During task execution, the motion planning of a robotic arm involves safely navigating from a given starting point to a target location while avoiding all obstacles and collisions, all within the constraints of kinematics and dynamics. This element is crucial for ensuring the safe and efficient execution of the task and is also the technical guarantee for the successful operation of the robotic arm. Motion planning methods are mainly divided into two categories: learning-based and non-learning-based. Non-learning-based methods typically employ centralized or distributed architectures. Although they are computationally efficient, they cannot guarantee global optimality and may face the risk of deadlock. These shortcomings limit the practical application of this method.

[0004] Among learning-based algorithms, MADDPG has attracted widespread attention from scholars worldwide due to its simple structure and applicability to continuous action spaces. However, it still faces two major challenges. The first challenge is the inherent overestimation bias and policy overfitting in DDPG, which are further exacerbated in multi-agent scenarios. The second challenge is effectively capturing the nonlocal dependencies between agents in complex high-dimensional environments, which leads to slow convergence speed and limited policy stability. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-arm cooperative action planning method based on an improved reinforcement learning algorithm, which solves the problems of dimensionality explosion and low learning efficiency when the traditional MADDPG is applied to multi-arm cooperative action planning.

[0006] The specific technical solution for achieving the objective of this invention is as follows:

[0007] A multi-arm cooperative action planning method based on an improved reinforcement learning algorithm includes the following steps:

[0008] Step 1: Establish a kinematic model of a six-degree-of-freedom robotic arm using the DH method;

[0009] Step 2: Model the multi-arm collaborative reinforcement learning process of the robotic arm as a Markov decision process, and define the action space, state space and reward function;

[0010] Step 3: Solve the multi-arm cooperative reinforcement learning process of the robotic arm based on the improved ATP-MADDPG algorithm to obtain the optimal multi-arm cooperative motion planning scheme.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0012] (1) The present invention addresses the inherent overestimation bias and policy overfitting problems of the existing DDPG framework by designing an improved ATP-MADDPG algorithm. The algorithm embeds an attention module in the value network to achieve key information focus, and combines priority sampling and importance sampling weights for bias correction. At the same time, it adopts a course learning strategy to gradually increase the task complexity, thereby significantly improving the value estimation accuracy and policy stability.

[0013] (2) This scheme addresses the problem of insufficient modeling of nonlocal dependencies among multiple agents by improving the ATP-MADDPG algorithm to dynamically calculate the association weights between agents by scaling dot product attention and constructing context vectors to aggregate global collaborative information. This enables policy gradient updates to adaptively focus on key interaction features, greatly enhancing the ability to capture long-term dependencies and convergence efficiency in multi-arm collaboration.

[0014] The present invention will be further described below with reference to specific embodiments. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the multi-arm cooperative motion planning method based on the improved reinforcement learning algorithm of the present invention.

[0016] Figure 2 This is a schematic diagram of a six-degree-of-freedom robotic arm in an embodiment of the present invention.

[0017] Figure 3 This is a general linear rod model of a six-degree-of-freedom robotic arm in an embodiment of the present invention.

[0018] Figure 4 This is a schematic diagram illustrating the process of decomposing tasks using a course learning strategy according to the present invention.

[0019] Figure 5 This is a schematic diagram of the improved ATP-MADDPG algorithm of the present invention.

[0020] Figure 6 This is a simulation diagram of the reward based on the algorithm of this scheme and two baseline algorithms in an embodiment of the present invention.

[0021] Figure 7 The above is a simulation graph showing the success rate of the algorithm based on this scheme and two baseline algorithms in an embodiment of the present invention. Detailed Implementation

[0022] Example

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0025] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0026] Combination Figure 1 A multi-arm cooperative action planning method based on an improved reinforcement learning algorithm includes the following steps:

[0027] Step 1: Establish a kinematic model of a six-DOF robotic arm using the DH method:

[0028] like Figure 2 The diagram shown is a schematic of the six-degree-of-freedom robotic arm studied in this application. For ease of identification, it has been converted to... Figure 3 A typical robotic arm linear model;

[0029] No. The joint coordinate system relative to the first joint coordinate system The homogeneous transformation matrix of the joint coordinate system is given by the following equation:

[0030]

[0031] Among them, subscript The table above indicates the serial number of the robotic arm. Indicates the joint number. This represents the rod length between adjacent joint coordinate systems. Represents the chain twist angle between adjacent joint coordinate systems. This indicates the axial offset of the previous joint. It is the rotation angle about the axis of the previous joint;

[0032] This formula defines the homogeneous transformation matrix for each robotic arm joint;

[0033] The attitude of the end effector of the entire robotic arm is obtained through chain multiplication:

[0034]

[0035] This represents the actuator attitude transformation matrix of the end effector joint. Let be the coordinate transformation matrix from the (j-1)th joint to the jth joint of the p-th robotic arm.

[0036] This formula represents the sequential product of all joint transformation matrices from the base to the end effector, used to calculate the attitude of the end effector.

[0037] Step 2: Model the multi-arm collaborative reinforcement learning process of the robotic arm as a Markov decision process, defining the action space, state space, and reward function:

[0038] Step 2-1: Represent the multi-arm collaboration of the robotic arm as a tuple. ,in, express A collection of robotic arms; It is a state space, in which Indicates the first The state of each robotic arm consists of the position and orientation of the end effector of each robotic arm in Cartesian space, as well as the position of the task target. It is the action space. Indicates the first The movement of a robotic arm, i.e., the joint angle or its increment , that is, the controllable variables in the transformation matrix; Define the action to be performed Then from the current state To the next state The state transition probability; Represents the reward set, in which It is the first The reward obtained by each robotic arm from the environment is obtained by transforming the position of each robotic arm's links and end effector in space through a transformation matrix. These positions are then used to evaluate constraints such as collisions and boundary violations and incorporate them into the reward function.

[0039] Step 2-2: Define the objective of the multi-arm cooperative motion planning problem in reinforcement learning, which is to find an optimal policy in the joint space to minimize the error between the end effector and the target while avoiding collisions. This is achieved by learning the optimal policy. To maximize one's cumulative rewards:

[0040]

[0041] Among them, Indicates in A discount factor at any given moment is used to balance current rewards with future rewards. Representation strategy, Let represent the reward of the i-th robotic arm at time t;

[0042] Steps 2-3: Define the action space and state space.

[0043] The action space is defined as the set of all possible actions, where the output of the policy network is called the agent's action. In robotic arm motion planning tasks, the dimensions of the action space correspond to the degrees of freedom (DOF) of the robotic arm.

[0044] In robotic arm motion planning tasks, the dimension of the motion space corresponds to the degrees of freedom (DOF) of the robotic arm; the agent's motion is defined as the joint angle increment of the robotic arm, denoted as... The activation function of the last layer of the policy network is the hyperbolic tangent (tanh) function, which restricts the output of each dimension to a certain value. Within a reasonable range. To ensure that the robotic arm's movements remain within a reasonable range, a scaling parameter is introduced. .parameter Setting it to 0.1 means that the single-step movement of each joint should not exceed 0.1 radians.

[0045] Define the state space:

[0046] At each time step, the joint angles of the robotic arm are calculated based on the joint angles of the previous time step:

[0047]

[0048] in, This is the gain coefficient;

[0049] By using joint angles and the forward kinematics model of the robotic arm, the positions of each joint and the end effector in the next time step are determined.

[0050] In addition, the rotation angle of each robotic arm's joints is constrained, with the lower limit set at... The upper limit is set to ;

[0051] Steps 2-4: Define the reward function. This solution proposes a newly designed reward function to guide the robotic arm towards a designated target while avoiding other drones and obstacles. This reward function comprises the following components:

[0052]

[0053] Includes: Arrival Reward: This reward mechanism is designed to guide the robotic arm's end effector to quickly approach and reach the target. The closer the end effector is to the target point, the higher the reward value.

[0054]

[0055] Collision / Avoidance Reward: This reward component aims to prevent collisions during the robot's operation. The reward function is constructed based on the distances between the robot's segments. To address the issue of sparse reward distribution, an improved Artificial Potential Field (APF) method is employed. This method combines attractive forces with exploratory behavior, effectively mitigating the local minima problem common in traditional APF methods. The artificial potential field method is used to establish attractive and repulsive fields within the workspace. The robot then moves under the combined influence of these fields, its motion being affected by both attractive and repulsive forces.

[0056] Attracting Rewards:

[0057]

[0058] That is, if the end effector moves closer to the target, a positive reward is given; otherwise, a penalty is imposed.

[0059] Rejection of rewards:

[0060]

[0061] If the minimum distance between robotic arms is less than the safe distance, a negative reward is applied;

[0062] Exploration Rewards:

[0063]

[0064] When the current state is worse than the "normal" level, exploratory actions that may temporarily perform poorly but help to get out of local minima may be rewarded with a certain probability.

[0065] in, Indicates reaching the reward. Indicates the position of the end effector of the robotic arm. Indicates the location of the target point. Indicates an attractive reward or incentive. This indicates the distance between the robotic arm's end effector and the target at the current time step. This indicates the distance between the end effector of the robotic arm and the target at the previous time step. , This represents the reward weighting coefficient; This indicates a rejection of rewards. This represents the minimum inter-arm distance outside the end effector of the robotic arm. To establish a safe distance, This represents the set repulsion reward gain coefficient. Indicates exploration reward, This indicates the set reward value. It is a random number between 0 and 1. This indicates the energy change of the robotic arm. These are the training parameters in the current iteration.

[0066] Step 3: Solve the multi-arm cooperative reinforcement learning process of the robotic arm based on the improved ATP-MADDPG algorithm to obtain the optimal multi-arm cooperative motion planning scheme.

[0067] Combination Figure 5 In the reinforcement learning process, the robotic arm is first initialized with a random joint configuration and target position. Based on the current joint angle and the constructed six-degree-of-freedom robotic arm kinematic model, the attitude of the end effector is calculated. This attitude, together with the joint angle and joint angular velocity, constitutes the state representation of the agent.

[0068] The state representation is input into the policy network of the improved ATP-MADDPG algorithm, which outputs a motion in the form of joint angle increments and updates the joint configuration with the motion. The updated configuration is then substituted into the kinematic model of the 6-DOF robotic arm to calculate the new end effector posture.

[0069] At this point, the value of the action is calculated by evaluating the distance between the end effector and the target, as well as the minimum distance between multiple robotic arms, using a reward function.

[0070] Finally, the current state, the action performed, the reward obtained, and the next state are stored in the experience buffer for updating the policy and value network, thus completing an iterative process.

[0071] In this scheme, the improved ATP-MADDPG algorithm introduces a multi-head attention mechanism in the value network to adaptively focus on key observation features and filter out irrelevant information;

[0072] The improved MADDPG framework introduces a scaled dot product attention mechanism for information aggregation in the value network, enabling each agent to adaptively focus on information from other agents associated with it. For the i-th robotic arm, its input features are:

[0073]

[0074] in, This represents the observation of the i-th robotic arm. This represents the action of the i-th robotic arm;

[0075] The information set of other robotic arms is

[0076]

[0077] The attention module in the value network calculates a similarity score, denoted as . This is used to quantify the influence of other robotic arms j on robotic arm i. The attention score represents the correlation between robotic arm i and robotic arm j.

[0078]

[0079] in and These are the linear projection matrices of the query vector and the key vector, respectively. The normalization factor for the feature dimension is used, and the aggregated context vector is:

[0080]

[0081] A linear projection matrix representing value;

[0082] The input to the value network is expanded to:

[0083]

[0084] Based on this, the estimation expression for the joint action value function is:

[0085]

[0086] in This represents the computation performed on the improved value network;

[0087] The improved policy network is represented as follows:

[0088]

[0089] in The parameters of the policy network are represented. It is a multilayer perceptron (MLP). These are the linear projections of the query, key, and value, respectively. This represents a multi-head attention operation, and the attention weights are calculated as follows:

[0090]

[0091] in The similarity score is calculated by evaluating the attention module in the evaluation network and is used to assess the influence of robotic arm j on robotic arm i.

[0092]

[0093] Attention weight and It reflects the dynamic correlation between different features, enabling the policy gradient to adaptively emphasize the optimization of key features;

[0094] The policy gradient update direction changes from uniform feature mapping to a weighted feature aggregation form, i.e.

[0095]

[0096] Policy gradient It was calculated using the standard backpropagation algorithm.

[0097] Furthermore, in solving the multi-arm cooperative reinforcement learning process of the robotic arm based on the improved ATP-MADDPG algorithm, a priority sequence sampling strategy is used to extract continuous action sequences of the robotic arm cooperative motion from the experience replay buffer.

[0098] During sampling, the Sum tree algorithm is introduced to achieve efficient weighted sampling of multi-arm motion trajectory data, ensuring that the sampling process can focus on interactive segments with high collaborative information.

[0099] Here, the priority weight of the i-th sample in efficient weighted sampling is... for:

[0100]

[0101] in, Indicates the length of the sequence. The coefficient representing the control over the influence of TD error. This represents the TD error at each time step;

[0102] The sampling probability for each action sequence is:

[0103]

[0104] j represents the number of consecutive action sequences in the experience playback buffer;

[0105] In addition, this scheme also uses importance sampling (IS) weights to compensate for the bias introduced by non-uniform sampling:

[0106]

[0107] in, The importance sampling weight is represented by γ, where N is the total capacity of the empirical replay buffer, and γ is the exponential decay coefficient of the importance sampling weight. During gradient update calculation, the contribution of each sample is multiplied by its corresponding IS weight ω. i This balances the sampling bias;

[0108] This weight value reflects the preference for sampling higher-priority samples. When calculating gradient updates or TD error updates, the contribution of each sample is multiplied by the corresponding IS weight ω. i This is equivalent to giving greater weight to "samples sampled with low probability" and less weight to "samples sampled with high probability", thereby balancing the sampling bias.

[0109] The sampled continuous action sequence is directly used for the collaborative optimization of the policy network and the value network, and its temporal difference error is recalculated based on the updated network parameters;

[0110] Then, through the reverse indexing mechanism, all sequences containing key robotic arm state transitions are quickly located, including trajectory segments of multi-arm cooperative obstacle avoidance, synchronous operation, or dynamic target tracking.

[0111] Subsequently, the Sum tree is based on priority weights. The system will be updated synchronously based on the latest calculated error and the performance of the robotic arm, forming a closed-loop optimization process:

[0112] This process is continuously iterated to ensure that the playback process highlights the most informative sequences while maintaining the temporal correlation within the data.

[0113] Finally, in the multi-arm collaborative motion planning, this solution adopts a course learning strategy, which decomposes complex tasks into task models with progressively increasing difficulty for robotic arm motion learning.

[0114] In the initial stage, the robotic arm will learn how to reach a fixed target point in an obstacle-free environment;

[0115] After learning is completed, based on the accumulated data, the system enters the task environment of moving the target point for further learning.

[0116] Finally, obstacles are introduced into the environment, raising the task difficulty to its highest level, combining obstacles with moving target points.

[0117] As training progresses, the robotic arm can gradually master basic skills such as dynamic collision avoidance, cooperative motion planning, and synchronous multi-arm operation, effectively reducing the exploration space, lowering the risk of catastrophic failure, and reducing the sample complexity required to learn high-dimensional coordination strategies.

[0118] Figure 6 and Figure 7 The performance of the proposed ATP-MADDPG algorithm compared with existing baseline methods in multi-arm cooperative tasks is shown;

[0119] As can be seen, compared with other methods, the ATP-MADDPG algorithm applied in this scheme significantly improves the task success rate and achieves the goal more reliably. Furthermore, the algorithm maintains a high cumulative reward value throughout the entire training process. The reward curve exhibits a smooth upward trend, further demonstrating its faster convergence speed and more stable training performance, enabling it to handle complex scenarios involving multi-arm collaboration and obstacle avoidance more efficiently.

[0120] This solution also provides a multi-arm cooperative action planning system based on an improved reinforcement learning algorithm, including the following modules:

[0121] Kinematics Modeling Module: Used to build a kinematic model of a six-DOF robotic arm using the DH method;

[0122] Multi-arm cooperative motion planning module: used to model the multi-arm cooperative reinforcement learning process of robotic arms as a Markov decision process, defining the action space, state space and reward function;

[0123] The improved ATP-MADDPG algorithm is used to solve the multi-arm cooperative reinforcement learning process of the robotic arm and obtain the optimal multi-arm cooperative motion planning scheme.

[0124] This solution also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0125] Step 1: Establish a kinematic model of a six-degree-of-freedom robotic arm using the DH method;

[0126] Step 2: Model the multi-arm collaborative reinforcement learning process of the robotic arm as a Markov decision process, and define the action space, state space and reward function;

[0127] Step 3: Solve the multi-arm cooperative reinforcement learning process of the robotic arm based on the improved ATP-MADDPG algorithm to obtain the optimal multi-arm cooperative motion planning scheme.

[0128] The embodiments described above are merely one implementation method of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A multi-arm cooperative action planning method based on an improved reinforcement learning algorithm, characterized in that, Includes the following steps: Step 1: Establish a kinematic model of a six-degree-of-freedom robotic arm using the DH method; Step 2: Model the multi-arm collaborative reinforcement learning process of the robotic arm as a Markov decision process, and define the action space, state space and reward function; Step 3: Solve the multi-arm cooperative reinforcement learning process of the robotic arm based on the improved ATP-MADDPG algorithm to obtain the optimal multi-arm cooperative motion planning scheme.

2. The multi-arm cooperative action planning method based on the improved reinforcement learning algorithm according to claim 1, characterized in that, The kinematic model of the 6-DOF robotic arm in step 1 is as follows: ； Among them, subscript The table above indicates the serial number of the robotic arm. Indicates the joint number. This represents the rod length between adjacent joint coordinate systems. Represents the chain twist angle between adjacent joint coordinate systems. This indicates the axial offset of the previous joint. It is the rotation angle about the axis of the previous joint; ； This represents the actuator attitude transformation matrix of the end effector joint. Let be the coordinate transformation matrix from the (j-1)th joint to the jth joint of the p-th robotic arm.

3. The multi-arm cooperative action planning method based on an improved reinforcement learning algorithm according to claim 1, characterized in that, Step 2, which models the multi-arm collaborative reinforcement learning process of the robotic arm as a Markov decision process, specifically involves: Step 2-1: Represent the multi-arm collaboration of the robotic arm as a tuple. ,in, express A collection of robotic arms; It is a state space, in which Indicates the first The state of each robotic arm consists of the position and orientation of the end effector of each robotic arm in Cartesian space, as well as the position of the task target. It is the action space. Indicates the first The movement of a robotic arm, i.e., the joint angle or its increment , that is, the controllable variables in the transformation matrix; Define the action to be performed Then from the current state To the next state The state transition probability; Represents the reward set, in which It is the first The reward that a robotic arm obtains from the environment is obtained by transforming the position of each robotic arm's links and end effector in space using a transformation matrix; Step 2-2: Define the objective of the multi-arm cooperative motion planning problem in reinforcement learning: ； Among them, Indicates in Discount factor of time, Representation strategy, Let represent the reward of the i-th robotic arm at time t; Steps 2-3: Define the action space and state space; Define the action space: In robotic arm motion planning tasks, the dimension of the motion space corresponds to the degrees of freedom (DOF) of the robotic arm; the agent's motion is defined as the joint angle increment of the robotic arm, denoted as... ; Define the state space: At each time step, the joint angles of the robotic arm are calculated based on the joint angles of the previous time step: ； in, This is the gain coefficient; By using joint angles and the forward kinematics model of the robotic arm, the positions of each joint and the end effector in the next time step are determined. In addition, the rotation angle of each robotic arm's joints is constrained, with the lower limit set at... The upper limit is set to ; Steps 2-4: Define the reward function: ； ； ； ； ； in, Indicates reaching the reward. Indicates the position of the end effector of the robotic arm. Indicates the location of the target point. Indicates an attractive reward or incentive. This indicates the distance between the robotic arm's end effector and the target at the current time step. This indicates the distance between the end effector of the robotic arm and the target at the previous time step. , This represents the reward weighting coefficient; This indicates a rejection of rewards. This represents the minimum inter-arm distance outside the end effector of the robotic arm. To establish a safe distance, This represents the set repulsion reward gain coefficient. Indicates exploration reward, This indicates the set reward value. It is a random number between 0 and 1. This indicates the energy change of the robotic arm. These are the training parameters in the current iteration.

4. The multi-arm cooperative action planning method based on the improved reinforcement learning algorithm according to claim 2, characterized in that, In step 3, the improved ATP-MADDPG algorithm is used to iteratively solve the multi-arm cooperative reinforcement learning process of the robotic arm to obtain the optimal multi-arm cooperative motion planning scheme, specifically as follows: During reinforcement learning, the robotic arm is first initialized with a random joint configuration and target position. Based on the current joint angles and the constructed six-degree-of-freedom robotic arm kinematic model, the attitude of the end effector is calculated. This attitude, together with the joint angles and joint angular velocities, constitutes the state representation of the agent. The state representation is input into the policy network of the improved ATP-MADDPG algorithm, which outputs a motion in the form of joint angle increments and updates the joint configuration with the motion. The updated configuration is then substituted into the kinematic model of the 6-DOF robotic arm to calculate the new end effector posture. At this point, the value of the action is calculated by evaluating the distance between the end effector and the target, as well as the minimum distance between multiple robotic arms, using a reward function. Finally, the current state, the action performed, the reward obtained, and the next state are stored in the experience buffer for updating the policy and value network, thus completing an iterative process.

5. The multi-arm cooperative action planning method based on the improved reinforcement learning algorithm according to claim 4, characterized in that, The improved ATP-MADDPG algorithm introduces a multi-head attention mechanism into the value network to adaptively focus on key observation features and filter out irrelevant information. The improved MADDPG framework introduces a scaled dot product attention mechanism for information aggregation in the value network, enabling each agent to adaptively focus on information from other agents associated with it. For the i-th robotic arm, its input features are: ； in, This represents the observation of the i-th robotic arm. This represents the action of the i-th robotic arm; The information set of other robotic arms is ； The attention module in the value network calculates a similarity score, denoted as . This is used to quantify the influence of other robotic arms j on robotic arm i. The attention score represents the correlation between robotic arm i and robotic arm j. ； in and These are the linear projection matrices of the query vector and the key vector, respectively. The normalization factor for the feature dimension is used, and the aggregated context vector is: ； A linear projection matrix representing value; The input to the value network is expanded to: ； Based on this, the estimation expression for the joint action value function is: ； in This represents the computation performed on the improved value network; The improved policy network is represented as follows: ； in The parameters of the policy network are represented. It is a multilayer perceptron (MLP). These are the linear projections of the query, key, and value, respectively. This represents a multi-head attention operation, and the attention weights are calculated as follows: ； in The similarity score is calculated by evaluating the attention module in the evaluation network and is used to assess the influence of robotic arm j on robotic arm i. ； Attention weight and It reflects the dynamic correlation between different features, enabling the policy gradient to adaptively emphasize the optimization of key features; The policy gradient update direction changes from uniform feature mapping to a weighted feature aggregation form, i.e. ； Policy gradient It was calculated using the standard backpropagation algorithm.

6. The multi-arm cooperative action planning method based on the improved reinforcement learning algorithm according to claim 4, characterized in that, In step 3, during the process of solving the multi-arm cooperative reinforcement learning process of the robotic arm based on the improved ATP-MADDPG algorithm, a priority sequence sampling strategy is used to extract the continuous action sequence of the robotic arm's cooperative motion from the experience replay buffer. During sampling, the Sum tree algorithm is introduced to achieve efficient weighted sampling of multi-arm motion trajectory data, ensuring that the sampling process can focus on interactive segments with high collaborative information. The sampled continuous action sequence is directly used for the collaborative optimization of the policy network and the value network, and its temporal difference error is recalculated based on the updated network parameters; Then, through the reverse indexing mechanism, all sequences containing key robotic arm state transitions are quickly located, including trajectory segments of multi-arm cooperative obstacle avoidance, synchronous operation, or dynamic target tracking. Subsequently, the Sum tree, based on priority weights, will be updated synchronously according to the latest calculated error and the performance of the robotic arm task, forming a closed-loop optimization process; This process is continuously iterated to ensure that the playback process highlights the most informative sequences while maintaining the temporal correlation within the data.

7. The multi-arm cooperative action planning method based on the improved reinforcement learning algorithm according to claim 6, characterized in that, In sampling, the priority weight of the i-th sample. for: ； in, Indicates the length of the sequence. The coefficient representing the control over the influence of TD error. This represents the TD error at each time step; The sampling probability for each action sequence is: ； j represents the number of consecutive action sequences in the experience playback buffer; In addition, importance sampling weights are used to compensate for the bias introduced by non-uniform sampling: ； in, The importance sampling weight is represented by γ, where N is the total capacity of the experience playback buffer, and γ is the exponential decay coefficient of the set importance sampling weight.

8. The multi-arm cooperative action planning method based on an improved reinforcement learning algorithm according to any one of claims 1-7, characterized in that, In multi-arm collaborative motion planning, a course learning strategy is adopted to decompose complex tasks into task models with progressively increasing difficulty for robotic arm motion learning. In the initial stage, the robotic arm will learn how to reach a fixed target point in an obstacle-free environment; After learning is completed, based on the accumulated data, the system enters the task environment of moving the target point for further learning. Finally, obstacles are introduced into the environment, raising the task difficulty to its highest level, combining obstacles with moving target points.

9. A multi-arm cooperative action planning system based on an improved reinforcement learning algorithm, characterized in that, Includes the following modules: Kinematics Modeling Module: Used to build a kinematic model of a six-DOF robotic arm using the DH method; Multi-arm cooperative motion planning module: used to model the multi-arm cooperative reinforcement learning process of robotic arms as a Markov decision process, defining the action space, state space and reward function; The improved ATP-MADDPG algorithm is used to solve the multi-arm cooperative reinforcement learning process of the robotic arm and obtain the optimal multi-arm cooperative motion planning scheme.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-7.

Images