A dual-arm robot cooperative motion control method based on deep reinforcement learning

Abstract

The application discloses a kind of based on deep reinforcement learning's double-arm robot collaborative motion control method, DATD3 algorithm is used to train the mechanical arm collaborative motion control of double-arm robot, and the architecture of central training distribution execution is used in mechanical arm collaborative motion control system, and the control strategy of Critic evaluator in two intelligent agents will be used by each other;Action noise is used as the input of two intelligent agents, and the action value of two intelligent agents is output;The three-dimensional simulation environment of mechanical arm is established, and the action value output by two intelligent agents in step 1 is input into simulation environment, and state value is obtained by simulation training in simulation environment;State value is sampled and optimized using HER algorithm, and then the collaborative motion control strategy of double-arm robot is obtained;And based on the state value updated after optimization, the sampling strategy in two intelligent agents is updated;The method can effectively increase the learning efficiency of double-arm robot, and improve productivity.

Description

Technical Field

[0001] This invention belongs to the field of robot control, and in particular relates to a method for cooperative motion control of a dual-arm robot based on deep reinforcement learning. Background Technology

[0002] With the continuous development of robotics technology, robot assistance is becoming increasingly prevalent in production and daily life. Single-arm robots are already widely used in industrial handling, medical services, high-risk welding, and assembly line sorting. However, in more complex application scenarios, dual-arm collaborative robots are often more efficient than single-arm robots, such as completing agricultural harvesting tasks. Collaborative control is the foundation for dual-arm robots to flexibly complete complex agricultural harvesting tasks. It mainly refers to the coordinated movement between the two robotic arms to complete the task. This process involves multi-agent collaboration, motion control, and target perception.

[0003] Due to the high flexibility, high coupling, and high redundancy of six-DOF dual-arm robots, their control methods are extremely complex. Traditional methods first plan the robot's trajectory to address collision issues, then solve the forward and inverse kinematic equations to ensure multiple suitable solutions exist for the inverse kinematic equations. Finally, to ensure accurate execution of actions without damaging the target, appropriate dynamic constraints must be set, and the internal stresses of the target and the robot must be controlled. This method is computationally intensive and has poor adaptability to the unstructured spaces of agricultural harvesting.

[0004] Multi-agent motion control algorithms based on deep reinforcement learning provide a solution to this problem. Deep reinforcement learning algorithms enable agents to autonomously learn the mapping relationship between environmental states and actions. Agents explore spatial interactions through actions to obtain samples and adjust their action outputs based on feedback signals, thereby improving the collaborative ability of both arms.

[0005] The current deep reinforcement learning algorithm TD3 has begun to be gradually applied to the control of robotic arms, but the following problems still exist in the implementation of agricultural harvesting through the cooperative motion control of dual-arm robots: (1) In the training process of the robot, how to improve the adaptability and robustness of the algorithm while using deep reinforcement learning to achieve the cooperative motion control of the robot's dual arms; (2) Any physical system needs a certain amount of time to reach the state expected by the command. The most common method is to update the replay buffer at each step, but this method has a slow reading speed and stores a lot of invalid constraint boundary data, which cannot quickly provide effective experience for learning. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this application proposes a cooperative motion control method for dual-arm robots based on deep reinforcement learning. This invention improves the replay buffer update method and uses the HER method to enhance the training samples in the sparse reward-based DRL algorithm, which can effectively increase the learning efficiency of dual-arm robots and improve productivity.

[0007] The technical solution adopted in this invention is as follows:

[0008] A cooperative motion control method for a dual-arm robot based on deep reinforcement learning includes the following steps:

[0009] Step 1: The DATD3 algorithm is used to train the cooperative motion control of the dual-arm robot. The cooperative motion control system adopts a central training and distributed execution architecture. The Critic evaluator in the two agents will use each other's control strategies. Motion noise is used as the input of the two agents, and the two agents output motion values.

[0010] Step 2: Establish a three-dimensional simulation environment for the robotic arm, and input the action values ​​output by the two agents in Step 1 into the simulation environment. Obtain state values ​​through simulation training in the simulation environment; sample the state values ​​and use the HER algorithm to optimize the state values, thereby obtaining the collaborative motion control strategy of the dual-arm robot; and update the sampling strategy in the two agents based on the optimized state values.

[0011] Furthermore, the method for training the agent to obtain the control policy model in step 1 is as follows:

[0012] Step 1.1: Initialize the DATD3 network and set the hyperparameters of the Actor-Critic network for the two agents respectively; Step 1.2: Store the experience records as tuples B = (s, a, r, s′, done), where the parameters s, a, r, s′, and done correspond to the agent's current state, action, reward, next state, and round completion state, respectively; Step 1.3: Set the reward method, and at each timestep, judge and select the reward for the robot's action; Step 1.4: Process the sampled samples using the HER algorithm.

[0013] Step 1.5: At each time step, the Actor and Critic networks are updated by uniformly sampling from the experience replay pool.

[0014] Furthermore, the method for processing the sampled samples using the HER algorithm in step 1.4 is as follows:

[0015] Randomly select small batches of data from the experience replay pool, denote the original target as g, and replace the original target in turn;

[0016] From {s 1 a 1 g 1 ag 1 , ..., s t a t g t ag t Samples (s) are drawn from} i a i g i The sampling time is set to step i. The initial target g will be replayed by ag in step k, at which point g′←ag is obtained. Then the reward r′ is recalculated and the new tuple (s||g′,a,r′,s′||g′) is replaced and stored in the experience replay pool. The experience replay pool records the experience of all agents, t is the time, and ag is the state that the agent has reached.

[0017] Furthermore, the movement actions of the intelligent agent are divided into the following three states:

[0018] When the timestep is less than 10000, the agent accumulates learning experience through random actions;

[0019] When the timestep is in the range (10000, 40000), the above-mentioned AC network is used to add motion noise so that similar actions can produce similar output results.

[0020] When the timestep is greater than 40000, the motion noise is gradually reduced so that the agent can obtain more accurate output and complete smooth motion.

[0021] Furthermore, a 3D simulation environment for the robotic arm was established based on PyTorch.

[0022] Furthermore, when creating a 3D simulation environment for robotic arms in agricultural harvesting scenarios, it is necessary to simulate and calculate the working sampling areas of the two robotic arms, set a reward function, randomize the position of the harvested object, save the settings, and exit the environment.

[0023] Furthermore, the process of establishing a three-dimensional simulation environment for the robotic arm in an agricultural harvesting scenario is as follows:

[0024] Step 2.1.1: Install and activate Gym using Git based on the PyTorch framework;

[0025] Step 2.1.2: Create a description file for the robotic arm for 3D simulation, and import the STL (Stereo Lithography Apparatus) file corresponding to the UR5 dual-arm robot into the MuJoCo simulator. Use URDF to set the corresponding assembly relationship of each assembly to obtain the static model of the robot and determine the world coordinate points.

[0026] Step 2.1.3: Add a crop growth environment model and a deciduous crop model to the static model of the robot, create a picking task scene, set collision relationships, set site points on the end effector and configure two Mocap to obtain the robot arm motion observation values.

[0027] Step 2.1.4: Based on the requirements of the harvesting task, use Matlab to simulate and calculate the working sampling area of ​​the two robotic arms using the Monte Carlo algorithm;

[0028] Step 2.1.5: Define the reward function;

[0029] Step 2.1.6: Randomize the position of the object during algorithm training to ensure that the agent learns a more generalized cooperative motion strategy, thereby improving the accuracy of the simulation evaluation results;

[0030] Step 2.1.7: Save the above settings and exit the environment.

[0031] Furthermore, the reward function is divided into five parts:

[0032] One is that when the distance between the site point in the robotic arm model and the grasped target is less than 0.005m, the task is considered successful and a reward of +500 is given.

[0033] Secondly, the reward is inversely proportional to the distance. When the distance between the site point in the robotic arm model and the target being grasped is greater than 0.005m, the task is considered to have failed. In this case, a penalty of 5 times the distance between the site point and the target being grasped is given for each step. The reward is Reward = -(target distance * 5).

[0034] The third penalty is -1000 when the two robotic arms collide with each other;

[0035] Fourth, a penalty of -500 will be imposed when the two robotic arms move outside the designated work area;

[0036] Fifth, a penalty of -500 is imposed when the robot's action update steps exceed the maximum round steps by 1000.

[0037] Finally, the status, rewards, task completion information, etc. are returned in list form.

[0038] Furthermore, the process of sampling the state values ​​and optimizing them using the HER algorithm is as follows:

[0039] Step 2.2.1: Initialize the Critic network parameters for each agent. and Actor network parameters θ μ Where Q1 and Q2 represent the Critic network parameters of agent 1 and agent 2, respectively.

[0040] Step 2.2.2: Initialize target network parameters θ μ′ ←θ μ ;in, and θ μ′ To complete the initialization θ μ ;

[0041] Step 2.2.3: Initialize experience replay pool B;

[0042] Step 2.2.4: Initialize the random process N for the agent's exploration space actions and initialize the sampling target g and the initial state s;

[0043] Step 2.2.5: Select actions a to θ based on the noise feedback from each agent's exploration. μ (s)+N(0,σ); and calculate the reward r, and store the sample (s||g, a, r, s′, done||g) into B; where N is a random process, σ is action noise, a is the action, s′ is the next state, and done is the round completion state;

[0044] Step 2.2.6: Sample the target set in the current environment to replay an additional target set G, and sample another set of targets g′ in the target set G;

[0045] Step 2.2.7: For each agent, iterate through g′ in G and calculate the reward r′ = r(s, a, g′) corresponding to each g′, and store the replacement (s||g′, a, r′, s′, done||g′) into B; where g′ and r′ are the replaced sampling target and reward; Step 2.2.8: For each agent, uniformly sample 100 samples from B for network updates;

[0046] Step 2.2.9: Calculate the action a′=μ′s′|θ under state s′ using the Target Actor network. μ′ ); where μ′, θ μ′ , where represents the policy and Actor network parameters corresponding to state s′;

[0047] Step 2.2.10: Based on the target policy smoothing regularization, add noise ∈ to the target action a′, a′=a′+∈, ∈~clip(N(0,σ),-c,c);

[0048] Step 2.2.11: Based on the idea of ​​dual networks, calculate the target Q-value y of Critic1 and Critic2 networks respectively:

[0049]

[0050] When the timestep is greater than the set value, the noise range is reduced. Where y is the target Q value, r is the reward, and γ is the discount factor;

[0051] Step 2.2.12: Calculate the minimized evaluation value Q using the gradient descent algorithm. i Error between the target Q value and the target Q value Therefore, the Adam optimizer is used to optimize the parameters θ in the Critic1 and Critic2 networks. Q Update;

[0052] Step 2.2.13: After updating the Critic1 and Critic2 networks by step d, update the Actor network and use the Actor network to calculate the new action a under state s. new =μ(s|θ) μ 0;

[0053] Step 2.2.14: Use the Critic1 or Critic2 network to calculate action pairs (s, a) new The evaluation value q new The gradient ascent algorithm is used to maximize q. new Then, the Adam optimizer is used to optimize the Actor network parameters θ. μ Update;

[0054] Step 2.2.15: Finally, the following soft update method is used: the old target network parameters and the new target network parameters are weighted and averaged, and then assigned to the target Critic and Actor networks for updating.

[0055]

[0056] Where τ is the soft update parameter.

[0057] Furthermore, the error in step 2.2.12 is expressed as:

[0058] The beneficial effects of this invention are:

[0059] 1. This invention employs a dual-intelligence online learning reinforcement learning method to control the coordinated movement of a dual-arm robot, solving the problem that a single-arm robot cannot complete continuous movements in complex environments.

[0060] 2. The training speed and accuracy are improved by using the DATD3 algorithm, and the overall error loss is reduced by using the target network method.

[0061] 3. The architecture of centralized training and distributed execution is adopted, which improves the adaptability to unstable environments.

[0062] 4. The HER method is used to improve the training samples in the sparse reward-based DRL algorithm, thereby increasing training efficiency. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the structure of the deep reinforcement learning DATD3+HER algorithm used in this invention.

[0064] Figure 2 This is a flowchart illustrating the interaction between the algorithm of this invention and the dual-arm robot in the Mujoco simulation environment. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0066] Combined with appendix Figure 1 The present invention proposes a cooperative motion control method for a dual-arm robot based on deep reinforcement learning, which specifically includes the following steps:

[0067] Step 1: The DATD3 algorithm is used to train the collaborative motion control of the dual-arm robot. The collaborative motion control system adopts a central training and distributed execution architecture. The Critic evaluator in the two agents will use each other's control strategies. The motion noise (OU noise) is used as the input of the two agents, and the two agents output motion values.

[0068] In this embodiment, during the training of two agents, the actions and control strategies of the other agent can be evaluated in real time to achieve more efficient dual-arm coordinated movement. The HER method is used to improve the training samples in the sparse reward-based DRL algorithm, and the control strategy model is obtained by training the agents. The training steps are as follows:

[0069] Step 1.1: Initialize the DATD3 network. Set the hyperparameters of the AC (Actor-Critic) network for each agent: discount factor γ is set to 0.98, experience pool size M is set to 1e6, batch size m is set to 100, soft update parameter τ = 0.005, learning rate of the Actor network is set to 1e-4, learning rate of the Critic network is set to 1e-3, action noise σ is set to a maximum of 1.5 and a minimum of 0.2 as needed for training, max episode is set to 2000, and max timestep per round is set to 2000. Reset the robot to its initial position before each round's action execution.

[0070] Step 1.2: Store the experience record as a tuple B = (s, a, r, s′, done), where the parameters s, a, r, s′, and done correspond to the agent's current state, action, reward, next state, and round completion state, respectively.

[0071] The actions of an intelligent agent mainly fall into the following three states:

[0072] When the timestep is less than 10000, the agent accumulates learning experience through random actions;

[0073] When the timestep is in the range (10000, 40000), the above-mentioned AC network is used to add motion noise so that similar actions can produce similar output results.

[0074] When the timestep is greater than 40000, the motion noise is gradually reduced so that the agent can obtain more accurate output and complete smooth motion.

[0075] Step 1.3: At each timestep, the reward for the robot's action is determined and selected. In this application, a harvesting robot is used as an example, so its reward method consists of 5 parts, which are represented as follows:

[0076]

[0077] When the robot's movement exceeds the workspace or the number of steps exceeds the maximum number of rounds, the round ends (done=1). When the distance between the site point in the robotic arm model and the grasping target is less than 0.005m, the round is also considered to end and the robot returns to step 1.2. In other states, the robot continues to work.

[0078] Step 1.4: To improve the convergence speed and effectiveness of the algorithm, the HER algorithm is used to process the sampled samples.

[0079] Randomly select small batches of data from the replay buffer, denote the selected original target as g, and replace the target accordingly. From {s1 a 1 g 1 ag 1 , ..., s t a t g t ag t Samples (s) are drawn from} i a i g i The sampling time is set to step i. The initial goal g will be replayed by ag in step k (k=4). At this time, we can get g′←ag. Then we can recalculate the reward r′ and replace it with the new tuple (s||g′, a, r′, s′||g′) and store it in the replay buffer. The replay buffer will record the experience of all agents. ag is the state that the current agent has reached, that is, the state at the sampling termination time (achieved goal).

[0080] Step 1.5: At each time step, the Actor and Critic networks update by uniformly sampling a small batch of samples from the replay buffer.

[0081] Step 2: Establish a three-dimensional simulation environment for the robotic arm, and input the action values ​​output by the two agents in Step 1 into the simulation environment. Obtain state values ​​through simulation training in the simulation environment; sample the state values ​​and use the HER algorithm to optimize the state values, thereby obtaining the collaborative motion control strategy of the dual-arm robot; and update the sampling strategy in the two agents based on the optimized state values.

[0082] Step 2.1: Establish a 3D simulation environment for the robotic arm based on PyTorch. This mainly includes activating the deep reinforcement learning environment Gym, initializing the MuJoCo simulator, creating and setting observation points for the agricultural harvesting scenario, simulating and calculating the working sampling areas of the two robotic arms, setting the reward function, randomizing the position of the harvested object, saving the settings, and exiting the environment. The specific process is as follows:

[0083] Step 2.1.1: Install and activate Gym using Git based on the PyTorch framework;

[0084] Step 2.1.2: Create a description file for the robotic arm for 3D simulation, and import the STL (Stereo Lithography Apparatus) file corresponding to the UR5 dual-arm robot into the MuJoCo simulator. Use URDF to set the corresponding assembly relationship of each assembly to obtain the static model of the robot and determine the world coordinate points.

[0085] Step 2.1.3: Add a crop growth environment model and a deciduous crop model to the static model of the robot, create a picking task scene, set collision relationships, set site points on the end effector and configure two Mocap to obtain the robot arm motion observation values.

[0086] Step 2.1.4: Based on the requirements of the harvesting task, use Matlab to simulate and calculate the working sampling area of ​​the two robotic arms using the Monte Carlo algorithm;

[0087] Step 2.1.5: Define the reward function, which consists of the following five parts:

[0088] One is that when the distance between the site point in the robotic arm model and the grasped target is less than 0.005m, the task is considered successful and a reward of +500 is given.

[0089] Secondly, the reward is inversely proportional to the distance. When the distance between the site point in the robotic arm model and the target being grasped is greater than 0.005m, the task is considered to have failed. In this case, a penalty of 5 times the distance between the site point and the target being grasped is given for each step. The reward is Reward = -(target distance * 5).

[0090] The third penalty is -1000 when the two robotic arms collide with each other;

[0091] Fourth, a penalty of -500 will be imposed when the two robotic arms move outside the designated work area;

[0092] Fifth, a penalty of -500 is imposed when the robot's action update steps exceed the maximum round steps by 1000.

[0093] Finally, the status, rewards, task completion information, etc. are returned in list form.

[0094] Step 2.1.6: Randomize the position of the object during algorithm training to ensure that the agent learns a more generalized cooperative motion strategy, thereby improving the accuracy of the simulation evaluation results;

[0095] Step 2.1.7: Save the above settings and exit the environment.

[0096] Step 2.2, Combining Figure 1 The diagram shown illustrates the DATD3+HER algorithm structure of this invention. The specific process of sampling state values ​​and optimizing them using the HER algorithm in this method is as follows:

[0097] Step 2.2.1: Initialize the Critic network parameters for each agent. and Actor network parameters θ μWhere Q1 and Q2 represent the Critic network parameters of agent 1 and agent 2, respectively.

[0098] Step 2.2.2: Initialize target network parameters θ μ′ ←θ μ ;in, and θ μ′ To complete the initialization θ μ .

[0099] Step 2.2.3: Initialize the experience replay pool B (replay buffer B);

[0100] Step 2.2.4: Initialize the random process N for the agent's exploration space actions and initialize the sampling target g and the initial state s;

[0101] Step 2.2.5: Select actions a to θ based on the noise feedback from each agent's exploration. μ (s)+N(0,σ); and calculate the reward r, and store the sample (s||g, a, r, s′, done||g) into B; where N is a random process, σ is action noise, a is action, s′ is the next state, and done is the round completion state.

[0102] Step 2.2.6: Sample the target set in the current environment to replay an additional target set G, and sample another set of targets g′ in the target set G;

[0103] Step 2.2.7: For each agent, traverse g′ in G and calculate the reward r′ = r(s, a, g′) corresponding to each g′, and store the replacement (s||g′, a, r′, s′, done||g′) into B; where g′ and r′ are the replaced sampling target and reward.

[0104] Step 2.2.8: For each agent, uniformly sample 100 samples from B for network updates;

[0105] Step 2.2.9: Calculate the action a′=μ′(s′|θ) under state s′ using the Target Actor network. μ′ ); where μ′, θ μ′ , where represents the policy and Actor network parameters corresponding to state s′;

[0106] Step 2.2.10: Based on the target policy smoothing regularization, add noise ∈ to the target action a′, a′=a′+∈, ∈~clip(N(0,σ),-c,c);

[0107] Step 2.2.11: Based on the idea of ​​dual networks, calculate the target Q-value y of Critic1 and Critic2 networks respectively:

[0108]

[0109] When the timestep is greater than the set value, the noise range is reduced. Where y is the target Q value, r is the reward, and γ is the discount factor;

[0110] Step 2.2.12: Calculate the minimized evaluation value Q using the gradient descent algorithm. i Error between the target Q value and the target Q value

[0111]

[0112] Therefore, the Adam optimizer is used to optimize the parameters θ in the Critic1 and Critic2 networks. Q Perform the update; Step 2.2.13: After updating the Critic1 and Critic2 networks by step d, update the Actor network and use the Actor network to calculate the new action a under state s. new =μ(s|θ) μ );

[0113] Step 2.2.14: Use the Critic1 or Critic2 network to calculate action pairs (s, a) new ) evaluation value

[0114]

[0115] Maximizing q is achieved using the gradient ascent algorithm. new Then, the Adam optimizer is used to optimize the Actor network parameters θ. μ Update;

[0116] Step 2.2.15: Finally, the following soft update method is used: the old target network parameters and the new target network parameters are weighted and averaged, and then assigned to the target Critic and Actor networks for updating.

[0117]

[0118] Where τ is the soft update parameter.

[0119] The interaction flowchart between the algorithm of this invention and the dual-arm robot in the Mujoco simulation environment is as follows: Figure 2As shown, the trained DATD3 network model is saved every 10 iterations, and the last 20 training iterations are selected and imported into the simulation environment for testing. The test is set to a maximum of 1000 steps per round, with 50 rounds. The test success rate and average reward are output, and the convergence speed of the algorithm is judged and the algorithm performance is evaluated based on the average reward curve.

[0120] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A method for cooperative motion control of a dual-arm robot based on deep reinforcement learning, characterized in that, Includes the following steps: Step 1: The DATD3 algorithm is used to train the cooperative motion control of the dual-arm robot. The cooperative motion control system adopts a central training and distributed execution architecture. The Critic evaluator in the two agents will use each other's control strategies. Motion noise is used as the input of the two agents, and the two agents output motion values. Step 2: Establish a three-dimensional simulation environment for the robotic arm, and input the action values ​​output by the two agents in Step 1 into the simulation environment. Obtain state values ​​through simulation training in the simulation environment; sample the state values ​​and use the HER algorithm to optimize the state values, thereby obtaining the collaborative motion control strategy of the dual-arm robot; and update the sampling strategy in the two agents based on the optimized state values.

2. The method for cooperative motion control of a dual-arm robot based on deep reinforcement learning according to claim 1, characterized in that, The method for training the agent to obtain the control policy model in step 1 is as follows: Step 1.1: Initialize the DATD3 network and set the hyperparameters of the Actor-Critic network for the two agents respectively; Step 1.2: Store the experience records as tuples B = (s, a, r, s′, done), where the parameters s, a, r, s′, and done correspond to the agent's current state, action, reward, next state, and round completion state, respectively; Step 1.3: Set the reward method, and at each timestep, judge and select the reward for the robot's action; Step 1.4: Process the sampled samples using the HER algorithm. Step 1.5: At each time step, the Actor and Critic networks are updated by uniformly sampling from the experience replay pool.

3. The method for cooperative motion control of a dual-arm robot based on deep reinforcement learning according to claim 2, characterized in that, The method for processing the sampled samples using the HER algorithm in step 1.4 is as follows: Randomly select small batches of data from the experience replay pool, denote the original target as g, and replace the original target in turn; From {s 1 a 1 g 1 ag 1 , ..., s t a t g t ag t Samples (s) are drawn from} i a i g i The sampling time is set to step i. The initial target g will be replayed by ag in step k, at which point g′←ag is obtained. Then the reward r′ is recalculated and the new tuple (s||g′,a,r′,s′||g′) is replaced and stored in the experience replay pool. The experience replay pool records the experience of all agents, t is the time, and ag is the state that the agent has reached.

4. The method for cooperative motion control of a dual-arm robot based on deep reinforcement learning according to claim 2, characterized in that, The agent's movements are divided into the following three states: When the timestep is less than 10000, the agent accumulates learning experience through random actions; When the timestep is in the range (10000, 40000), the AC network is used to add motion noise so that similar actions can produce similar output results. When the timestep is greater than 40000, the motion noise is gradually reduced so that the agent can obtain more accurate output and complete smooth motion.

5. A method for cooperative motion control of a dual-arm robot based on deep reinforcement learning according to any one of claims 1-4, characterized in that, A 3D simulation environment for the robotic arm was built using PyTorch.

6. The method for cooperative motion control of a dual-arm robot based on deep reinforcement learning according to claim 5, characterized in that, When creating a 3D simulation environment for robotic arms in an agricultural harvesting scenario, it is necessary to simulate and calculate the working sampling areas of the two robotic arms, set a reward function, randomize the position of the harvested object, save the settings, and exit the environment.

7. The method for cooperative motion control of a dual-arm robot based on deep reinforcement learning according to claim 5, characterized in that, The process of establishing a 3D simulation environment for a robotic arm in an agricultural harvesting scenario is as follows: Step 2.1.1: Install and activate Gym using Git based on the PyTorch framework; Step 2.1.2: Create a description file for the robotic arm for 3D simulation, and import the STL (Stereo Lithography Apparatus) file corresponding to the UR5 dual-arm robot into the MuJoCo simulator. Use URDF to set the corresponding assembly relationship of each assembly to obtain the static model of the robot and determine the world coordinate points. Step 2.1.3: Add a crop growth environment model and a deciduous crop model to the static model of the robot, create a picking task scene, set collision relationships, set site points on the end effector and configure two Mocap to obtain the robot arm motion observation values. Step 2.1.4: Based on the requirements of the harvesting task, use Matlab to simulate and calculate the working sampling area of ​​the two robotic arms using the Monte Carlo algorithm; Step 2.1.5: Define the reward function; Step 2.1.6: Randomize the position of the object during algorithm training to ensure that the agent learns a more generalized cooperative motion strategy, thereby improving the accuracy of the simulation evaluation results; Step 2.1.7: Save the above settings and exit the environment.

8. The method for cooperative motion control of a dual-arm robot based on deep reinforcement learning according to claim 7, characterized in that, The reward function consists of five parts: One is that when the distance between the site point in the robotic arm model and the grasped target is less than 0.005m, the task is considered successful and a reward of +500 is given. Secondly, the reward is inversely proportional to the distance. When the distance between the site point in the robotic arm model and the target being grasped is greater than 0.005m, the task is considered to have failed. In this case, a penalty of 5 times the distance between the site point and the target being grasped is given for each step. The reward is Reward = -(target distance * 5). The third penalty is -1000 when the two robotic arms collide with each other; Fourth, a penalty of -500 will be imposed when the two robotic arms move outside the designated work area; Fifth, a penalty of -500 is imposed when the robot's action update steps exceed the maximum round steps by 1000. Finally, the status, rewards, task completion information, etc. are returned in list form.

9. The method for cooperative motion control of a dual-arm robot based on deep reinforcement learning according to claim 7, characterized in that, The process of sampling the state values ​​and optimizing them using the HER algorithm is as follows: Step 2.2.1: Initialize the Critic network parameters for each agent. and Actor network parameters θ μ Where Q1 and Q2 represent the Critic network parameters of agent 1 and agent 2, respectively. Step 2.2.2: Initialize target network parameters θ μ′ ←θ μ ;in, and θ μ′ To complete the initialization θ μ ; Step 2.2.3: Initialize experience replay pool B; Step 2.2.4: Initialize the random process N for the agent's exploration space actions and initialize the sampling target g and the initial state s; Step 2.2.5: Select actions a to θ based on the noise feedback from each agent's exploration. μ (s)+N(0,σ); and calculate the reward r, and store the sample (s||g, a, r, s′, done||g) into B; where N is a random process, σ is action noise, a is action, s′ is the next state, and done is the round completion state; Step 2.2.6: Sample the target set in the current environment to replay an additional target set G, and sample another set of targets g′ in the target set G; Step 2.2.7: For each agent, iterate through g′ in G and calculate the reward r′ = r(s, a, g′) corresponding to each g′, and store the replacement (s||g′, a, r′, s′, done||g′) into B; where g′ and r′ are the replaced sampling target and reward; Step 2.2.8: For each agent, uniformly sample 100 samples from B for network updates; Step 2.2.9: Calculate the action a′=μ′(s′|θ) under state s′ using the Target Actor network. μ′ ); where μ′, θ μ′ , where represents the policy and Actor network parameters corresponding to state s′; Step 2.2.10: Based on the target policy smoothing regularization, add noise ∈ to the target action a′, a′=a′+∈, ∈~clip(N(0,σ),-c,c); Step 2.2.11: Based on the idea of ​​dual networks, calculate the target Q-value y of Critic1 and Critic2 networks respectively: When the timestep is greater than the set value, the noise range is reduced. Where y is the target Q value, r is the reward, and γ is the discount factor; Step 2.2.12: Calculate the minimized evaluation value Q using the gradient descent algorithm. i Error between the target Q value and the target Q value Therefore, the Adam optimizer is used to optimize the parameters θ in the Critic1 and Critic2 networks. Q Update; Step 2.2.13: After updating the Critic1 and Critic2 networks by step d, update the Actor network and use the Actor network to calculate the new action a under state s. new =μ(s|θ) μ ); Step 2.2.14: Use the Critic1 or Critic2 network to calculate action pairs (s, a) new The evaluation value q new The gradient ascent algorithm is used to maximize q. new Then, the Adam optimizer is used to optimize the Actor network parameters θ. μ Update; Step 2.2.15: Finally, the following soft update method is used: the old target network parameters and the new target network parameters are weighted and averaged, and then assigned to the target Critic and Actor networks for updating. Where τ is the soft update parameter.

10. A method for cooperative motion control of a dual-arm robot based on deep reinforcement learning according to claim 9, characterized in that, The error in step 2.2.12 is expressed as follows:

Images