Double-arm robot cooperative control method based on multi-agent reinforcement learning

Abstract

The invention provides a dual-arm robot cooperative control method based on multi-agent reinforcement learning. The dual-arm robot cooperative control method comprises the steps that system control of a dual-arm robot is converted into a Markov decision problem; establishing a double-arm robot simulation environment, and designing rounds and tasks according to reinforcement learning problems; and constructing a multi-agent reinforcement learning model, and training the simulation model through the multi-agent reinforcement learning model. According to the method, the two mechanical arms are regarded as independent agents for multi-agent reinforcement learning training, the actual movement condition is better met, and the success rate of a cooperative task is effectively increased; when the actions are selected, the actions are scored through the Critic network, then the scores of the two actions are compared, the action with the higher score is selected for execution, the problem of local optimum of the actions occurring in the training process and the problem of over-estimation or underestimation occurring when the Critic network judges the value of the actions are solved, and the stability and accuracy of cooperative control tasks of the two-arm robot are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of robot collaborative control, and in particular to a dual-arm robot collaborative control method based on multi-agent reinforcement learning. Background Art

[0002] Industrial robots are generally industrial robotic arms, which are composed of sensors, mechanical structures, electrical control systems and other parts. They can perform operations such as handling, assembly, and welding. At present, most industrial robots are single-arm systems, and the tasks they can perform are relatively simple. As production tasks become more and more complex and the application fields become larger and larger, in actual applications, single-arm robots are affected by factors such as their own mechanical structure limitations or unstable environments, and there are increasingly obvious limitations in control.

[0003] In contrast, a dual-arm robot has two mechanical arms that can be coordinated and work together to complete more complex tasks. A dual-arm robot is composed of two mechanical arms that are independent of each other but also dependent on each other, and the control algorithm is more complex.

[0004] Existing dual-arm robot control strategies mainly include master-slave control and position / force hybrid control. In the master-slave control strategy, the master arm performs trajectory planning, and the slave arm plans its own movements accordingly. The position / force hybrid control strategy improves the response speed and accuracy of the system by decomposing the position and force control in the task space. However, these traditional control strategies are usually built on precise mathematical models, which limits their adaptability and flexibility, especially when the task requirements or environment change.

[0005] In order to solve the above technical problems, deep reinforcement learning provides a solution. However, the existing reinforcement learning algorithms have numerous and sensitive hyperparameters, and the model performance of different training batches may vary significantly, affecting the stability of the algorithm. The dual-arm robot requires information interaction between the two arms, and the traditional reinforcement learning algorithm cannot meet this requirement. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a dual-arm robot collaborative control method based on multi-agent reinforcement learning. The present invention effectively improves the success rate of the dual-arm robot collaborative movement through the multi-agent reinforcement learning model MADARC.

[0007] The technical solution of the present invention is: a dual-arm robot collaborative control method based on multi-agent reinforcement learning, comprising the following steps:

[0008] S1), transforming the system control of the dual-arm robot into a Markov decision problem;

[0009] S2) Establish a dual-arm robot simulation environment and design rounds and tasks based on reinforcement learning problems;

[0010] S3) Build a multi-agent reinforcement learning model MADARC, and train the simulation model through the multi-agent reinforcement learning model MADARC.

[0011] Preferably, in step S1), the system control of the dual-arm robot is converted into a Markov decision problem, comprising the following steps:

[0012] S11), state space design, select the state observation information Observation as the position information related to the end fixture of the robot arm and the target object;

[0013] S12), action space design, the action is represented by a vector containing four elements a = [a1, a2, a3, a4];

[0014] S13) Reward function design: Set the reward function Reward according to the scene task and environment:

[0015]

[0016] Where d is the spatial distance between the end fixture and the object;

[0017] S14), round design, according to the simulation task, design three round end situations, after each round ends, give a new object position.

[0018] Preferably, in step S11), the position information includes the three-dimensional spatial coordinates of the end fixture of the robot arm and the target object, the relative coordinates of the fixture and the object, the spatial posture of the object, and the opening and closing state of the fixture.

[0019] Preferably, in step S12), each action element is controlled within the range of [-0.5, 0.5], and the action elements a1, a2, a3 represent the displacement increments Δx, Δy, Δz of the fixture on the X, Y, and Z axes, respectively; the action element a4 represents the movement distance of a single finger in the two-finger fixture, and the direction is represented by positive and negative.

[0020] Preferably, in step S14), the three rounds are:

[0021] In the first round, when the spatial distance d between the end fixture and the object is less than 0.002m, the object is successfully grasped, the task is completed, and the round ends;

[0022] In the second round, when the number of action steps of the robot arm exceeds the maximum number of steps of the task, the round ends;

[0023] In the third round, if a robot arm collision occurs during the task, the round ends immediately.

[0024] Preferably, in step S2), the establishment of the dual-arm robot simulation environment specifically includes the following steps:

[0025] S21), establishing a three-dimensional simulation environment of a dual-arm robot based on the Pytorch framework, creating a three-dimensional simulation robot arm description file in PyBullet, importing the dual-arm robot model file, and constructing a static model of the robot by setting the corresponding assembly relationship of each assembly;

[0026] S22), based on the static model of the robot, add the model file corresponding to the model of the object to be grasped, and set the collision relationship between the fixture and the object;

[0027] S23), setting the working space of the two robotic arms.

[0028] Preferably, in step S2), when the task is completed or failed, the state, action, reward, and completion status of the task are stored in the experience pool Replay Buffer in the form of a tuple (s, a, r, s', done); wherein s is the current state of the agent, a is the current action, r is the reward, s' is the next state, and done is the task completion status of the current round; and before the start of each round, the position of the object is randomly initialized.

[0029] Preferably, in step S3), the multi-agent reinforcement learning model MADARC includes two agents, and the two robotic arms are trained as two independent agents respectively, and each agent includes two pairs of Actor-Critic networks and two pairs of target networks Target-Actor-Critic, and an optimizer. The cumulative reward is calculated by weighting the action value Q of the two pairs of Actor-Critic networks, which can effectively improve the accuracy of the action value. When selecting an action, the two pairs of target networks Target-Actor-Critic will output an action, and the action will first be evaluated by the Actor-Critic network, and then the action with a higher score will be selected for execution, which effectively alleviates the problem of the action being in the local optimum during exploration.

[0030] Preferably, in step S3), the two pairs of Actor-Critic networks are Critic1 network and Actor1 network, and Critic2 network and Actor2 network respectively;

[0031] The two pairs of target networks Target-Actor-Critic are Target1-critic network and Target1-actor network, Target2-critic network and Target2-actor network respectively.

[0032] Preferably, in step S3), the multi-agent reinforcement learning model MADARC is used to train the simulation model, which specifically includes the following steps:

[0033] S31), initialize the multi-agent reinforcement learning model MADARC, and initialize the hyperparameters of two pairs of Actor-Critic networks of two agents and the target network Target-Actor-Critic;

[0034] S32), Initialize the experience pool Replay Buffer;

[0035] S33), initializing the dual-arm robot simulation environment and the object g;

[0036] S34), the actions a1 and a2 output by the Actor1 network and the Actor2 network are evaluated by the Critic1 network and the Critic2 network to obtain the values ​​Q1 and Q2 of the two actions, and the values ​​Q1 and Q2 are compared, and the action with the larger value is taken as the final output, that is:

[0037]

[0038] S35), adding random noise ψ to the action finally output in step S34) and inputting it into the dual-arm robot model to complete the action;

[0039] S36), after the dual-arm robot completes the action, it calculates the reward r and stores the sample (s, a, r, s′, done) of this round in the experience pool Replay Buffer;

[0040] S37), sampling another set of target blocks g′ in the experience pool Replay Buffer, for each agent, calculating the reward r′=r(s,a,g′) corresponding to each target block g′, and storing the replaced sample (s,a,r′,s′,done) in the experience pool, where r′ is the reward after replacing the block;

[0041] S38), for each agent, use the Target Actor network to calculate the action a′=μ′(s′|θ) in state s′ a′) and add random noise ψ to action a′; where μ′ and θa′ are the strategy and Target Actor network parameters corresponding to state s′ respectively;

[0042] S39) Calculate the target Q value of Critic1 network and Critic2 network, and calculate the cumulative reward y of the current state by weighted summation:

[0043] y=r+γ*(τ*min i＝1,2 Q i (s′,a′)+(1-τ)*max i＝1,2 Q i (s′,a′))

[0044] In the formula, r represents the reward corresponding to the target object g, γ is the discount factor, τ represents the weighted summation coefficient, and Q i Indicates the value of Critic1 network or Critic2 network;

[0045] S40), using the gradient descent algorithm to calculate L and regularize it to further prevent the problem of underestimation, that is:

[0046]

[0047] In the formula, is the loss function corresponding to the Critic i network; is the regularization weight, Q i represents the value of the Critic i network; Q i′ Relative to Q i The value of another Critic network, that is, when i=1, i′=2; when i=2, i′=1;

[0048] Then use the optimizer to adjust the parameters θ of the Critic1 network or the Critic2 network Q1 ,θ Q2 Make updates;

[0049] S41) Use Actor1 network and Actor2 network to calculate the new action a under state s new1 =μ(s|θ 1 ), a new2 =μ(s|θ 2 ) ; and evaluate the new action a through the Critic1 network and the Critic2 network new1 and a new2 ,Right now:

[0050] q newi =-Q i (s,a newi ),i=1,2

[0051] In the formula, q newi Indicates the action a obtained by the Actor i network in state s newi Ratings;

[0052] The obtained q newi As the loss function of Actor i network, the network parameters θ of Actor 1 network and Actor 2 network are adjusted by the optimizer. a1 ,θ a2 Make updates;

[0053] S42) adopts the soft update method to perform weighted average of the old target network parameters and the new network parameters and assign them to the Target Critic network and the Target Actor network for updating, thereby completing the training of the dual-arm robot in this round, namely:

[0054]

[0055] In the formula, represents the network parameters of the Target Critic network; σ represents the soft update parameter; Represents the network parameters before the Critic network is updated, Represents the network parameters of the Target1-actor network; Represents the network parameters before the Actor network is updated.

[0056] The beneficial effects of the present invention are:

[0057] 1. The problem of information interaction between the two robotic arms of the present invention is that the two robotic arms are regarded as independent intelligent agents for multi-agent reinforcement learning algorithm training, which is more in line with their actual movement conditions and can effectively improve the success rate of collaborative tasks;

[0058] 2. When selecting an action, the present invention first scores the action through the Critic network, then compares the scores of the two actions, and selects the action with the higher score to execute, which effectively alleviates the problem of the action being in the local optimum during the exploration period; solves the problem of the action being in the local optimum during the training process and the problem of overestimation or underestimation when the Critic network judges the action value, and improves the stability and accuracy of the dual-arm robot collaborative control task;

[0059] 3. The present invention calculates the cumulative reward by weighting the Q values ​​of the two Critic networks, which can effectively prevent over-estimation and underestimation problems; the reward function adopts a sparse reward form to reduce the complexity of the reward function design, simplify the learning problem, and reduce the design cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a schematic diagram of the process of the present invention;

[0061] Figure 2 This is a structural framework diagram of the multi-agent reinforcement learning model MADARC of the present invention;

[0062] Figure 3 A schematic diagram of the training process of the simulated dual-arm robot collaborative control system of the present invention;

[0063] Figure 4 This is a curve chart showing the results of the dual-arm robot collaborative control task training of the present invention. DETAILED DESCRIPTION

[0064] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings:

[0065] like Figure 1 As shown, the embodiment provides a dual-arm robot collaborative control method based on multi-agent reinforcement learning, comprising the following steps:

[0066] S1), transforming the system control of the dual-arm robot into a Markov decision problem;

[0067] The specific steps include:

[0068] S11), state space design, select the state observation information Observation as the position information related to the end fixture of the robot arm and the target object;

[0069] The position information includes the three-dimensional spatial coordinates of the end fixture of the robot arm and the target object, the relative coordinates of the fixture and the object, the spatial posture of the object, and the opening and closing state of the fixture.

[0070] S12), motion space design, the action is represented by a vector containing four elements a=[a1, a2, a3, a4]; each action element is controlled in the range of [-0.5, 0.5], and the action elements a1, a2, a3 represent the displacement increments Δx, Δy, Δz of the fixture on the X, Y, and Z axes respectively; the action element a4 represents the moving distance of a single finger in the two-finger fixture, and the direction is represented by positive and negative.

[0071] S13), reward function design, according to the simulation task, define the spatial distance between the end fixture and the object as d, and set the reward function Reward according to the scene task and environment:

[0072]

[0073] Where d is the spatial distance between the end fixture and the object;

[0074] S14), round design, according to the simulation task, design three round end situations, after each round ends, give a new object position. The three rounds are:

[0075] In the first round, when the spatial distance d between the end fixture and the object is less than 0.002m, the object is successfully grasped, the task is completed, and the round ends;

[0076] In the second round, in order to prevent the task from running endlessly, the maximum number of steps of the task is preset in this embodiment. When the number of action steps of the robot arm exceeds the maximum number of steps of the task, the round ends;

[0077] In the third round, if a robot arm collision occurs during the task, the round ends immediately.

[0078] S2) Establish a dual-arm robot simulation environment and design rounds and tasks based on reinforcement learning problems; specifically, the following steps are included:

[0079] S21), based on the Pytorch framework, establish a three-dimensional simulation environment of the dual-arm robot, create a three-dimensional simulation robot arm description file in PyBullet, import the dual-arm robot model file of model UR5, and build a static model of the robot by setting the corresponding assembly relationship of each assembly;

[0080] S22), based on the static model of the robot, add the model file corresponding to the model of the object to be grasped, and set the collision relationship between the fixture and the object;

[0081] S23), setting the working space of the two robotic arms.

[0082] This example controls a dual-arm robot to collaboratively grab a regular target object to a specified position. The target object is a rectangular block with a length of 40 cm and a width and height of 3 cm. The task is to control the UR5 robot end gripper to move to the target object grabbing position, close the gripper, grab the target object, and finally move the object to the target position. The target position is randomly generated in the workspace according to the initial position of the object.

[0083] When the task is completed or failed, the state, action, reward, and completion status of the task are stored in the experience pool Replay Buffer in the form of a tuple (s, a, r, s', done); where s is the current state of the agent, a is the current action, r is the reward, s' is the next state, and done is the task completion status of the current round; and before the start of each round, the position of the object is randomly initialized.

[0084] S3) Build a multi-agent reinforcement learning model MADARC, and train the simulation model through the multi-agent reinforcement learning model MADARC.

[0085] In this embodiment, Figure 2 As shown, the multi-agent reinforcement learning model MADARC includes two agents, and the two robotic arms are trained as two independent agents respectively, and each agent includes two pairs of Actor-Critic networks and two pairs of target networks Target-Actor-Critic, and an optimizer. The cumulative reward is calculated by weighting the action value Q of the two pairs of Actor-Critic networks, which can effectively improve the accuracy of the action value. When selecting an action, the two pairs of target networks Target-Actor-Critic will output an action, and first evaluate the value of the action through the Actor-Critic network, and then select the action with a higher score to execute, which effectively alleviates the problem of the action being in the local optimum during exploration.

[0086] like Figure 2 As shown, the two pairs of Actor-Critic networks are Critic1 network and Actor1 network, Critic2 network and Actor2 network respectively; the two pairs of target networks Target-Actor-Critic are Target1-critic network and Target1-actor network, Target2-critic network and Target2-actor network respectively.

[0087] In this embodiment, the multi-agent reinforcement learning model MADARC is used to train the simulation model, which specifically includes the following steps:

[0088] S31) Initialize the multi-agent reinforcement learning model MADARC and set the corresponding parameters, where the network learning rate is set to 0.001, the discount factor γ is set to 0.98, the sample batch is set to 256, the experience pool Replay Buffer size is set to 1000000, the soft update rate σ is set to 0.95, and the maximum number of round steps is set to 1000.

[0089] Then, the hyperparameters of the two pairs of Actor-Critic networks and the target network Target-Actor-Critic of the two agents are initialized; specifically, they include:

[0090] Initialize the Critic network parameters θ for each agent Q1 ,θ Q2 , Actor network parameters θ a1 ,θ a2 Among them, θ Q1 ,θ Q2The network parameters of Critic1 and Critic2 in each agent, θ a1 ,θ a2 The network parameters of Actor1 network and Actor2 in each agent.

[0091] Initialize the target network parameters θ Q1′ ←←θ Q1 ,θ Q2′ ←θ Q2 ,θ a1′ ←←θ a1 ,θ a2′ ←←θ a2 , where θ Q1′ ,θ Q2′ are the network parameters of Target1-critic and Target2-critic in each agent, θ a1′ ,θ a2′ are the network parameters of Target1-actor and Target2-actor in each agent respectively.

[0092] S32), Initialize the experience pool Replay Buffer;

[0093] S33), initializing the dual-arm robot simulation environment and the object g;

[0094] S34), the actions a1 and a2 output by the Actor1 network and the Actor2 network are evaluated by the Critic1 network and the Critic2 network to obtain the values ​​Q1 and Q2 of the two actions, and the values ​​Q1 and Q2 are compared, and the action with the larger value is taken as the final output, that is:

[0095]

[0096] S35), add random noise ψ to the action finally output in step S34) and input it into the dual-arm robot model to complete the action; in this embodiment, the random noise ψ is the current action value multiplied by the random value ξ, where ξ∈(0, 0.01).

[0097] S36), after the dual-arm robot completes the action, it calculates the reward r and stores the sample (s, a, r, s′, done) of this round in the experience pool Replay Buffer;

[0098] S37), sampling another set of target blocks g′ in the experience pool Replay Buffer, for each agent, calculating the reward r′=r(s,a,g′) corresponding to each target block g′, and storing the replaced sample (s,a,r′,s′,done) in the experience pool, where r′ is the reward after replacing the block;

[0099] S38) For each agent, uniformly sample a batch of 256 samples from the experience pool Replay Buffer for network update, and use the Target Actor network to calculate the action a′=μ′(s′|θ) in state s′ a′ ) and add random noise ψ to action a′; where μ′, θ a′ They are the strategy and Target Actor network parameters corresponding to the state s′ respectively;

[0100] S39) To prevent overestimation or underestimation, calculate the target Q value of the Critic1 network and the Critic2 network, and calculate the cumulative reward y of the current state by weighted summation:

[0101] y=r+γ*(τ*min i＝1,2 Q i (s′,a′)+(1-τ)*max i＝1,2 Q i (s′,a′))

[0102] In the formula, r represents the reward corresponding to the target object g, γ is the discount factor, τ represents the weighted summation coefficient, and Q i Indicates the value of Critic1 network or Critic2 network;

[0103] S40), using the gradient descent algorithm to calculate L and regularize it to further prevent the problem of underestimation, that is:

[0104]

[0105] In the formula, is the loss function corresponding to the Critic i network; is the regularization weight, Q i represents the value of the Critic i network; Q i′ Relative to Q i The value of another Critic network, that is, when i=1, i′=2; when i=2, i′=1;

[0106] Then use the optimizer to adjust the parameters θ of the Critic1 network or the Critic2 network Q1 ,θ Q2 Make updates;

[0107] S41) Use Actor1 network and Actor2 network to calculate the new action a under state s new1 =μ(s|θ 1 ), a new2 =μ(s|θ 2 ) ; and evaluate the new action a through the Critic1 network and the Critic2 network new1 and a new2 ,Right now:

[0108] q newi =-Q i (s,a newi ),i=1,2

[0109] In the formula, q newi Indicates the action a obtained by the Actor i network in state s newi Ratings;

[0110] The obtained q newi As the loss function of Actor i network, the network parameters θ of Actor 1 network and Actor 2 network are adjusted by the optimizer. a1 ,θ a2 Make updates;

[0111] S42) adopts the soft update method to perform weighted average of the old target network parameters and the new network parameters and assign them to the Target Critic network and the Target Actor network for updating, thereby completing the training of the dual-arm robot in this round, namely:

[0112]

[0113] In the formula, represents the network parameters of the Target Critic network; σ represents the soft update parameter; Represents the network parameters before the Critic network is updated, Represents the network parameters of the Target1-actor network; Represents the network parameters before the Actor network is updated.

[0114] The interaction process between this embodiment and the dual-arm robot in the Pybullet simulation environment is as follows: Figure 3 As shown in Figure 1, the MADARC algorithm and the DDPG algorithm are applied to the dual-arm robot collaborative control task at the same time. The maximum number of training steps per round is set to 1000, and 150 rounds of training are performed. The test success rate is output, and the algorithm performance is evaluated based on the success rate curve. The training curve obtained after the training is as follows Figure 4As shown, it can be seen from the training curve that the success rate and stability of the task are significantly improved, verifying the feasibility of this embodiment.

[0115] When selecting an action, this embodiment first scores the action through the Critic network, then compares the scores of the two actions, and selects the action with the higher score to execute, which effectively alleviates the problem of the action being in the local optimum during the exploration period; at the same time, by weighting the Q value of the two Critic networks to calculate the cumulative reward, it can effectively prevent the problem of overestimation and underestimation. In addition, this embodiment takes into account the problem of information interaction between the two mechanical arms in the actual work of the dual-arm robot. It is more in line with its actual movement situation to regard the two as an independent intelligent agent to train the multi-agent reinforcement learning algorithm, which can effectively improve the success rate of collaborative tasks.

[0116] The above embodiments and descriptions are only for illustrating the principles and best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, all of which fall within the scope of the present invention to be protected.

Claims

1. A dual-arm robot collaborative control method based on multi-agent reinforcement learning, characterized in that: The steps include: S1), transforming the system control of the dual-arm robot into a Markov decision problem; S2) Establish a dual-arm robot simulation environment and design rounds and tasks based on reinforcement learning problems; S3) Build a multi-agent reinforcement learning model MADARC, and train the simulation model through the multi-agent reinforcement learning model MADARC.

2. The method for collaborative control of a dual-arm robot based on multi-agent reinforcement learning according to claim 1, characterized in that: In step S1), the system control of the dual-arm robot is converted into a Markov decision problem, including the following steps: S11), state space design, select the state observation information Observation as the position information related to the end fixture of the robot arm and the target object; S12), action space design, the action is represented by a vector containing four elements a = [a1, a2, a3, a4]; S13) Reward function design: Set the reward function Reward according to the scene task and environment: Where d is the spatial distance between the end fixture and the object; S14), round design, according to the simulation task, design three round end situations, after each round ends, give a new object position.

3. The method for collaborative control of a dual-arm robot based on multi-agent reinforcement learning according to claim 2, characterized in that: In step S14), the three rounds are: In the first round, when the spatial distance d between the end fixture and the object is less than 0.002m, the object is successfully grasped, the task is completed, and the round ends; In the second round, when the number of action steps of the robot arm exceeds the maximum number of steps of the task, the round ends; In the third round, if a robot arm collision occurs during the task, the round ends immediately.

4. The method for collaborative control of a dual-arm robot based on multi-agent reinforcement learning according to claim 1, characterized in that: In step S2), the establishment of the dual-arm robot simulation environment specifically includes the following steps: S21), establishing a three-dimensional simulation environment of a dual-arm robot based on the Pytorch framework, creating a three-dimensional simulation robot arm description file in PyBullet, importing the dual-arm robot model file, and constructing a static model of the robot by setting the corresponding assembly relationship of each assembly; S22), based on the static model of the robot, add the model file corresponding to the model of the object to be grasped, and set the collision relationship between the fixture and the object; S23), setting the working space of the two robotic arms.

5. The method for collaborative control of a dual-arm robot based on multi-agent reinforcement learning according to claim 4, characterized in that: In step S2), when the task is completed or failed, the state, action, reward, and completion status of the task are stored in the experience pool Replay Buffer in the form of a tuple (s, a, r, s', done); where s is the current state of the agent, a is the current action, r is the reward, s' is the next state, and done is the task completion status of the current round; and before the start of each round, the position of the object is randomly initialized.

6. The method for collaborative control of a dual-arm robot based on multi-agent reinforcement learning according to claim 1, characterized in that: In step S3), the multi-agent reinforcement learning model MADARC includes two agents, and the two robotic arms are trained as two independent agents respectively, and each agent includes two pairs of Actor-Critic networks and two pairs of target networks Target-Actor-Critic, and an optimizer. The cumulative reward is calculated by weighting the action value Q of the two pairs of Actor-Critic networks, and when selecting an action, the two pairs of target networks Target-Actor-Critic will output an action, and the action will first be evaluated by the Actor-Critic network, and then the action with a higher score will be selected for execution.

7. The method for collaborative control of a dual-arm robot based on multi-agent reinforcement learning according to claim 6, characterized in that: In step S3), the two pairs of Actor-Critic networks are respectively Critic1 network and Actor1 network, and Critic2 network and Actor2 network; The two pairs of target networks Target-Actor-Critic are Target1-critic network and Target1-actor network, Target2-critic network and Target2-actor network respectively.

8. The method for collaborative control of a dual-arm robot based on multi-agent reinforcement learning according to claim 7, characterized in that: In step S3), the simulation model is trained using the multi-agent reinforcement learning model MADARC, which specifically includes the following steps: S31), initialize the multi-agent reinforcement learning model MADARC, and initialize the hyperparameters of two pairs of Actor-Critic networks of two agents and the target network Target-Actor-Critic; S32), Initialize the experience pool Replay Buffer; S33), initialize the dual-arm robot simulation environment and object g.

9. A dual-arm robot collaborative control method based on multi-agent reinforcement learning according to claim 8, characterized in that: The simulation model is trained using the multi-agent reinforcement learning model MADARC, which also includes the following steps: S34), the actions a1 and a2 output by the Actor1 network and the Actor2 network are evaluated by the Critic1 network and the Critic2 network to obtain the values ​​Q1 and Q2 of the two actions, and the values ​​Q1 and Q2 are compared, and the action with the larger value is taken as the final output, that is: S35), adding random noise ψ to the action finally output in step S34) and inputting it into the dual-arm robot model to complete the action; S36), after the dual-arm robot completes the action, it calculates the reward r and stores the sample (s, a, r, s′, done) of this round in the experience pool Replay Buffer; S37), sampling another set of target blocks g′ in the experience pool Replay Buffer, for each agent, calculating the reward r′=r(s,a,g′) corresponding to each target block g′, and storing the replaced sample (s,a,r′,s′,done) in the experience pool, where r′ is the reward after replacing the block; S38), for each agent, use the Target Actor network to calculate the action a′=μ′(s′|θ) in state s′ a′ ) and add random noise ψ to action a′; where μ′, θ a′ They are the strategy and Target Actor network parameters corresponding to the state s′ respectively; S39) Calculate the target Q value of Critic1 network and Critic2 network, and calculate the cumulative reward y of the current state by weighted summation: y=r+γ*(τ*min i＝1,2 Q i (s′,a′)+(1-τ)*max i＝1,2 Q i (s′,a′)) In the formula, r represents the reward corresponding to the target block g, γ is the discount factor, τ represents the weighted summation coefficient, and Q i Indicates the value of Critic1 network or Critic2 network.

10. A dual-arm robot collaborative control method based on multi-agent reinforcement learning according to claim 9, characterized in that: The simulation model is trained using the multi-agent reinforcement learning model MADARC, which also includes the following steps: S40), using the gradient descent algorithm to calculate L and regularize it to further prevent the problem of underestimation, that is: In the formula, is the loss function corresponding to the Critic i network; is the regularization weight, Q i represents the value of the Critic i network; Q i′ Relative to Q i The value of another Critic network, that is, when i=1, i′=2; when i=2, i′=1; Then use the optimizer to adjust the parameters θ of the Critic1 network or the Critic2 network Q1 ,θ Q2 Make updates; S41) Use Actor1 network and Actor2 network to calculate the new action a under state s new1 =μ(s|θ 1 ), a new2 =μ(s|θ 2 ) ; and evaluate the new action a through the Critic1 network and the Critic2 network new1 and a new2 ,Right now: q newi =-Q i (s,a newi ),i=1,2 In the formula, q newi Indicates the action a obtained by the Actor i network in state s newi Ratings; The obtained q newi As the loss function of Actor i network, the network parameters θ of Actor 1 network and Actor 2 network are adjusted by the optimizer. a1 ,θ a2 Make updates; S42) adopts the soft update method to perform weighted average of the old target network parameters and the new network parameters and assign them to the Target Critic network and the Target Actor network for updating, thereby completing the training of the dual-arm robot in this round, namely: In the formula, represents the network parameters of the Target Critic network; σ represents the soft update parameter; Represents the network parameters before the Critic network is updated, Represents the network parameters of the Target1-actor network; Represents the network parameters before the Actor network is updated.

Images