Multi-mechanical-arm deep reinforcement learning control method and device for sorting regenerated articles

Abstract

The invention discloses a multi-mechanical-arm deep reinforcement learning control method and device for sorting regenerated articles. The method comprises the steps that state information of a target object and a multi-mechanical-arm system is obtained; the method comprises the following steps: constructing a depth deterministic policy gradient (DDPG) algorithm network architecture, and setting an experience playback pool and an agent reward function; and deploying to a real sorting environment for motion guidance. According to the method, in a DDPG algorithm strategy network architecture, a recurrent neural network (RNN) is used as a trunk feature extraction network, so that the long-term dependency relationship of input data is captured, and the grabbing strategy of a mechanical arm is better guided; a one-to-many discrimination architecture is introduced into an evaluation network architecture to adapt to training of a multi-mechanical-arm system; a composite reward function and a double-pool experience playback mechanism are provided, more timely and richer feedback signals are provided for the mechanical arms, data correlation is reduced, the sample utilization rate is increased, and the success rate and efficiency of multi-mechanical-arm regenerated article sorting are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-robot system collaborative control, and in particular to a multi-robot recycled object sorting control method and device based on a deep deterministic policy gradient algorithm. Background Art

[0002] With the growth of population and the acceleration of urbanization, garbage classification and treatment has become an important issue in environmental protection and resource recycling. The traditional method of garbage classification mainly relies on manual sorting, but this method is inefficient and costly, and it is difficult to meet the needs of large-scale garbage disposal. With the advancement of science and technology, mechanical sorting, air-blowing sorting and optical sorting technologies have been widely used, which significantly improves the efficiency and accuracy of classification, and provides solid technical support for achieving the social goal of green environmental protection.

[0003] In the field of robotic arm sorting, precise control directly affects the reliability and stability of the entire sorting system. Traditional control methods such as proportional integral derivative (PID) control and calculated torque control often rely on precise system models and parameters, and are difficult to cope with complex system characteristics such as nonlinearity and coupling. In recent years, with the continuous development of deep learning technology, the Deep Deterministic Policy Gradient (DDPG) algorithm, as a deep reinforcement learning method that combines deep neural networks and deterministic policy gradient optimization, has the advantages of stable training and fast convergence in continuous action space, and is widely used in the field of robotic arm control.

[0004] Although the DDPG algorithm has made significant progress in the precise control and task execution of a single robot, when it comes to the collaborative operation of multiple robots, especially in complex and dynamic tasks such as recycled object sorting, the DDPG algorithm is relatively less used and faces a series of challenges. First, the multi-robot system involves not only the independent control strategy of a single arm, but also the interaction and coordination mechanism between them. Secondly, although the DDPG algorithm has advantages in dealing with continuous action space problems, it also has some inherent disadvantages. For example, a large amount of data is often required to explore the environment during training, and the correlation of this data is usually high, resulting in low sample utilization. This inefficient data utilization not only prolongs the training time, but also may make it difficult for the algorithm to converge to the optimal strategy. Summary of the invention

[0005] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a multi-robot arm recycled object sorting control method based on an improved DDPG algorithm to solve the problems of sparse rewards, low sampling efficiency of the experience pool, long-term dependencies in the state space, and multi-robot collaboration conflicts existing in the traditional DDPG algorithm in the multi-robot arm recycled object sorting task.

[0006] A first aspect of the present invention provides a multi-manipulator deep reinforcement learning control method for sorting recycled items, comprising the following steps:

[0007] Step 1, target determination: determine the target object that the robot arm needs to grasp, such as a plastic bottle, and the specific position and posture of the target to be grasped;

[0008] Step 2: Based on the DDPG algorithm, an improved DDPG network structure is constructed to obtain the RNN-MADDPG multi-manipulator recycled object sorting control algorithm;

[0009] Furthermore, the RNN-MADDPG algorithm improvement part is mainly divided into the following three parts, namely, algorithm network architecture design, reward function design, and experience replay pool design;

[0010] Step A: Add sequence data dependency detection and replace the ordinary fully connected layer neural network in the DDPG algorithm strategy network architecture with the RNN recurrent neural network. In the collaborative operation of multiple robotic arms, the action of each arm depends not only on the current environmental state, but also on the previous action and state. RNN can capture and use this time series information to better predict the next optimal action and better guide the grasping strategy of the robotic arm.

[0011] Step B: Introduce a one-to-many discriminant architecture into the DDPG algorithm evaluation network architecture to adapt to the training of multi-robot systems. The one-to-many discriminant architecture allows a single evaluation network to output respective value assessments for different robotic arms. This structure enables the network to learn multiple tasks (here, the control strategy of each robotic arm) at the same time, thereby capturing the commonality and individuality of different robotic arm behaviors in a shared network structure, and solving the problems of collaboration and competition in a multi-agent environment.

[0012] Step C: A composite reward function integrating multiple reward strategies is proposed. This method designs a composite reward function R for a multi-robot recycled item sorting scenario. The reward function consists of three parts, which respectively consider the distance between the end position of the robot arm and the target object, the adaptability of the gripping direction of the robot arm end effector and the posture of the target object, and the obstacle avoidance of the robot arm. This reduces the reward sparsity problem of the DDPG algorithm in the garbage sorting robot arm sorting scenario, provides more timely and richer feedback signals for the robot arm, and reduces the training time of the multi-robot arm system.

[0013] Step D: Improve the experience pool sampling mechanism. Based on the original DDPG algorithm single experience pool, introduce a dual-pool experience playback mechanism to increase the capacity of the experience pool and provide sufficient data support for uniform sampling. At the same time, combined with the original DDPG algorithm experience pool random sampling, this method reduces the correlation between samples, improves sample utilization, and enhances the diversity and stability of the learning process.

[0014] Step 3, virtual simulation training: The recycled object sorting robot first randomly performs grasping actions in the virtual environment, and records the state and perception value of the robot, completes sample collection, and starts parameter training when the experience pool is filled. Combined with the perception data obtained in step 1, the RNN-MADDPG multi-robot recycled object sorting control algorithm in step 2 is trained. In each round of the experiment, there is an interactive process and parameter update at the same time until all network parameters are updated. After training, each network parameter is updated to obtain the maximum expected reward return and the optimal strategy.

[0015] Step 4: Execution of sorting tasks: Use the trained DDPG algorithm to deploy it to the actual multi-robot recycled item sorting task, monitor the execution of the robot arms, and make adjustments and improvements as needed.

[0016] The second aspect of the present invention relates to a multi-robot arm deep reinforcement learning control device for sorting recycled items, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a multi-robot arm deep reinforcement learning control method for sorting recycled items of the present invention.

[0017] A computer-readable storage medium stores a program, which, when executed by a processor, implements a multi-robot deep reinforcement learning control method for recycling object sorting of the present invention.

[0018] The present invention includes: obtaining target object and multi-manipulator system state information; constructing a deep deterministic policy gradient (DDPG) algorithm network architecture, setting an experience replay pool, and an agent reward function; and deploying to a real sorting environment for motion guidance. Among them, the method uses a recurrent neural network (RNN) as a backbone feature extraction network in the DDPG algorithm strategy network architecture to capture the long-term dependency of input data and better guide the grasping strategy of the manipulator; introduces a one-to-many discriminant architecture in the evaluation network architecture to adapt to the training of the multi-manipulator system; proposes a composite reward function and a dual-pool experience replay mechanism to provide more timely and richer feedback signals for the manipulator, reduce data correlation, improve sample utilization, and improve the success rate and efficiency of multi-manipulator recycled object sorting.

[0019] The beneficial effects of the present invention are as follows: by using the RNN recurrent neural network as the backbone feature extraction network of the policy network in the DDPG algorithm, the method can effectively capture the long-term dependencies in the state space when training the robotic arm, and better guide the grasping strategy of the robotic arm; the DDPG algorithm evaluation network is improved, and a one-to-many discrimination architecture is introduced to adapt to the training of multi-robotic arm systems; a composite reward function and a dual-pool experience playback mechanism suitable for multi-robotic arm collaborative control are designed to provide more timely and richer feedback signals for the robotic arm, reduce data correlation, improve sample utilization, and improve the success rate and efficiency of multi-robotic arm recycled object sorting. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is an overall schematic diagram for implementing the method of the present invention;

[0021] Figure 2 This is a schematic diagram of the structure of the deep deterministic policy gradient DDPG algorithm improved by the method of the present invention;

[0022] Figure 3 Schematic diagram of the recurrent neural network RNN ​​structure in the method of the present invention;

[0023] Figure 4 It is a schematic diagram of the improved structure of the experience pool in the method of the present invention. DETAILED DESCRIPTION

[0024] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0025] Example 1

[0026] refer to Figure 1 This embodiment provides a multi-manipulator deep reinforcement learning control method for recycling object sorting, including the following steps:

[0027] Step 1, target determination: In the recycled items sorting task, the sorting target is determined based on the characteristics of the robot arm, such as gripping ability and operating accuracy.

[0028] Specifically, the target objects can be metal cans, glass bottles, etc. These objects are relatively large in length and width, moderate in size, and easy to pick up.

[0029] Step 1-1, determine the image position and posture of the target object: The sampling camera is responsible for continuous image acquisition of the recycled items on the conveyor belt, and transmits the captured high-definition images back to the industrial computer running the advanced visual recognition algorithm through the Ethernet communication standard to identify the object category, determine the position of the target object and the rotation angle in the vertical direction of the plane.

[0030] Step 1-2, determine the position and posture of the target object relative to the robotic arm: the original pixel coordinates of the target object are obtained from the image plane, and the target position in the pixel coordinate system is converted to the robotic arm coordinate system through a series of coordinate transformations, so that the robotic arm can accurately locate and perform grasping actions.

[0031] Step 2: Construction of the multi-robot recycled object sorting control algorithm RNN-MADDPG, refer to Figure 2 : Further, the RNN-MADDPG algorithm is constructed into the following five parts, namely, network architecture, state set, action set, reward function, and experience replay pool;

[0032] Step 2-1, set up the network structure required by the RNN-MADDPG algorithm: including the main network Actor network, Critic network and target network Target Actor network, Target Critic network with the same network structure. Each robot in the robot system has an Actor network for learning grasping strategies, and the entire robot system has only one Critic network for estimating action values.

[0033] The RNN-MADDPG algorithm uses RNN as the backbone feature extraction network of the Actor network. Its logical structure is as follows: Figure 3 Specifically, the previous information is stored and memorized in the hidden layer, and then input into the hidden unit of the current calculation, so that the input of the hidden layer can be composed of multiple parts, the output of the input layer and the output of the hidden layer at the previous moment, that is, the nodes in the hidden layer are self-connected. This mechanism allows information to flow continuously across time steps, which helps to capture dynamic changes and long-term dependencies in sequence data and better guide the grasping strategy of the robot arm.

[0034] The RNN-MADDPG algorithm uses a traditional fully connected neural network as the Critic network. During the training phase, the Critic network can access the information of all the robotic arms in the system, including their states and actions, which allows it to accurately evaluate the expected return of each action. However, during the execution phase, the Actor network of each robotic arm can only make decisions based on its own local observations to solve the collaboration and competition problems of multiple robotic arms.

[0035] Step 2-2, set the state set S of a single robot arm: The state set S of a robot arm can be defined as a vector containing all necessary state information, which is used to describe the dynamic characteristics of the robot arm and the spatial relationship between the robot arm and the target object. Specifically, the state set S of a single robot arm is:

[0036]

[0037] Among them, θ i Represents the angle of each joint, ω i represents the angular velocity of each joint, d x , d y , d z Represents the distance from the end position of the robot arm to the target object in the x, y, and z directions. Represents the gripping direction of the robot arm and the rotation angle error of the target object in the plane relative to the vertical direction

[0038] For a multi-robot system, the state space of the system, i.e., the input data of the Critic network, should contain the states of all the robot arms, i.e., ∑S.

[0039] Step 2-3, setting the action set A of a single robotic arm: The action set A of the robotic arm can be defined as a set containing all possible control inputs, which are used to drive the various joints of the robotic arm to achieve the desired movement.

[0040] Therefore, the action set A can be expressed as:

[0041] {τ1, τ2, τ3, τ4, τ5, τ6}

[0042] Among them, τ i Represents the control torque of each joint.

[0043] Step 2-4, design a composite reward function R that integrates multiple reward strategies: The reward function R consists of three parts. Specifically, the following factors are considered:

[0044] ①r1 is the distance reward function: the distance from the end position of the robot arm to the center of the target object is:

[0045]

[0046] ②r2 is the grasping posture reward function: the fitness of the robot arm’s grasping direction and the target object’s posture is:

[0047]

[0048] ③r3 is the obstacle avoidance reward function: the robot arm may collide with environmental objects during movement, and should avoid:

[0049]

[0050] In the formula, when the robot reaches the target position, Δd is 0, r1 is 10, and r1 is 0 in other cases; when the grasping direction of the robot matches the posture of the target object When the robot arm collides, r3 is -20, and in other cases, r3 is 0. This method reduces the reward sparseness problem of the traditional DDPG algorithm in the multi-robot sorting scenario of recycled objects, provides more timely and richer feedback signals for the robot arm, and reduces the training time of the multi-robot system.

[0051] Step 2-5, improve the experience pool sampling mechanism and introduce a dual-pool experience playback mechanism, such as Figure 4 :The experience data that the agent will get t , a t , r t ,s t+1} is stored in two experience replay pools, s t represents the state of the agent at time t, a t represents the action taken by the agent at time t, r t represents the reward obtained after taking action at time t, s t+1 It represents the state reached at time t+1 after the agent takes action. When updating the network parameters, a combination of random sampling and uniform sampling is used. A new experience replay pool is added to expand the sample data set, providing conditions for uniform sampling. Random sampling retains flexibility. The number of samples is:

[0052] μ=w5*u1+w6*u2

[0053] Among them, w5 and w6 are the ratios of random sampling and uniform sampling to the total number of samples, and u1 and u2 are the number of samples of random sampling and uniform sampling.

[0054] Furthermore, after ten rounds of training, the average Q value of the ten rounds was calculated, and the hill climbing method was used to dynamically adjust the ratio of random sampling to uniform sampling to improve the sampling utilization rate, thereby optimizing the experience replay strategy and improving the accuracy and adaptability of the robotic arm's grasping.

[0055] Step 3, RNN-MADDPG algorithm training process:

[0056] Step 3-1, initialize network parameters, initialize the weights and biases of the Actor strategy network and the Critic evaluation network, and initialize the experience pool.

[0057] Step 3-2: Randomly perform grasping actions to sense the state of the robot arm s t And perception data, input Actor network to obtain action a t , and execute the action, observing the new state s t+1 , while the Critic evaluation network obtains the reward signal r t , the data (s t , a t , r t ,s t+1 ) into the experience pool.

[0058] Step 3-3: After the two experience pools are filled, interactive optimization is started by using random sampling and uniform sampling to increase sample diversity and reduce sample correlation during training.

[0059] Step 3-4, through the above settings, the intelligent agent continuously interacts with the environment, repeatedly executes the process of selecting actions, interacting with the environment, storing experience and training the network, so that the intelligent agent gradually learns the optimal strategy.

[0060] Step 4: Load the trained model in the real world to control and grasp the robotic arm.

[0061] Through the above steps, this garbage sorting robot arm grasping control method based on the improved deep reinforcement learning DDPG algorithm can successfully perform grasping tasks under different environments and object characteristics.

[0062] Example 2

[0063] The present embodiment relates to a multi-robot arm deep reinforcement learning control device for sorting recycled items, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a multi-robot arm deep reinforcement learning control method for sorting recycled items according to embodiment 1.

[0064] Example 3

[0065] This embodiment relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a multi-robot arm deep reinforcement learning control method for sorting recycled items according to embodiment 1 is implemented.

[0066] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A multi-manipulator deep reinforcement learning control method for recycling object sorting, comprising the following steps: Step 1: determine the target object that the robot arm needs to grasp, as well as the specific position and posture of the target to be grasped; Step 2: Based on the DDPG algorithm, an improved DDPG network structure is constructed to obtain the RNN-MADDPG multi-manipulator recycled object sorting control algorithm; Step 3, virtual simulation training: The recycled object sorting robot first randomly performs grasping actions in the virtual environment, and records the state and perception value of the robot to complete sample collection. When the experience pool is filled, parameter training begins; combined with the perception data training obtained in step 1 and the RNN-MADDPG multi-robot recycled object sorting control algorithm in step 2, there is an interactive process and parameter update in each round of experiments until all network parameters are updated; after training, each network parameter is updated to obtain the maximum expected reward return and the optimal strategy; Step 4, perform sorting tasks: Use the trained DDPG algorithm to deploy it to the actual multi-robot recycled item sorting task, monitor the execution of the robot arms, and make adjustments and improvements as needed.

2. A multi-manipulator deep reinforcement learning control method for recycling product sorting as claimed in claim 1, characterized in that: Step 1 specifically includes: Step 1-1, determine the image position and posture of the target object: the sampling camera is responsible for continuous image acquisition of the recycled items on the conveyor belt, and transmits the captured high-definition images back to the industrial computer running the advanced visual recognition algorithm through the Ethernet communication standard to identify the object category, determine the position of the target object and the rotation angle in the vertical direction of the plane; Step 1-2, determine the position and posture of the target object relative to the robotic arm: the original pixel coordinates of the target object are obtained from the image plane, and the target position in the pixel coordinate system is converted to the robotic arm coordinate system through a series of coordinate transformations, so that the robotic arm can accurately locate and perform grasping actions.

3. A multi-manipulator deep reinforcement learning control method for recycling product sorting as claimed in claim 1, characterized in that: Step 2 specifically includes: the RNN-MADDPG algorithm construction includes: network architecture, state set, action set, reward function, experience replay pool; specifically includes: Step 2-1, set up the network structure required by the RNN-MADDPG algorithm: including the main network Actor network, Critic network and target network Target Actor network, Target Critic network with the same network structure; each robotic arm in the robotic arm system has an Actor network for learning grasping strategies, and the entire robotic arm system has only one Critic network for estimating action values; The RNN-MADDPG algorithm uses RNN as the backbone feature extraction network of the Actor network. The network includes: storing and memorizing previous information in the hidden layer, and then inputting it into the hidden unit of the current calculation, so that the input of the hidden layer can be composed of multiple parts, the output of the input layer and the output of the hidden layer at the previous moment, that is, the nodes in the hidden layer are self-connected. This mechanism allows information to flow continuously across time steps, which helps to capture dynamic changes and long-term dependencies in sequence data, and better guide the grasping strategy of the robot arm; The RNN-MADDPG algorithm uses a traditional fully connected neural network as the Critic network. During the training phase, the Critic network can access the information of all the robotic arms in the system, including their states and actions, which allows it to accurately evaluate the expected return of each action. However, during the execution phase, the Actor network of each robotic arm can only make decisions based on its own local observations to solve the collaboration and competition problems of multiple robotic arms. Step 2-2, set the state set S of a single robotic arm: The state set S of the robotic arm can be defined as a vector containing all necessary state information, which is used to describe the dynamic characteristics of the robotic arm and the spatial relationship between the robotic arm and the target object; specifically, the state set S of a single robotic arm is: Among them, θ i Represents the angle of each joint, ω i represents the angular velocity of each joint, d x , d y , d z Represents the distance from the end position of the robot arm to the target object in the x, y, and z directions. Represents the gripping direction of the robot arm and the rotation angle error of the target object in the plane relative to the vertical direction For a multi-manipulator system, the state space of the system, i.e., the input data of the Critic network, should contain the states of all the manipulators, i.e., ∑S; Step 2-3, setting the action set A of a single robot arm: the action set A of the robot arm can be defined as a set of all possible control inputs, which are used to drive the joints of the robot arm to achieve the desired movement; Therefore, the action set A can be expressed as: {τ1, τ2, τ3, τ4, τ5, τ6} Among them, τ i Represents the control torque of each joint; Step 2-4, design a composite reward function R that integrates multiple reward strategies: The reward function R consists of three parts. Specifically, the following factors are considered: ①r1 is the distance reward function: the distance from the end position of the robot arm to the center of the target object is: ②r2 is the grasping posture reward function: the fitness of the robot arm’s grasping direction and the target object’s posture is: ③r3 is the obstacle avoidance reward function: the robot arm may collide with environmental objects during movement, and should avoid: In the formula, when the robot reaches the target position, Δd is 0, r1 is 10, and r1 is 0 in other cases; when the grasping direction of the robot matches the posture of the target object is 0, then r2 is 10, and r2 is 0 in other cases; when the robot arm collides, r3 is -20, and r3 is 0 in other cases; this method reduces the reward sparseness problem of the traditional DDPG algorithm in the multi-robot arm sorting scenario of recycled objects, provides more timely and richer feedback signals for the robot arm, and reduces the training time of the multi-robot arm system; Step 2-5, improve the experience pool sampling mechanism and introduce a dual-pool experience playback mechanism: the agent will obtain the experience data {s t , a t , r t ,s t+1 } is stored in two experience replay pools, s t represents the state of the agent at time t, a t represents the action taken by the agent at time t, r t represents the reward obtained after taking action at time t, s t+1 It represents the state reached at time t+1 after the agent takes action; when updating the network parameters, a combination of random sampling and uniform sampling is used. A new experience replay pool is added to expand the sample data set, which provides conditions for uniform sampling. Random sampling retains flexibility. The number of samples is: μ=w5*u1+w6*u2 Among them, w5 and w6 are the ratios of random sampling and uniform sampling to the total number of samples, and u1 and u2 are the number of samples of random sampling and uniform sampling.

4. A multi-manipulator deep reinforcement learning control method for recycling product sorting as claimed in claim 1, characterized in that: In step 2.5, after ten rounds of training, the average Q value of ten rounds is calculated, and the hill climbing method is used to dynamically adjust the ratio of random sampling to uniform sampling.

5. A multi-manipulator deep reinforcement learning control method for recycling product sorting as claimed in claim 1, characterized in that: The RNN-MADDPG algorithm training process in step 3 is as follows: Step 3-1, initialize network parameters, initialize the weights and biases of the Actor strategy network and the Critic evaluation network, and initialize the experience pool; Step 3-2: Randomly perform grasping actions to sense the state of the robot arm s t And perception data, input into the Actor network to obtain action a t , and execute the action, observing the new state s t+1 , while the Critic evaluation network obtains the reward signal r t , the data (s t , a t , r t ,s t+1 ) into the experience pool; Step 3-3: After the two experience pools are filled, interactive optimization is started by using random sampling and uniform sampling to increase sample diversity and reduce sample correlation during training; Step 3-4, through the above settings, the intelligent agent continuously interacts with the environment, repeatedly executes the process of selecting actions, interacting with the environment, storing experience and training the network, so that the intelligent agent gradually learns the optimal strategy.

6. A multi-manipulator deep reinforcement learning control device for sorting recycled items, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement a multi-robot arm deep reinforcement learning control method for sorting recycled items as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, a multi-robot arm deep reinforcement learning control method for sorting recycled items as described in any one of claims 1-5 is implemented.

Images