A multi-robot deep reinforcement learning control method and device for sorting recycled items

Abstract

A deep reinforcement learning control method and device for multi-manipulator sorting of recycled items, comprising: acquiring target object and multi-manipulator system state information; constructing a Deep Deterministic Policy Gradient (DDPG) algorithm network architecture, setting an experience replay pool, and an agent reward function; and deploying it in a real sorting environment for motion guidance. Specifically, this method uses a recurrent neural network (RNN) as the backbone feature extraction network within the DDPG algorithm policy network architecture to capture long-term dependencies in input data and better guide the robot's grasping strategy; introduces a one-to-many discriminant architecture into the evaluation network architecture to accommodate multi-manipulator system training; and proposes a composite reward function and dual-pool experience replay mechanism to provide the robot with more timely and richer feedback signals, reducing data correlation, improving sample utilization, and enhancing the success rate and efficiency of multi-manipulator recycled item sorting.

Description

Technical Field

[0001] The present invention relates to the technical field of collaborative control of multi-manipulator systems, and in particular to a multi-manipulator recycled item sorting control method and device based on a deep deterministic policy gradient algorithm. Background Art

[0002] With population growth and accelerated urbanization, waste sorting and disposal has become a crucial issue for environmental protection and resource recycling. Traditional waste sorting relies primarily on manual sorting, but this method is inefficient and costly, making it difficult to meet the demands of large-scale waste disposal. With technological advancements, technologies such as mechanical sorting, air-blowing sorting, and optical sorting have been widely adopted, significantly improving sorting efficiency and accuracy and providing solid technical support for achieving green and environmentally friendly social goals.

[0003] In the field of robotic arm sorting, precise control directly affects the reliability and stability of the entire sorting system. Traditional control methods such as Proportional Integral Derivative (PID) control and calculated torque control often rely on precise system models and parameters, and have difficulty coping with complex system characteristics such as nonlinearity and coupling. In recent years, with the continuous development of deep learning technology, the Deep Deterministic Policy Gradient (DDPG) algorithm, a deep reinforcement learning method that combines deep neural networks and deterministic policy gradient optimization, has the advantages of stable training and fast convergence in continuous action space, and has been widely used in the field of robotic arm control.

[0004] Although the DDPG algorithm has made significant progress in the precise control and task execution of single robotic arms, its application is relatively limited and faces a series of challenges when it comes to the collaborative operation of multiple robotic arms, especially in complex and dynamic tasks such as recycled object sorting. First, multi-robotic arm systems involve not only independent control strategies for each arm, but also the interaction and coordination mechanisms between them. Second, although the DDPG algorithm has advantages in handling continuous action space problems, it also has some inherent shortcomings. For example, during training, a large amount of data is often required to explore the environment, and this data is often highly correlated, resulting in low sample utilization. This inefficient data utilization not only prolongs training time but also makes it difficult for the algorithm to converge to the optimal strategy. Summary of the Invention

[0005] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a multi-robot arm recycled item sorting control method based on an improved DDPG algorithm to solve the problems of sparse rewards, low sampling efficiency of the experience pool, long-term dependencies in the state space, and multi-robot collaboration conflicts existing in the traditional DDPG algorithm in the multi-robot arm recycled item sorting task.

[0006] A first aspect of the present invention provides a multi-manipulator deep reinforcement learning control method for sorting recycled items, comprising the following steps:

[0007] Step 1, target determination: Determine the target object that the robot arm needs to grasp, such as a plastic bottle, as well as the specific position and posture of the target to be grasped;

[0008] Step 2: Based on the DDPG algorithm, an improved DDPG network structure is constructed to obtain the RNN-MADDPG multi-manipulator recycled item sorting control algorithm;

[0009] Furthermore, the RNN-MADDPG algorithm improvement is mainly divided into the following three parts: algorithm network architecture design, reward function design, and experience replay pool design;

[0010] Step A: Add sequence data dependency detection and replace the standard fully connected neural network layer in the DDPG algorithm's strategy network architecture with an RNN recurrent neural network. In multi-manipulator collaborative operations, each arm's action depends not only on the current state of the environment but also on previous actions and states. RNNs can capture and utilize this time series information to better predict the optimal next action and guide the robot's grasping strategy.

[0011] Step B: Introduce a one-to-many discriminant architecture into the DDPG algorithm evaluation network architecture to accommodate the training of multi-manipulator systems. This one-to-many discriminant architecture allows a single evaluation network to output separate value assessments for different manipulators. This structure enables the network to simultaneously learn multiple tasks (in this case, the control strategies for each manipulator), thereby capturing the commonalities and individualities of different manipulator behaviors within a shared network structure, addressing the challenges of collaboration and competition in multi-agent environments.

[0012] Step C: A composite reward function integrating multiple reward strategies is proposed. This method designs a composite reward function R for a multi-manipulator recycled item sorting scenario. The reward function consists of three parts, which respectively consider the distance between the end position of the manipulator and the target object, the adaptability of the gripping direction of the manipulator end effector and the posture of the target object, and the obstacle avoidance of the manipulator. This reduces the reward sparsity problem of the DDPG algorithm in the garbage sorting robot arm sorting scenario, provides the manipulator with more timely and richer feedback signals, and reduces the training time of the multi-manipulator system.

[0013] Step D: Improve the experience pool sampling mechanism. Based on the original DDPG algorithm's single experience pool, a dual-pool experience replay mechanism is introduced to increase the capacity of the experience pool, providing sufficient data support for uniform sampling. At the same time, combined with the original DDPG algorithm's experience pool random sampling, this method reduces the correlation between samples, improves sample utilization, and enhances the diversity and stability of the learning process.

[0014] Step 3: Virtual Simulation Training: The recycled object sorting robot first randomly performs grasping actions in a virtual environment, recording its state and perception values. This completes sample collection, and once the experience pool is full, parameter training begins. The perception data obtained in Step 1 is used to train the RNN-MADDPG multi-robot recycled object sorting control algorithm in Step 2. Each round of the experiment involves simultaneous interaction and parameter updates until all network parameters are updated. After training, the network parameters are updated, achieving the maximum expected reward and optimal strategy.

[0015] Step 4: Execute the sorting task: Use the trained DDPG algorithm and deploy it to the actual multi-arm recycled item sorting task. Monitor the execution of the robotic arms and make adjustments and improvements as needed.

[0016] The second aspect of the present invention relates to a multi-robot arm deep reinforcement learning control device for sorting recycled items, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a multi-robot arm deep reinforcement learning control method for sorting recycled items of the present invention.

[0017] A computer-readable storage medium stores a program, which, when executed by a processor, implements a multi-manipulator deep reinforcement learning control method for recycling item sorting according to the present invention.

[0018] The present invention includes: obtaining target object and multi-manipulator system state information; constructing a Deep Deterministic Policy Gradient (DDPG) algorithm network architecture, setting an experience replay pool, and an agent reward function; and deploying it in a real sorting environment for motion guidance. Specifically, the method uses a recurrent neural network (RNN) as the backbone feature extraction network within the DDPG algorithm policy network architecture to capture long-term dependencies in input data and better guide the robot's grasping strategy; introduces a one-to-many discriminant architecture into the evaluation network architecture to accommodate multi-manipulator system training; and proposes a composite reward function and a dual-pool experience replay mechanism to provide the robot with more timely and richer feedback signals, reducing data correlation, improving sample utilization, and enhancing the success rate and efficiency of multi-manipulator recycled object sorting.

[0019] The beneficial effects of the present invention are as follows: by using the RNN recurrent neural network as the backbone feature extraction network of the policy network in the DDPG algorithm, the method can effectively capture the long-term dependencies in the state space when training the robotic arm, and better guide the grasping strategy of the robotic arm; the DDPG algorithm evaluation network is improved, and a one-to-many discriminant architecture is introduced to adapt to the training of multi-robotic arm systems; a composite reward function and a dual-pool experience replay mechanism suitable for multi-robotic arm collaborative control are designed to provide the robotic arm with more timely and richer feedback signals, reduce data correlation, improve sample utilization, and improve the success rate and efficiency of multi-robotic arm recycled item sorting. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is an overall schematic diagram of the implementation of the method of the present invention;

[0021] Figure 2 This is a schematic diagram of the structure of the deep deterministic policy gradient DDPG algorithm improved by the method of the present invention;

[0022] Figure 3 Schematic diagram of the recurrent neural network RNN ​​structure in the method of the present invention;

[0023] Figure 4 Schematic diagram of the improved structure of the experience pool in the method of the present invention. DETAILED DESCRIPTION

[0024] The technical solution of the present invention is further described below with reference to the accompanying drawings.

[0025] Example 1

[0026] refer to Figure 1 This embodiment provides a multi-manipulator deep reinforcement learning control method for recycling item sorting, including the following steps:

[0027] Step 1, target determination: In the recycled items sorting task, the sorting target is determined based on the characteristics of the robot arm, such as gripping capacity and operating accuracy.

[0028] Specifically, the target objects can be metal cans, glass bottles, etc. These objects are relatively large in length and width, moderate in size, and easy to pick up.

[0029] Step 1-1, determine the target object's image position and posture: The sampling camera is responsible for continuously capturing images of recycled items on the conveyor belt and transmitting the captured high-definition images back to an industrial computer running an advanced visual recognition algorithm via Ethernet communication standards to identify the object category and determine the target object's position and rotation angle in the vertical direction of the plane.

[0030] Step 1-2, determine the position and posture of the target object relative to the robotic arm: the original pixel coordinates of the target object are obtained from the image plane, and the target position in the pixel coordinate system is converted to the robotic arm coordinate system through a series of coordinate transformations, so that the robotic arm can accurately locate and perform grasping actions.

[0031] Step 2: Construct the RNN-MADDPG control algorithm for sorting recycled items with multiple robotic arms. Figure 2 : Furthermore, the RNN-MADDPG algorithm is constructed into the following five parts: network architecture, state set, action set, reward function, and experience replay pool;

[0032] Step 2-1: Set up the network structure required by the RNN-MADDPG algorithm: This includes the main network (Actor network), the Critic network, and the target network (Target Actor network and Target Critic network). Each robotic arm in the system has an Actor network for learning grasping strategies, and the entire robotic arm system has only one Critic network for estimating action values.

[0033] The RNN-MADDPG algorithm uses RNN as the backbone feature extraction network of the Actor network. Its logical structure is as follows: Figure 3 Specifically, the hidden layer stores and memorizes previous information and then inputs it into the currently calculated hidden units. This allows the hidden layer input to consist of multiple parts, including the output of the input layer and the output of the hidden layer at the previous moment. That is, the nodes within the hidden layer are self-connected. This mechanism allows information to flow continuously across time steps, helping to capture dynamic changes and long-term dependencies in sequence data and better guide the robot's grasping strategy.

[0034] The RNN-MADDPG algorithm uses a traditional fully connected neural network as the critic network. During training, the critic network has access to information about all robotic arms in the system, including their states and actions. This allows it to accurately assess the expected reward of each action. However, during execution, each robotic arm's actor network can only make decisions based on its own local observations, addressing the issues of collaboration and competition among multiple robotic arms.

[0035] Step 2-2, set the state set S of a single robotic arm: The state set S of a robotic arm can be defined as a vector containing all necessary state information, which is used to describe the dynamic characteristics of the robotic arm and the spatial relationship between it and the target object. Specifically, the state set S of a single robotic arm is:

[0036]

[0037] Among them, θ i Represents the angle of each joint, ω i Represents the angular velocity of each joint, d x , d y , d z Represents the distance from the end position of the robot arm to the target object in the x, y, and z directions, Represents the rotation angle error between the gripping direction of the robot arm and the target object in the plane relative to the vertical direction

[0038] For a multi-manipulator system, the state space of the system, that is, the input data of the Critic network, should contain the states of all manipulators, that is, ∑S.

[0039] Step 2-3, setting the action set A of a single robotic arm: The action set A of the robotic arm can be defined as a set containing all possible control inputs, which are used to drive the various joints of the robotic arm to achieve the desired movement.

[0040] Therefore, the action set A can be expressed as:

[0041] {τ1, τ2, τ3, τ4, τ5, τ6}

[0042] Among them, τ i Represents the control torque of each joint.

[0043] Step 2-4: Design a composite reward function R that integrates multiple reward strategies: The reward function R consists of three parts. Specifically, the following factors are considered:

[0044] ①r1 is the distance reward function: the distance from the end position of the robotic arm to the center of the target object is:

[0045]

[0046] ②r2 is the grasping posture reward function: the fitness of the manipulator’s grasping direction and the target object’s posture is:

[0047]

[0048] ③r3 is the obstacle avoidance reward function: the robot arm may collide with environmental objects during movement, and should avoid:

[0049]

[0050] In the formula, when the robot reaches the target position, Δd is 0, r1 is 10, and r1 is 0 in other cases; when the grasping direction of the robot is consistent with the posture of the target object, When the robot arm collides, r2 is 10, and r2 is 0 in all other cases. When the robot arm collides, r3 is -20, and r3 is 0 in all other cases. This method reduces the reward sparseness problem of the traditional DDPG algorithm in the multi-robot sorting scenario of recycled items, provides more timely and richer feedback signals to the robot arm, and reduces the training time of the multi-robot system.

[0051] Steps 2-5, improve the experience pool sampling mechanism and introduce a dual-pool experience playback mechanism, such as Figure 4 :The experience data that the agent will get t , a t , r t , s t+1} is stored in two experience replay pools, s t represents the state of the agent at time t, a t represents the action taken by the agent at time t, r t represents the reward obtained after taking action at time t, s t+1 It represents the state reached at time t+1 after the agent takes an action. When updating the network parameters, a combination of random sampling and uniform sampling is used. A new experience replay pool is added to expand the sample data set, providing conditions for uniform sampling. Random sampling retains flexibility. The number of samples is:

[0052] μ=w5*u1+w6*u2

[0053] Among them, w5 and w6 are the ratios of random sampling and uniform sampling to the total number of samples, and u1 and u2 are the number of samples of random sampling and uniform sampling.

[0054] Furthermore, after ten rounds of training, the average Q value of the ten rounds was calculated, and the hill climbing method was used to dynamically adjust the ratio of random sampling to uniform sampling to improve the sampling utilization rate, thereby optimizing the experience replay strategy and improving the accuracy and adaptability of the robotic arm's grasping.

[0055] Step 3, RNN-MADDPG algorithm training process:

[0056] Step 3-1: Initialize network parameters, initialize the weights and biases of the Actor strategy network and Critic evaluation network, and initialize the experience pool.

[0057] Step 3-2, randomly perform grasping actions and sense the state of the robot arm s t And perception data, input Actor network to obtain action a t , and execute the action, observing the new state s t+1 , while the Critic evaluation network obtains the reward signal r t , the data (s t , a t , r t , s t+1 ) into the experience pool.

[0058] In step 3-3, after the two experience pools are filled, interactive optimization is started using random sampling and uniform sampling to increase sample diversity and reduce sample correlation during training.

[0059] In steps 3-4, the agent interacts with the environment continuously through the above settings, repeatedly executing the process of selecting actions, interacting with the environment, storing experience, and training the network, so that the agent gradually learns the optimal strategy.

[0060] Step 4: Load the trained model in the real world to control and grasp the robotic arm.

[0061] Through the above steps, this improved deep reinforcement learning DDPG algorithm garbage sorting robot arm grasping control method can successfully perform grasping tasks in different environments and object characteristics.

[0062] Example 2

[0063] This embodiment relates to a multi-robot arm deep reinforcement learning control device for sorting recycled items, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a multi-robot arm deep reinforcement learning control method for sorting recycled items in Example 1.

[0064] Example 3

[0065] This embodiment relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the multi-robot arm deep reinforcement learning control method for recycling item sorting according to embodiment 1 is implemented.

[0066] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be regarded as limited to the specific forms described in the embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A multi-manipulator deep reinforcement learning control method for recycling item sorting, comprising the following steps: Step 1: Determine the target object that the robotic arm needs to grasp, as well as the specific position and posture of the target to be grasped; Step 2: Based on the DDPG algorithm, an improved DDPG network structure is constructed to obtain the RNN-MADDPG multi-manipulator recycled item sorting control algorithm; Step 3: Virtual simulation training: The recycled object sorting robot first randomly performs grasping actions in the virtual environment and records the robot's state and perception values ​​to complete sample collection. Once the experience pool is filled, parameter training begins. Training is performed by combining the perception data obtained in Step 1 with the RNN-MADDPG multi-robot recycled object sorting control algorithm in Step 2. In each round of trials, the interaction process and parameter update are carried out simultaneously until all network parameters are updated. After training, the network parameters are updated to obtain the maximum expected reward and the optimal strategy. Step 4: Execute the sorting task: Use the trained RNN-MADDPG algorithm and deploy it to the actual multi-robotic arm recycled item sorting task, monitor the execution of the robotic arms, and make adjustments and improvements.

2. The multi-manipulator deep reinforcement learning control method for recycling item sorting according to claim 1, characterized in that: Step 1 specifically includes: Step 1-1: Determine the target object's image position and posture: The sampling camera continuously captures images of recycled items on the conveyor belt and transmits the captured high-definition images via Ethernet communication standards to an industrial computer running a visual recognition algorithm to identify the object category and determine the target object's position and rotation angle in the vertical direction of the plane. Step 1-2, determine the position and posture of the target object relative to the robotic arm: the original pixel coordinates of the target object are obtained from the image plane, and the target position in the pixel coordinate system is converted to the robotic arm coordinate system through a series of coordinate transformations, so that the robotic arm can accurately locate and perform grasping actions.

3. The multi-manipulator deep reinforcement learning control method for recycling item sorting according to claim 1, characterized in that: Step 2 specifically includes: the RNN-MADDPG algorithm construction includes: network architecture, state set, action set, reward function, and experience replay pool; specifically includes: Step 2-1: Set up the network structure required by the RNN-MADDPG algorithm: including the main network Actor network, Critic network, and target network Target Actor network and Target Critic network. Each robotic arm in the system has an Actor network for learning grasping strategies, and the entire robotic arm system has only one Critic network for estimating action values. The RNN-MADDPG algorithm uses RNN as the backbone feature extraction network of the Actor network. The network includes: storing and memorizing previous information in the hidden layer, and then inputting it into the hidden unit of the current calculation, so that the input of the hidden layer can be composed of multiple parts, the output of the input layer and the output of the hidden layer at the previous moment, that is, the nodes in the hidden layer are self-connected; The RNN-MADDPG algorithm uses a traditional fully connected neural network as the critic network. During the training phase, the critic network accesses information about all robotic arms in the system, including their states and actions, to accurately assess the expected reward of each action. Step 2-2, set the state set S of a single robotic arm: The state set S of a robotic arm is defined as a vector containing all necessary state information, which is used to describe the dynamic characteristics of the robotic arm and the spatial relationship between it and the target object; the state set S of a single robotic arm is: Among them, θ i Represents the angle of each joint, ω i Represents the angular velocity of each joint, d x , d y , d z Represents the distance from the end position of the robot arm to the target object in the x, y, and z directions, Represents the rotation angle error between the gripping direction of the robot arm and the target object in the plane relative to the vertical direction; For a multi-manipulator system, the state space of the system, i.e., the input data of the Critic network, should contain the states of all the manipulators, i.e., ∑S; Step 2-3, set the action set A of a single robot arm: The action set A of the robot arm is defined as a set of all possible control inputs used to drive each joint of the robot arm to achieve the desired motion; The action set A is expressed as: {τ1, τ2, τ3, τ4, τ5, τ6} Among them, τ i Represents the control torque of each joint; Step 2-4, design a composite reward function R that integrates multiple reward strategies: The reward function R consists of three parts: ①r1 is the distance reward function: the distance from the end position of the robotic arm to the center of the target object is: ②r2 is the grasping posture reward function: the fitness of the manipulator’s grasping direction and the target object’s posture is: ③r3 is the obstacle avoidance reward function: the possibility of collision between the robot arm and the environment during movement: In the formula, when the robot reaches the target position, Δd is 0, r1 is 10, and r1 is 0 in other cases; when the grasping direction of the robot is consistent with the posture of the target object, is 0, then r2 is 10, and in other cases r2 is 0; when the robot arm collides, r3 is -20, and in other cases r3 is 0; Step 2-5, improve the experience pool sampling mechanism and introduce a dual-pool experience playback mechanism: the agent will get the experience data {s t , a t , r t , s t+1 } is stored in two experience replay pools, s t represents the state of the agent at time t, a t represents the action taken by the agent at time t, r t represents the reward obtained after taking action at time t, s t+1 It represents the state reached at time t+1 after the agent takes action; when updating the network parameters, random sampling and uniform sampling are combined, and the number of sampling samples is: μ=w5*u1+w6*u2 Among them, w5 and w6 are the ratios of random sampling and uniform sampling to the total number of samples, and u1 and u2 are the number of samples of random sampling and uniform sampling.

4. The multi-manipulator deep reinforcement learning control method for recycling item sorting according to claim 3, characterized in that: In steps 2-5, after ten rounds of training, the average Q value of the ten rounds is calculated, and the hill climbing method is used to dynamically adjust the ratio of random sampling to uniform sampling.

5. The multi-manipulator deep reinforcement learning control method for recycling item sorting according to claim 1, characterized in that: The RNN-MADDPG algorithm training process in step 3 is as follows: Step 3-1, initialize network parameters, initialize the weights and biases of the Actor strategy network and the Critic evaluation network, and initialize the experience pool; Step 3-2, randomly perform grasping actions and sense the state of the robot arm s t And perception data, input Actor network to obtain action a t , and execute the action, observing the new state s t+1 , while the Critic evaluation network obtains the reward signal r t , the data (s t , a t , r t , s t+1 ) into the experience pool; Step 3-3: After the two experience pools are filled, interactive optimization is started using random sampling and uniform sampling to increase sample diversity and reduce sample correlation during training; In steps 3-4, the agent interacts with the environment continuously through the above settings, repeatedly executing the process of selecting actions, interacting with the environment, storing experience, and training the network, so that the agent gradually learns the optimal strategy.

6. A multi-manipulator deep reinforcement learning control device for recycling item sorting, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement a multi-robot arm deep reinforcement learning control method for sorting recycled items according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that A program is stored thereon, which, when executed by a processor, implements a multi-robot arm deep reinforcement learning control method for sorting recycled items as described in any one of claims 1-5.

Images