MAPPO double-arm cooperative intelligent picking method

Abstract

The invention discloses a double-arm collaborative picking method based on reinforcement learning, a double-arm six-degree-of-freedom robot is adopted, a double-agent algorithm based on reinforcement learning is stored in an electric control device, and the electric control device comprises an actor network, a commentator network, an experience playback buffer area, two agents and an environment module storing st information; each of the two intelligent agents corresponds to one mechanical arm; continuously reading jointt and evaluating the sum of rewards rt by the two intelligent agents in work; the objective function of the actor network is calculated through a dominant function # imgabs0 #, the commentator network is optimized, a dynamic obstacle avoidance strategy is executed, a multi-objective greedy picking strategy is executed, an independent working space and a public working space are divided, and efficient apple picking through double-arm cooperation is achieved. According to the method, MAPPO reinforcement learning and a multi-target greedy picking strategy (MOGPS) algorithm are fused, the problems of double-arm action synchronization and task allocation are solved, the limitation of a traditional algorithm in double-arm cooperation is overcome, the double-arm cooperation picking efficiency is improved, and the adaptability of the robot in different operation scenes is enhanced.

Description

Technical Field

[0001] The invention relates to the technical field of apple picking, and in particular to a dual-arm robot picking technology. Background Art

[0002] 1. Limitations of existing technologies related to single-arm robots.

[0003] In the field of fruit picking, although single-arm picking robots still dominate, their limitations are gradually becoming apparent. Limited by the range of joint motion, single-arm robots have difficulty reaching areas outside the workspace, resulting in unsatisfactory picking efficiency. Single-arm robots have limited ability to handle multiple tasks, which restricts their actual application in environments where multiple tasks need to be performed simultaneously. In addition, single-arm robots also have the following shortcomings: insufficient adaptability to different crops and environmental changes, and weak obstacle avoidance capabilities in complex environments.

[0004] In terms of obstacle avoidance, obstacles such as branches and leaves often make it difficult for single-arm robots to plan a suitable picking path, which not only affects the smoothness of the operation, but also reduces the overall picking success rate.

[0005] The complexity and variability of the agricultural harvesting environment make it difficult for traditional automation equipment to adapt. Especially in multi-robot collaborative operations, efficient task allocation and picking accuracy have become key challenges that need to be solved. These limiting factors have hindered the in-depth application of automation technology in the field of agricultural harvesting.

[0006] 2. Compared with single-arm robots, dual-arm robots are more suitable for the unstructured task of picking apples in orchards.

[0007] The two arms of a dual-arm robot can perform different tasks at the same time, thus improving picking efficiency. Compared with single-arm robots, dual-arm robots can adapt to more complex and changeable working environments. When performing unstructured tasks, dual-arm robots are more flexible and adaptable. Therefore, in the unstructured task of picking apples in orchards, using dual-arm robots to pick apples will become a future development trend.

[0008] 3. Problems encountered by the inventor when applying the dual-arm robot to the unstructured task of picking apples in an orchard.

[0009] (1) Two-arm collaboration problem:

[0010] Existing dual-arm robot technology usually adopts a master-slave control mode, in which the master arm is responsible for performing the main operation tasks, while the slave arm assists the master arm in operation. The two achieve collaboration through tightly synchronized action sequences. The control architecture relies on real-time monitoring and dynamic path planning techniques to avoid collisions between the two arms during operation and ensure the synchronization and accuracy of collaborative actions. The implementation of the master-slave control strategy involves optimization algorithms and multi-sensor data fusion technology, which work together to improve the robot's adaptive ability in uncertain environments.

[0011] In the unstructured environment of an orchard, apples do not grow in a fixed position and are often obscured by branches and leaves, which poses a challenge to the efficient collaboration of dual-arm robotic arms.

[0012] Existing technologies make it difficult to achieve precise synchronization and coordination between the two arms to adapt to complex and changing picking environments.

[0013] (2) Dynamic obstacle avoidance problem:

[0014] In the complex operating environment of the orchard, the dynamic obstacle avoidance capability of the dual-arm robot is crucial, which requires the system to be able to process environmental status information in real time and accurately to effectively prevent potential collisions and damage events. However, the existing path planning algorithms often fail to meet the requirements of dynamic obstacle avoidance when dealing with the ever-changing and complex scenes in the orchard, thus affecting the performance and efficiency of the obstacle avoidance operation.

[0015] In terms of obstacle avoidance technology, existing harvesting robots have obvious performance deficiencies, especially when dealing with complex orchard scenes. Their dynamic obstacle avoidance capabilities are relatively fragile, and it is difficult to accurately identify and predict obstacles in the orchard. In addition, the obstacle avoidance strategy is not robust enough. When faced with complex obstacles, the risk of obstacle avoidance failure increases significantly, which not only reduces the efficiency of harvesting operations, but also increases the safety risks of robot operations.

[0016] (3) Picking strategy issues:

[0017] In the operation process of the dual-arm picking robot, the combination of task allocation optimization and real-time decision-making capabilities is the core link. The system must continuously monitor the position of the fruit and changes in the surrounding environment, such as the obstruction of branches and leaves or the uneven distribution of fruits. Based on this information, how to adjust the task allocation in real time to ensure that the two robotic arms can perform the most appropriate operations according to the current situation is a problem that needs to be solved.

[0018] In the actual orchard environment, factors such as the dynamic changes in the positions of fruits and obstacles still pose a test to the picking efficiency and accuracy of the robotic arm.

[0019] The current picking decision architecture has a significant rigidity problem and lacks the necessary adaptability and flexibility. The architecture fails to fully combine the specific characteristics of fruit growth distribution, resulting in the inability to make real-time dynamic adjustments to the specific attributes of the fruit. In the dual-arm collaborative picking operation, the operational flexibility and adaptability of the system need to be improved. The existing picking execution mechanism shows obvious rigidity characteristics and fails to give full play to the operational flexibility and maneuverability of collaborative robots, which greatly limits the flexibility and efficiency of the picking operation. Therefore, the optimization and upgrading of the existing picking system is particularly critical, especially in enhancing the operational flexibility of the system and improving the control precision. The picking decision framework must be reconstructed to achieve a more intelligent and adaptive picking strategy, thereby improving the overall intelligence level and operational efficiency of the picking operation.

[0020] (4) Challenges of adaptive picking path planning in a dynamic orchard environment

[0021] When faced with the dynamic changes in the orchard environment, the path planning adaptability problem of the picking robot becomes particularly prominent. The traditional static path planning method is not sufficient to cope with the real-time changes in the environment and lacks the necessary immediate response capabilities. In addition, due to the lack of an efficient global optimization algorithm, it is difficult for the robot to optimize the path in a complex orchard environment. The limitations of the adaptive strategy make it impossible for the robot to flexibly adapt the strategy to different orchard layouts and fruit distribution characteristics, which significantly restricts its operating efficiency in a changing environment. Summary of the invention

[0022] The purpose of the present invention is to provide a dual-arm collaborative picking method based on reinforcement learning, to improve the efficiency and adaptability of dual-arm collaboration, and to improve the efficiency of apple picking in an unstructured environment of an orchard.

[0023] To achieve the above purpose, a dual-arm collaborative picking method based on reinforcement learning is provided in the present invention, which adopts a dual-arm six-degree-of-freedom robot. The dual-arm six-degree-of-freedom robot includes two mechanical arms and an electric control device. The end of each mechanical arm has an end effector for picking apples. The electric control device is an industrial control computer. The dual-arm six-degree-of-freedom robot collects the current state information of the target object in real time through the cameras installed on the two mechanical arms. t ,s t Including the location information of the target apple, the posture information of the target apple, the distance information of the target apple relative to the two robotic arms, the location information of the obstacle, and the distance information between the two robotic arms and the obstacle;

[0024] The electronic control device stores a dual-agent algorithm based on reinforcement learning;

[0025] The algorithm structure of the dual-agent algorithm based on reinforcement learning includes an actor network, a critic network, an experience replay buffer, two agents, and a storage medium. t The environment module of the information; two agents correspond to one robot arm respectively;

[0026] The dual-arm six-degree-of-freedom robot stores joint status information that reflects the current joint status. t , joint t Including joint coordinates, joint angular velocity, joint angular acceleration, joint angle, and the coordinates and rotational posture of the end effector;

[0027] The two agents read the joint continuously during work t , and in the current joint state information joint t Information about the current environment t The sum of the rewards after the association evaluation is r t ;

[0028] The two agents receive and execute action instructions issued by the actor network in real time. t The dual-arm six-degree-of-freedom robot monitors the state of the next moment through the camera t+1 ; Set the current s t 、a t 、r t and t+1 The set of experience tuples is called experience tuples. Let experience tuple N = (s t ,a t ,r t ,s t+1 ) is stored in the experience replay buffer;

[0029] The critic network randomly samples from the experience replay buffer to calculate each state-action pair (s t -a t )'s advantage function Specifically represents the advantage function estimation of agent i at the kth time step; use the critic network to estimate each state s t The value function V is calculated according to the advantage estimation formula

[0030] The advantage estimate formula is: in, is the temporal difference error of agent i at the kth time step; γ is the discount factor in the hyperparameters of the generalized advantage estimate, which is used to balance the immediate reward and future reward; λ is the smoothing parameter in the hyperparameters of the generalized advantage estimate, which is used to control the bias-variance trade-off of the advantage estimate;

[0031] Using advantage function Calculate the objective function of the actor network and guide the actor network to update parameters along the gradient ascent direction by calculating the policy loss gradient; at the same time, calculate the objective function of the critic network and optimize its parameters along the gradient descent direction by calculating the loss gradient;

[0032] Use the advantage estimate to calculate each pair of s t -a t The objective function formula is:

[0033]

[0034] in is the strategy update ratio of agent i; ε represents the proportional coefficient of the clipping range; Clip represents

[0035] Indicates that the policy update ratio is clipped within the clipping range; S represents the entropy parameter of the policy; σ is a parameter used to control the entropy coefficient;

[0036] The critic network optimization objective is:

[0037]

[0038] in is the discounted reward; B represents the size of batch_size, n represents the number of agents, and the value of n is 2; represents the critic network's valuation of the state; Represents the evaluation of the state by the critic network using the old parameters; ∈ represents the scale factor of the clipping range; Clip is used to clip the evaluation scope;

[0039] It also includes evaluating the total reward r through the collision risk evaluation function t Methods;

[0040] The electronic control device obtains the current joint status information of the dual-arm six-degree-of-freedom robot t and the current state information st of the target object, and by using the state transition model P(s t+1 |s t ,a t ) to predict the state st of the next moment after the arms perform action at +1 ;

[0041] Get s t+1 Afterwards, based on the distance between the two robotic arms and the obstacle and the distance between the two robotic arm joints, it is evaluated whether the two robotic arms will collide with the surrounding objects or collide with each other; t+1The predicted value and the result of collision detection are used to calculate the expected collision risk Ecrt; the obstacle is defined as:

[0042] Obstacle{(x min ,y min ,z min )≤(x,y,z)≤(x max ,y max ,z max )};

[0043] Assume that the coordinates of the robot joint Linki at time step t are P i =(x i ,y i ,z i ), the closest point between the robot arm and the obstacle is P min =(x o ,y o ,z o ), then the direction vector of joint i of the robot arm is for:

[0044]

[0045] The closest point P of the branch obstacle min Link with robot arm joint i The vector is:

[0046]

[0047] The shortest distance from the obstacle to the robot arm joint i is exist The vertical component of the projection in the direction is d. min ,

[0048] According to the positions of the joints of the two robotic arms in the world coordinate system, the minimum Euclidean distance between the joints is calculated, and the dynamic collision risk between the two robotic arms is evaluated based on the minimum Euclidean distance;

[0049] Ecrt is the expected collision risk at time step t, Ecrt uses an exponential relationship to evaluate the expected collision risk based on the Euclidean distance between the robot and the obstacle; the ε coefficient is used to control the exponential rate, and the σ coefficient is used to limit the collision risk threshold;

[0050] Reward sum r t =Rt+Rc; Rt is the task completion reward reflecting the task completion, while Rc is the penalty based on the collision risk assessment, Rc=γ·Ecrt; γ is a coefficient used to adjust the impact of the collision risk assessment on the total reward;

[0051] Combining the above formulas, we can get the collision risk assessment function, i.e. the total reward r t The calculation formula is:

[0052] The apple picking space is divided into a left arm workspace and a right arm workspace; the left arm workspace and the right arm workspace have an intersection and the intersection is called a public workspace, which is also called a collaborative workspace;

[0053] It also includes a multi-objective greedy picking strategy to determine the optimal picking sequence in an independent workspace, cope with the non-uniformity of apple distribution and the heterogeneity of the complexity of the picking operation environment, and coordinate the resource utilization during the two-arm picking process;

[0054] The first goal of the multi-objective greedy picking strategy is to minimize the picking distance D i , D i That is, the distance between the end effector of the robot arm and the apple; the position of the end effector of the robot arm in the world coordinate system is The position of the i-th apple in the world coordinate system is Picking distance D i The calculation formula is:

[0055]

[0056] The second goal of the multi-objective greedy picking strategy is to maximize the efficiency of avoiding obstacles. A dynamic obstacle avoidance strategy f o ,

[0057]

[0058] in, The obstacle-free degree of the path from the two robot end effectors to the target fruit is evaluated by quantifying the area occupancy of the apple contour in the projection vector direction.

[0059] The independent workspace and the public workspace are divided into several picking areas. The third goal of the multi-objective greedy picking strategy is to make the robot arm preferentially pick apples in the fruit-dense area. The fruit-dense area optimization strategy ρ is proposed. i ;

[0060] Among them, ρ i represents the fruit density of the ith region, N i represents the number of fruits in the area, V i Represents the volume of the area; when determining the picking order of multiple areas, press ρ i Arrange the picking order from high to low values.

[0061] The dual-arm six-degree-of-freedom robot picks apples in the following steps:

[0062] The first step is to obtain the s required by the environment module t information;

[0063] The second step is to divide the work area of ​​the current picking environment into independent workspaces corresponding to the left and right arms of the dual-arm six-degree-of-freedom robot and a public workspace corresponding to the left and right arms of the dual-arm six-degree-of-freedom robot. The public workspace is also called a collaborative workspace.

[0064] The third step is to execute a multi-objective greedy picking strategy for the left arm and the right arm at the same time, driving the left arm and the right arm to pick apples in the corresponding independent workspace at the same time;

[0065] When executing the multi-objective greedy picking strategy, the fruit-dense area optimization strategy is first executed to determine the priority of the operating area, and the picking order of the operating area is arranged according to the priority; then the picking distance D is calculated in the current operating area according to the picking order of the operating area. i , and then execute the dynamic obstacle avoidance strategy f o , determines the apple picking priority in the current operation area,

[0066] Then perform the area picking action; the area picking action is:

[0067] The robot arm and its end effector pick the target apples for current picking, put the picked apples into the apple collection container, and then determine whether the apples in the current operation area have been picked. If the apples in the current operation area have not been picked, the regional picking action is re-executed according to the apple picking priority in the current operation area until the apples in the current operation area are picked. Then, the robot arm returns to re-execute the multi-objective greedy picking strategy in the third step until all apples in the independent workspace corresponding to the current robot arm are picked, and then the fourth step is executed;

[0068] The fourth step is to prepare to pick apples in the public workspace and perform the number determination of the robotic arms;

[0069] The number of robotic arms is determined by: determining whether the number of robotic arms that have completed the apple picking task in the independent workspace is greater than 1;

[0070] If it is not greater than 1, a multi-objective greedy picking strategy is executed for a robotic arm that has completed the apple picking task in the independent workspace. The robot grabs the apples with a single arm, puts the picked apples into the apple collection container, and then executes the end of picking judgment;

[0071] If it is greater than 1, the multi-objective greedy picking strategy is executed for the two robotic arms, the apple grabbing tasks are dynamically allocated, and the total reward r is evaluated by the collision risk assessment function. t , and then the two agents receive and execute the action instructions a issued by the actor network in real time t , grab the apples with both arms, put the picked apples into the apple collection container, and then execute the end of picking judgment;

[0072] The end of picking judgment is: judge whether the apples in the public workspace have been picked. If not, return to re-execute the number of robotic arms judgment; if the apples have been picked, end all picking work.

[0073] The present invention has the following advantages:

[0074] The present invention adopts the MAPPO reinforcement learning algorithm to solve the problems of dual-arm movement synchronization and task allocation, improve the efficiency and adaptability of dual-arm collaboration, and overcome the limitations of traditional algorithms in dual-arm collaboration.

[0075] This invention cleverly combines MAPPO reinforcement learning and the multi-objective greedy picking strategy (MOGPS) algorithm, significantly improving the operating efficiency and quality of the dual-arm six-degree-of-freedom picking robot in the field of agricultural automation. By autonomously learning and optimizing the grasping strategy, the robot achieves precise picking in complex environments, and the introduction of dynamic collision evaluation functions ensures the safety of the operation process. The integrated application of these technologies not only improves the picking efficiency, but also enhances the robot's adaptability in different operating scenarios.

[0076] In the present invention, the two agents are isomorphic and share a set of network parameters, forming a compact proxy model. Accordingly, both agents follow the policy network π determined by the parameter θ. θ The value function network V is determined by the parameter φ φ , to achieve synchronous decision-making and action execution. This parameter sharing mechanism not only promotes the policy consistency between the two agents, but also significantly improves the collective learning efficiency and action coordination of the system through policy iteration among multiple agents. In addition, this structure also takes advantage of the actor-critic architecture, where the actor is responsible for action selection and the critic evaluates the action value in a given state, jointly driving the optimal behavior strategy learning of the agent in a dynamic environment.

[0077] Through the collision risk assessment function, that is, the sum of rewards r t The calculation formula for the total reward r tIt depends not only on the task completion Rt, but also on the collision risk estimate Ecrt based on the robot’s current state. In this way, the reward function can be dynamically adjusted to encourage the robot to take less risky actions while still pursuing task completion.

[0078] The collision risk assessment function in this invention goes beyond the traditional method of imposing penalties only after a collision occurs. Instead, it evaluates the robot's action selection in real time by predicting potential collision risks. The collision risk assessment is performed by in-depth analysis of the relative spatial distance between the robot's end effector and the target apple, as well as the Euclidean distance of the two robot arm joint configurations. Using this risk assessment result, the reward mechanism can be dynamically adjusted to optimize the robot's behavior strategy in complex environments.

[0079] The multi-objective greedy picking strategy can optimize the picking order and improve the picking efficiency; the dynamic obstacle avoidance strategy can reduce the probability of collision during the picking process, improve the picking efficiency and safety, and improve the continuity of the picking operation, optimize resource utilization, and enhance the adaptability and robustness of the dual-arm six-degree-of-freedom robot to the complexity and dynamic changes of the orchard environment. The fruit-dense area optimization strategy ensures that the fruit-dense area is picked first.

[0080] like Figure 4 As shown in the figure, the fruit density in the independent workspace of the left arm (left robotic arm) is greater than that in the public workspace, and the fruit density in the public workspace is greater than that in the workspace of the right arm (right robotic arm). Therefore, the right arm will first pick the apples in its corresponding independent workspace, so the fruits in the public workspace will be picked by the right arm. The division of independent workspace and public workspace makes the picking amount of the left and right arms more balanced, improving the picking efficiency and equipment utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 It is a network structure diagram of the dual-agent algorithm based on reinforcement learning in the present invention, namely the MAPPO reinforcement learning model.

[0082] Figure 2 This is a diagram of a dual robotic arm picking model.

[0083] Figure 3 It is a schematic diagram of the principles related to the collision risk assessment function.

[0084] Figure 4 This is a schematic diagram of the principle of the optimization strategy for fruit-dense areas. It can also be seen from the figure that the division of independent workspace and public workspace makes the picking amount of the left arm and the right arm more balanced, improving the picking efficiency and equipment utilization.

[0085] Figure 5It is a picking process schematic diagram of the present invention. DETAILED DESCRIPTION

[0086] like Figures 1 to 5 As shown, the present invention provides a dual-arm collaborative picking method based on reinforcement learning, which adopts a dual-arm six-degree-of-freedom robot. The dual-arm six-degree-of-freedom robot includes two mechanical arms (left arm and right arm) and an electric control device. The end of each mechanical arm has an end effector for picking apples; the electric control device is an industrial control computer (industrial controller); the dual-arm six-degree-of-freedom robot collects the current state information of the target object in real time through the cameras installed on the two mechanical arms. t Specifically, the electronic control device processes the image collected by the camera through the built-in image processing algorithm to obtain s t ;s t Including the location information of the target apple, the posture information of the target apple, the distance information of the target apple relative to the two robotic arms, the location information of the obstacle, and the distance information between the two robotic arms and the obstacle.

[0087] The electronic control device stores a dual-agent algorithm based on reinforcement learning; specifically, it is a MAPPO algorithm based on the actor-critic architecture.

[0088] The algorithm structure of the dual-agent algorithm based on reinforcement learning includes an actor network, a critic network, an experience replay buffer, two agents, and a storage medium. t The environment module (environment) of the information; the two agents correspond to one robotic arm respectively.

[0089] The dual-arm six-degree-of-freedom robot stores joint status information that reflects the current joint status. t , joint t Including joint coordinates, joint angular velocity, joint angular acceleration, joint angle, and the coordinates and rotational posture of the end effector.

[0090] The two agents read the joint continuously during work t , and in the current joint state information joint t Information about the current environment t The sum of the rewards after the association evaluation is r t .

[0091] The two agents receive and execute action instructions issued by the actor network in real time. t The dual-arm six-degree-of-freedom robot monitors the state of the next moment through the camera t+1 ; Set the current s t、a t 、r t and t+1 The set of experience tuples is called experience tuples. Let experience tuple N = (s t ,a t ,r t ,s t+1 ) is stored in the experience replay buffer.

[0092] The critic network randomly samples from the experience replay buffer to calculate each state-action pair (s t -a t )'s advantage function Specifically represents the advantage function estimation of agent i at the kth time step; use the critic network to estimate each state s t The value function V is calculated according to the advantage estimation formula In the present invention, there are two intelligent agents, so the value of i is 1 and 2, namely, intelligent agent 1 and intelligent agent 2.

[0093] The advantage estimate formula is: in, is the temporal difference error of agent i at the kth time step; γ is the discount factor in the hyperparameters of the generalized advantage estimation (GAE), which is used to balance the immediate reward and future reward; λ is the smoothing parameter in the hyperparameters of the generalized advantage estimation (GAE), which is used to control the bias-variance trade-off of the advantage estimation.

[0094] Using advantage function The objective function of the actor network is calculated, and the policy loss gradient is calculated to guide the actor network to update its parameters along the gradient ascent direction. At the same time, the objective function of the critic network is calculated, and the loss gradient is calculated to optimize its parameters along the gradient descent direction.

[0095] Use the advantage estimate to calculate each pair of s t -a t The objective function formula is:

[0096]

[0097] in is the strategy update ratio of agent i; ε represents the proportional coefficient of the clipping range; Clip represents

[0098] It indicates that the policy update ratio is clipped within the clipping range; S represents the entropy parameter of the policy; σ is the parameter used to control the entropy coefficient.

[0099] The critic network optimization objective is:

[0100]

[0101] in is the discounted reward; B represents the size of batch_size, n represents the number of agents, and the value of n is 2; represents the critic network's valuation of the state; Represents the evaluation of the state by the critic network using the old parameters; ∈ represents the scale factor of the clipping range; Clip is used to clip the evaluation range.

[0102] In the present invention, the two agents are isomorphic and share a set of network parameters, forming a compact proxy model. Accordingly, both agents follow the policy network π determined by the parameter θ. θ The value function network V is determined by the parameter φ φ , to achieve synchronous decision-making and action execution. This parameter sharing mechanism not only promotes the policy consistency between the two agents, but also significantly improves the collective learning efficiency and action coordination of the system through policy iteration among multiple agents. In addition, this structure also takes advantage of the actor-critic architecture, where the actor is responsible for action selection and the critic evaluates the action value in a given state, jointly driving the optimal behavior strategy learning of the agent in a dynamic environment.

[0103] It also includes evaluating the total reward r through the collision risk evaluation function t Methods;

[0104] The electronic control device obtains the current joint status information of the dual-arm six-degree-of-freedom robot t and the current state information st of the target object, and by using the state transition model P(s t+1 |s t ,a t ) to predict the state st of the next moment after the arms perform action at +1 ;

[0105] Get s t+1 Afterwards, based on the distance between the two robotic arms and the obstacle and the distance between the two robotic arm joints, it is evaluated whether the two robotic arms will collide with the surrounding objects or collide with each other; t+1 The predicted value and the result of collision detection are used to calculate the expected collision risk Ecrt; Figure 3 As shown, the obstacle is defined as:

[0106] Obstacle{(x min ,y min ,z min )≤(x,y,z)≤(x max ,ymax ,z max )};

[0107] Assume that the coordinates of the robot joint Linki at time step t are P i =(x i ,y i ,z i ), the closest point between the robot arm and the obstacle is P min =(x o ,y o ,z o ), then the direction vector of joint i of the robot arm is for:

[0108]

[0109] The closest point P of the branch obstacle min Link with robot arm joint i The vector is:

[0110]

[0111] The shortest distance from the obstacle to the robot arm joint i is exist The vertical component of the projection in the direction is d. min ,

[0112] Similarly, according to the positions of each joint of the two robotic arms in the world coordinate system, the minimum Euclidean distance between each joint is calculated, and the dynamic collision risk between the two robotic arms is evaluated based on the minimum Euclidean distance.

[0113] Ecrt is the expected collision risk at time step t. Ecrt uses an exponential relationship to evaluate the expected collision risk based on the Euclidean distance between the robot and the obstacle; the ε coefficient is used to control the exponential rate, and the σ coefficient is used to limit the collision risk threshold.

[0114] Reward sum r t =Rt+Rc; Rt is the task completion reward reflecting the task completion situation, and Rc is the penalty based on the collision risk assessment, Rc=γ·Ecrt; γ is a coefficient used to adjust the impact of the collision risk assessment on the total reward.

[0115] Combining the above formulas, we can get the collision risk assessment function, i.e. the total reward r t The calculation formula is:

[0116] Through the collision risk assessment function, that is, the sum of rewards rt The calculation formula for the total reward r t It depends not only on the task completion Rt, but also on the collision risk estimate Ecrt based on the robot’s current state. In this way, the reward function can be dynamically adjusted to encourage the robot to take less risky actions while still pursuing task completion.

[0117] The collision risk assessment function in this invention goes beyond the traditional method of imposing penalties only after a collision occurs. Instead, it evaluates the robot's action selection in real time by predicting potential collision risks. The collision risk assessment is performed by in-depth analysis of the relative spatial distance between the robot's end effector and the target apple, as well as the Euclidean distance of the two robot arm joint configurations. Using this risk assessment result, the reward mechanism can be dynamically adjusted to optimize the robot's behavior strategy in complex environments.

[0118] The apple picking space is divided into a left arm workspace and a right arm workspace; the left arm workspace and the right arm workspace have an intersection and the intersection is called a public workspace, which is also called a collaborative workspace.

[0119] It also includes a Multi-Objective Greedy Picking Strategy (MOGPS), which is used to determine the optimal picking order in an independent workspace, cope with the non-uniformity of apple distribution and the heterogeneity of the complexity of the picking operation environment (i.e., the environmental complexity varies in different areas), and coordinate resource utilization during the two-arm picking process.

[0120] The first goal of the multi-objective greedy picking strategy is to minimize the picking distance D i , D i That is, the distance between the end effector of the robot arm and the apple; let the position of the end effector of the robot arm in the world coordinate system be The position of the i-th apple in the world coordinate system is Picking distance D i The calculation formula is:

[0121]

[0122] Since the picking process is greatly affected by obstacles, we use the analysis method based on vector direction projection to evaluate whether there are obstacles in the path between the two robot end effectors and the apples.

[0123] The second goal of the multi-objective greedy picking strategy is to maximize the efficiency of avoiding obstacles. A dynamic obstacle avoidance strategy f o ,

[0124]

[0125] in, The obstacle-free degree of the path from the two robot end effectors to the target fruit is evaluated by quantifying the area occupancy of the obstacle on the apple contour in the direction of the projection vector.

[0126] The independent workspace and the public workspace are divided into several picking areas. The third goal of the multi-objective greedy picking strategy is to make the robot arm preferentially pick apples in the fruit-dense area. The fruit-dense area optimization strategy ρ is proposed. i ;

[0127] Among them, ρ i represents the fruit density of the ith region, N i represents the number of fruits in the area, V i Represents the volume of the area; when determining the picking order of multiple areas, press ρ i Arrange the picking order from high to low values.

[0128] In the multi-objective greedy picking strategy, each selection is based on minimizing D in the current state. i apples to be picked. Figure 4 The apples in the public workspace shown will be completed more by (or entirely by) the robotic arms with fewer apples in the corresponding independent workspace.

[0129] The multi-objective greedy picking strategy can optimize the picking order and improve the picking efficiency; the dynamic obstacle avoidance strategy can reduce the probability of collision during the picking process, improve the picking efficiency and safety, and improve the continuity of the picking operation, optimize resource utilization, and enhance the adaptability and robustness of the dual-arm six-degree-of-freedom robot to the complexity and dynamic changes of the orchard environment. The fruit-dense area optimization strategy ensures that the fruit-dense area is picked first.

[0130] like Figure 4 As shown in the figure, the fruit density in the independent workspace of the left arm (left robotic arm) is greater than that in the public workspace, and the fruit density in the public workspace is greater than that in the workspace of the right arm (right robotic arm). Therefore, the right arm will first pick the apples in its corresponding independent workspace, so the fruits in the public workspace will be picked by the right arm. The division of independent workspace and public workspace makes the picking amount of the left and right arms more balanced, improving the picking efficiency and equipment utilization.

[0131] The dual-arm six-degree-of-freedom robot picks apples in the following steps:

[0132] The first step is to obtain the s required by the environment module t information;

[0133] The second step is to divide the work area of ​​the current picking environment into independent workspaces corresponding to the left and right arms of the dual-arm six-degree-of-freedom robot and a public workspace corresponding to the left and right arms of the dual-arm six-degree-of-freedom robot. The public workspace is also called a collaborative workspace.

[0134] The third step is to execute a multi-objective greedy picking strategy for the left arm and the right arm at the same time, driving the left arm and the right arm to pick apples in the corresponding independent workspace at the same time;

[0135] When executing the multi-objective greedy picking strategy, the fruit-dense area optimization strategy is first executed to determine the priority of the operating area (the priority is the order of picking operations), and the picking order of the operating area is arranged according to the priority; then, according to the picking order of the operating area, the picking distance D is calculated in the current operating area. i , and then execute the dynamic obstacle avoidance strategy f o , determine the apple picking priority (i.e., apple picking order) in the current operation area;

[0136] Then perform the area picking action; the area picking action is:

[0137] The robot arm and its end effector pick the target apples for current picking, put the picked apples into the apple collection container, and then determine whether the apples in the current operation area have been picked. If the apples in the current operation area have not been picked, the regional picking action is re-executed according to the apple picking priority in the current operation area until the apples in the current operation area are picked. Then, the robot arm returns to re-execute the multi-objective greedy picking strategy in the third step until all apples in the independent workspace corresponding to the current robot arm are picked, and then the fourth step is executed;

[0138] The fourth step is to prepare to pick apples in the public workspace and perform the number determination of the robotic arms;

[0139] The number of robotic arms is determined by: determining whether the number of robotic arms that have currently completed the apple picking task in the independent workspace is greater than 1 (1 or 2);

[0140] If it is not greater than 1 (that is, the number of idle robotic arms is 1), a multi-objective greedy picking strategy is executed for a robotic arm that has currently completed the apple picking task in the independent workspace. The single arm grabs the apples, puts the picked apples into the apple collection container, and then executes the end of picking judgment;

[0141] If it is greater than 1 (that is, the number of idle robotic arms is 2), the multi-objective greedy picking strategy is executed for the two robotic arms, the apple grabbing tasks are dynamically allocated, and the total reward r is evaluated by the collision risk assessment function. t, and then the two agents receive and execute the action instructions a issued by the actor network in real time t , grab the apples with both arms, put the picked apples into the apple collection container, and then execute the end of picking judgment;

[0142] The end of picking judgment is: judge whether the apples in the public workspace have been picked. If not, return to re-execute the number of robotic arms judgment; if the apples have been picked, end all picking work.

[0143] The above embodiments are only used to illustrate rather than limit the technical solutions of the present invention. Although the present invention is described in detail with reference to the above embodiments, those skilled in the art should understand that the present invention can still be modified or replaced by equivalents. Any modification or partial replacement that does not depart from the spirit and scope of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A dual-arm collaborative picking method based on reinforcement learning, using a dual-arm six-degree-of-freedom robot, the dual-arm six-degree-of-freedom robot includes two mechanical arms and an electric control device, the end of each mechanical arm has an end effector for picking apples; the electric control device is an industrial control computer; the characteristics are: The dual-arm six-DOF robot collects the current state information of the target object in real time through the cameras installed on the two robotic arms. t ,s t Including the location information of the target apple, the posture information of the target apple, the distance information of the target apple relative to the two robotic arms, the location information of the obstacle, and the distance information between the two robotic arms and the obstacle; A dual-agent algorithm based on reinforcement learning is stored in the electronic control device; The algorithm structure of the dual-agent algorithm based on reinforcement learning includes an actor network, a critic network, an experience replay buffer, two agents, and a storage medium. t The environment module of the information; two agents correspond to one robot arm respectively; The dual-arm six-degree-of-freedom robot stores joint status information that reflects the current joint status. t , joint t Including joint coordinates, joint angular velocity, joint angular acceleration, joint angle, and the coordinates and rotational posture of the end effector; The two agents read the joint continuously during work t , and in the current joint state information joint t Information about the current environment t The sum of the rewards after the association evaluation is r t ; The two agents receive and execute the action instructions issued by the actor network in real time. t The dual-arm six-degree-of-freedom robot monitors the state of the next moment through the camera t+1 ; Set the current s t 、a t 、r t and t+1 The set of experience tuples is called experience tuples. Let experience tuple N = (s t ,a t ,r t ,s t+1 ) is stored in the experience replay buffer; The critic network randomly samples from the experience replay buffer to calculate each state-action pair (s t -a t )'s advantage function Specifically represents the advantage function estimation of agent i at the kth time step; use the critic network to estimate each state s t The value function V is calculated according to the advantage estimation formula The advantage estimate formula is: in, is the temporal difference error of agent i at the kth time step; γ is the discount factor in the hyperparameters of the generalized advantage estimate, which is used to balance the immediate reward and future reward; λ is the smoothing parameter in the hyperparameters of the generalized advantage estimate, which is used to control the bias-variance trade-off of the advantage estimate; Using advantage function Calculate the objective function of the actor network and guide the actor network to update parameters along the gradient ascent direction by calculating the policy loss gradient; at the same time, calculate the objective function of the critic network and optimize its parameters along the gradient descent direction by calculating the loss gradient; Use the advantage estimate to calculate each pair of s t -a t The objective function formula is: in is the strategy update ratio of agent i; ε represents the proportional coefficient of the clipping range; Clip represents clipping the strategy update ratio within the clipping range; S represents the entropy parameter of the strategy; σ is the parameter used to control the entropy coefficient; The critic network optimization objective is: in is the discounted reward; B represents the size of batch_size, n represents the number of agents, and the value of n is 2; represents the critic network's valuation of the state; Represents the evaluation of the state by the critic network using the old parameters; ∈ represents the scale factor of the clipping range; Clip is used to clip the evaluation range.

2. The dual-arm collaborative picking method based on reinforcement learning according to claim 1, characterized in that: It also includes evaluating the total reward r through the collision risk evaluation function t Methods; The electronic control device obtains the current joint status information of the dual-arm six-degree-of-freedom robot t and the current state information st of the target object, and by using the state transition model P(s t+1 |s t ,a t ) to predict the state st of the next moment after the arms perform action at +1 ; Get s t+1 Afterwards, based on the distance between the two robotic arms and the obstacle and the distance between the two robotic arm joints, it is evaluated whether the two robotic arms will collide with the surrounding objects or collide with each other; t+1 The predicted value and the result of collision detection are used to calculate the expected collision risk Ecrt; Obstacles are defined as: Obstacle{(x min ,y min ,z min )≤(x,y,z)≤(x max ,y max ,z max )}; Assume that the coordinates of the robot joint Linki at time step t are P i =(x i ,y i ,z i ), the closest point between the robot arm and the obstacle is P min =(x o ,y o ,z o ), then the direction vector of joint i of the robot arm is for: The closest point P of the tree branch obstacle min Link with robot arm joint i The vector is: The shortest distance from the obstacle to the robot arm joint i is exist The vertical component of the projection in the direction is d. min , According to the positions of the joints of the two robotic arms in the world coordinate system, the minimum Euclidean distance between the joints is calculated, and the dynamic collision risk between the two robotic arms is evaluated based on the minimum Euclidean distance; Ecrt is the expected collision risk at time step t, Ecrt uses an exponential relationship to evaluate the expected collision risk based on the Euclidean distance between the robot and the obstacle; the ε coefficient is used to control the exponential rate, and the σ coefficient is used to limit the collision risk threshold; Reward sum r t =Rt+Rc; Rt is the task completion reward that reflects the task completion, while Rc is the penalty based on the collision risk assessment, Rc = γ·Ecrt; γ is a coefficient used to adjust the impact of the collision risk assessment on the total reward; Combining the above formulas, we can get the collision risk assessment function, i.e. the total reward r t The calculation formula is:

3. A dual-arm collaborative picking method based on reinforcement learning according to claim 1 or 2, characterized in that: The apple picking space is divided into a left arm workspace and a right arm workspace; the left arm workspace and the right arm workspace have an intersection and the intersection is called a public workspace, which is also called a collaborative workspace; It also includes a multi-objective greedy picking strategy to determine the optimal picking sequence in an independent workspace, cope with the non-uniformity of apple distribution and the heterogeneity of the complexity of the picking operation environment, and coordinate the resource utilization during the two-arm picking process; The first goal of the multi-objective greedy picking strategy is to minimize the picking distance D i , D i That is, the distance between the end effector of the robot arm and the apple; the position of the end effector of the robot arm in the world coordinate system is The position of the i-th apple in the world coordinate system is Picking distance D i The calculation formula is: The second goal of the multi-objective greedy picking strategy is to maximize the efficiency of avoiding obstacles. A dynamic obstacle avoidance strategy f o , in, The obstacle-free degree of the path from the two robot end effectors to the target fruit is evaluated by quantifying the area occupancy of the apple contour in the projection vector direction. The independent workspace and the public workspace are divided into several picking areas. The third goal of the multi-objective greedy picking strategy is to make the robot arm preferentially pick apples in the fruit-dense area. The fruit-dense area optimization strategy ρ is proposed. i ; Among them, ρ i represents the fruit density of the ith region, N i represents the number of fruits in the area, V i Represents the volume of the area; when determining the picking order of multiple areas, press ρ i Arrange the picking order from high to low values.

4. The dual-arm collaborative picking method based on reinforcement learning according to claim 3 is characterized in that: The dual-arm six-degree-of-freedom robot picks apples in the following steps: The first step is to obtain the s required by the environment module t information; The second step is to divide the work area of ​​the current picking environment into independent workspaces corresponding to the left and right arms of the dual-arm six-degree-of-freedom robot and a public workspace corresponding to the left and right arms of the dual-arm six-degree-of-freedom robot. The public workspace is also called a collaborative workspace. The third step is to execute a multi-objective greedy picking strategy for the left arm and the right arm at the same time, driving the left arm and the right arm to pick apples in the corresponding independent workspace at the same time; When executing the multi-objective greedy picking strategy, the fruit-dense area optimization strategy is first executed to determine the priority of the operating area, and the picking order of the operating area is arranged according to the priority; then the picking distance D is calculated in the current operating area according to the picking order of the operating area. i , and then execute the dynamic obstacle avoidance strategy f o , determines the apple picking priority in the current operation area, Then perform the area picking action; the area picking action is: The robot arm and its end effector pick the target apples for current picking, put the picked apples into the apple collection container, and then determine whether the apples in the current operation area have been picked. If the apples in the current operation area have not been picked, the regional picking action is re-executed according to the apple picking priority in the current operation area until the apples in the current operation area are picked. Then, the robot arm returns to re-execute the multi-objective greedy picking strategy in the third step until all apples in the independent workspace corresponding to the current robot arm are picked, and then the fourth step is executed; The fourth step is to prepare to pick apples in the public workspace and perform the number determination of the robotic arms; The number of robotic arms is determined by: determining whether the number of robotic arms that have completed the apple picking task in the independent workspace is greater than 1; If it is not greater than 1, a multi-objective greedy picking strategy is executed for a robotic arm that has completed the apple picking task in the independent workspace. The robot grabs the apples with a single arm, puts the picked apples into the apple collection container, and then executes the end of picking judgment; If it is greater than 1, the multi-objective greedy picking strategy is executed for the two robotic arms, the apple grabbing tasks are dynamically allocated, and the total reward r is evaluated by the collision risk assessment function. t , and then the two agents receive and execute the action instructions a issued by the actor network in real time t , grab the apples with both arms, put the picked apples into the apple collection container, and then execute the end of picking judgment; The end of picking judgment is: judging whether the apples in the public workspace have been picked. If not, returning to re-execute the number of robot arms judgment; If the picking is complete, all picking work is finished.

Images