Multi-agent collaborative collision avoidance picking method based on digital twin and reinforcement learning

Abstract

The present invention discloses a multi-agent collaborative collision avoidance picking method based on digital twin and deep reinforcement learning, including establishing a digital orchard database and constructing a twin orchard scene; based on the Markov decision process framework, the virtual robot autonomous collision avoidance picking strategy problem is modeled and a simulation system is built, and the state space and action space are set; based on the artificial potential field method, a reward function is designed, and an APF-MADPPO learning model is established in combination with a multi-agent distributed proximal strategy optimization algorithm; an APF-MADPPO network model and training process are established, and a training method is set; based on transfer learning and inference learning, the optimal picking path is planned, and the real robot is driven by control instructions to complete the automatic collision avoidance picking task in a complex environment. The present invention can realize that in a real-time changing environment, multiple agents can work together and autonomously avoid collisions, thereby completing the automatic picking task.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent decision-making, and specifically relates to a multi-agent collaborative collision avoidance picking method based on digital twins and reinforcement learning. Background Art

[0002] Fruit picking and harvesting are key links in the orchard production process. Traditional picking methods are mostly done manually and cannot meet market demand. Agricultural picking robots can effectively replace manual labor to complete picking tasks and play a vital role in the field of smart agriculture. At present, research on picking robots can be divided into four parts: mechanical structure design, visual perception and positioning, decision-making and planning, and execution control. In a dynamic and unstructured environment, how to make the picking robot quickly adapt to the environment and accurately make the correct picking strategy is a key problem that needs to be solved urgently. At present, related research mainly focuses on the decision-making and path planning of a single agent, or the decision-making and path planning of multiple agents in a known environment and a simple scene. For example, Xiong Juntao et al. published "Virtual Robot Picking Path Obstacle Avoidance Planning Based on Deep Reinforcement Learning" (Journal of the Chinese Society of Agricultural Machinery, 2020(51), S2:1-10), which uses the DPPO algorithm to quickly plan the trajectory of the virtual robot and realize the obstacle avoidance path planning of the picking robot; but this method is only applicable to a single agent environment, and the simulated orchard environment is not time-varying. In the real-world field environment, the unknown picking target point and the randomness of obstacles make the behavior of agricultural picking robots more uncertain and complex. In order to further improve the production efficiency of agricultural products, it is of great significance to build a multi-agent collaborative collision avoidance picking path planning system. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a multi-agent collaborative collision avoidance picking method based on digital twins and deep reinforcement learning, which can enable multiple agents to work collaboratively and avoid collisions autonomously in a real-time changing environment, thereby completing the automatic picking task.

[0004] The purpose of the present invention is achieved through the following technical solutions:

[0005] A multi-agent collaborative collision avoidance picking method based on digital twins and deep reinforcement learning includes the following steps:

[0006] (1) The data acquisition system collects real orchard environmental information in real time and transmits it to the cloud server, combining data analysis and mining technology with agronomic knowledge to establish a digital orchard database;

[0007] (2) Establish data communication between the cloud server and the virtual simulation platform, use the real-time data of the cloud server to drive the three-dimensional model in the virtual simulation platform for intelligent simulation scheduling, build a twin orchard scene; and establish a virtual picking robot model;

[0008] (3) Based on the Markov decision process framework, the environment modeling and simulation system of the virtual robot autonomous collision avoidance picking strategy problem are built, and the state space and action space are set;

[0009] (4) Designing a reward function based on the artificial potential field method and establishing an APF-MADPPO learning model in combination with a multi-agent distributed proximal strategy optimization algorithm; the reward function includes a picking point guidance function, an obstacle collision penalty function, a collision penalty function between robots, and a time penalty function;

[0010] (5) Establish the APF-MADPPO network model and training process;

[0011] (6) Setting the training method, including setting the termination condition, configuring the training parameters, setting the training process, and using the APF-MADPPO learning model to perform multi-agent picking collision avoidance strategy learning and training;

[0012] (7) Based on the transfer learning method, the training result model is transferred to a dynamic and unstructured environment to perform reasoning learning on picking collision avoidance path planning and plan an optimal picking path;

[0013] (8) Using the optimal picking path as a reference, the real robot is driven by control instructions to complete the automatic collision avoidance picking task in a complex environment.

[0014] In step (1), the data acquisition system includes a processor chip module, a LoRa communication module, a camera and a sensor module, which are used to collect real-time status information of a real orchard; the sensor module includes an air temperature and humidity sensor, a soil moisture sensor, a soil pH sensor, a carbon dioxide concentration sensor and a light intensity sensor; the cloud server includes real-time data acquisition of the real orchard, data query of the database, and client response and feedback; the database includes digital orchard operation status data storage.

[0015] In step (2), the establishment of data communication between the cloud server and the virtual simulation platform refers to the establishment of interactive communication between the virtual simulation platform (such as the Unity3D virtual simulation platform) and the server (such as the Alibaba Cloud server and the Tencent Cloud server) through the Socket communication mechanism. The real-time data drive includes data transmission and data processing; the data transmission refers to the process in which the Unity3D virtual simulation platform accesses the target server through http; the data processing refers to the process in which the Unity3D virtual simulation platform obtains the real-time data of the orchard from the server and completes the data processing process through the coroutine mechanism. The construction of the twin orchard scene refers to setting relevant state parameters according to the real orchard scene information (including soil, fruit trees, etc.), and the state parameters correspond to and match the model attributes of the digital twin orchard. By obtaining the response signal from the server, the real-time data is transmitted to the simulation environment using the coroutine mechanism, and the intelligent simulation scheduling algorithm is used to drive the virtual platform models to construct the twin orchard scene.

[0016] In step (3), the environment modeling is performed for the problem of multi-agent collaborative collision avoidance picking strategy in a complex dynamic field environment, and the Markov decision process is used to describe the random decision process of the interaction between the agent and the environment; the random decision process is defined as a four-tuple (S t , A t , R t , S t+1 ), including:

[0017] S t : is the state of the agent at time t, constituting the system state space;

[0018] A t : is the action taken by the agent at time t, constituting the action space of the system;

[0019] R t : is the reward value obtained by the agent at time t, which constitutes the reward function of the system;

[0020] S t+1 : is the state of the agent at time t+1.

[0021] In step (4), the APF-MADPPO learning model includes the design of state space, action space and reward function; the state space includes the spatial position of the robot end effector Picking point spatial position P goal , obstacle spatial position P obs , the relative position of the robot end effector and the picking point The relative position of the center point of each rotation axis of the robot and the obstacle The distance between the robot end effector and the picking point The distance between the center point of each rotation axis of the robot and the obstacle Distances between the robots' rotation axes Recorded as Wherein i represents the number of intelligent agents (i.e., picking robots); the action space includes the changes in the rotation angles of each joint axis of each robot.

[0022] The reward function includes: picking point guidance function R guide , obstacle avoidance function R obs , collision penalty function R between robots am And the time penalty function R time ; The calculation method of the system cumulative reward value R is as follows:

[0023] R=R guide +R obs +R arm +R time

[0024] Among them, the picking point guidance function R guide The calculation method is to calculate the end effector of each robot arm and the target picking point position P goal =(x o ,y o , z o ) i=1.2.3...n is the number of agents, and the minimum distance at state t is obtained when When the distance between the target and the reward is gradually reduced, a low reward is given, and the low reward coefficient is k1, otherwise a penalty is given; when ... When it is 0, the maximum reward k2 is given and the round ends, as shown in the following formula:

[0025]

[0026]

[0027]

[0028] Among them, the obstacle avoidance function R obs The calculation method is to calculate the lateral distance between each robot rotation axis and the obstacle position when When it is greater than the lateral distance L of the rotation axis and the radius R of the obstacle warning area, the penalty function does not work; When it is greater than the sum of the obstacle radius r and the lateral distance L and less than the sum of the lateral distance L and the obstacle warning area radius R, a low penalty is given. The penalty value is inversely proportional to the distance. k3 is expressed as the low penalty coefficient. When it is less than the sum of the obstacle radius r and the lateral distance L, a maximum penalty k4 is imposed and the round ends; the specific details are shown in the following formula:

[0029]

[0030] Among them, the collision penalty function R between robots arm The calculation method is to determine the spatial set of robots Spatial aggregation with neighboring robots Is there an intersection? If there is an intersection, it means that the adjacent robots have collided. At this time, the maximum penalty k5 is given and the round ends. If there is no intersection, it means that the adjacent robots have not collided. At this time, the penalty function does not have any effect and the penalty value is 0. The specific formula is as follows:

[0031]

[0032] Among them, the time penalty function R time The calculation method is based on the distance from each agent to the target picking point in the initial state. Set it up, k6 represents the time penalty coefficient; the specific formula is as follows:

[0033]

[0034] In step (5), the establishment of the APF-MADPPO network model includes establishing an Actor network, a Critic network and an experience library; wherein the establishment of the Actor network is used to select an action strategy, and its input is camera image data, end effector position, obstacle position, target picking point position, the closest distance between the end effector and the target picking point, the distance between the obstacle and each rotation axis of the robot, and the distance between each rotation axis of the robot, and its output is the rotation angle information of each rotation axis of the robot; the establishment of the Critic network is used to evaluate the quality of the action strategy in the current state, and its input is the acquisition of camera image data, end effector position, obstacle position, target picking point position, the closest distance between the end effector and the target picking point, the distance between the obstacle and each rotation axis of the robot, and the distance between each rotation axis of the robot, and its output is the accumulated reward value obtained; the experience library is used to store exploration data.

[0035] In step (6), a human-machine collaborative training and learning method can be further adopted to design an interactive control interface, and the robot picking collision avoidance strategy learning process can be intervened through the control interface in combination with human active behavior, which can be used in dynamic scenes where collisions frequently occur. At the same time, the position information and number of picking target points and obstacles can be set through the control interface to conduct purposeful autonomous collision avoidance picking simulation experiments.

[0036] A multi-agent collaborative collision avoidance picking system based on digital twins and deep reinforcement learning, including a twin orchard scene simulation environment; an interactive control interface; a training learning module, a reasoning learning module, and a human-computer collaboration module.

[0037] The twin orchard scene simulation environment is a data acquisition system that collects real orchard information in real time and transmits it to a cloud server, builds a digital orchard database based on agronomic knowledge, and then uses data-driven technology to intelligently schedule the virtual platform three-dimensional model to build a twin orchard scene simulation environment; at the same time, it is used to receive real-time training process control effects for visual display, providing users with a more intuitive collision avoidance control effect for picking robots.

[0038] The interactive control interface is used to provide users with a customized interface to facilitate them to conduct purposeful simulation experiments, and also to view relevant historical data in real time, providing reference value for the collision avoidance path planning of the real picking robot.

[0039] The training learning module refers to using the ML-Agents plug-in to establish interactive communication between deep reinforcement learning and the simulation environment, and then using the APF-MAPPO learning model to learn the picking strategy of multiple agents.

[0040] The inference learning module refers to migrating the training result model to a dynamic and changeable environment to perform collision avoidance path planning for the picking robot based on the transfer learning method.

[0041] The human-machine collaboration module refers to combining human's active behavior through an interactive control interface during the training and learning process to improve the convergence speed of the strategy model, or setting the target point and obstacle position and number through the control interface to perform purposeful picking and collision avoidance path planning.

[0042] Compared with the prior art, the present invention has the following advantages and effects:

[0043] (1) The present invention combines the real-time data of the cloud server and uses the intelligent simulation scheduling algorithm to drive the three-dimensional model in the Unity3D virtual simulation platform to construct a time-varying simulation learning environment that is more in line with the requirements of the real-world wild orchard scene.

[0044] (2) Based on the Markov decision process framework, the present invention establishes an autonomous collision avoidance strategy environment model for a virtual picking robot and an APF-MADPPO multi-agent collaborative picking strategy learning model to effectively evaluate the behavior strategy of the agent and improve the convergence speed of the algorithm.

[0045] (3) The multi-agent collaborative collision avoidance picking system of the present invention includes a simulation environment and an interactive control interface to improve the operability and flexibility of the system; by setting a training learning module, a reasoning learning module and a human-machine collaboration module, users can improve the strategy learning speed through human-machine collaboration; at the same time, target points and obstacles can be set for purposeful autonomous picking and collision avoidance planning; in the reasoning learning module, the training result model can be transferred to different environments to quickly perform collision avoidance planning for picking robots, providing important basis and reference value for actual field picking robot operations.

[0046] (4) The present invention obtains the optimal picking path as a reference based on simulation experiments, and then drives the real picking robot through control instructions to quickly and stably avoid randomly appearing obstacles, reach the picking target point position, and complete the automatic collision avoidance picking task. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a framework diagram of the multi-agent collaborative collision avoidance picking method based on digital twins and deep reinforcement learning of the present invention.

[0048] Figure 2 This is an overall architecture diagram of the data-driven method of the present invention.

[0049] Figure 3 Schematic diagram of the Markov decision process of the present invention.

[0050] Figure 4 This is a framework diagram of optimizing the deep reinforcement learning reward function based on the artificial potential field method of the present invention.

[0051] Figure 5 This is the APF-MADPPO network model structure diagram of the present invention.

[0052] Figure 6 This is a multi-threaded processing framework diagram of the APF-MADPPO network model of the present invention.

[0053] Figure 7 This is a design diagram of the system interactive control interface of the present invention.

[0054] Figure 8 This is a diagram of the reinforcement learning training architecture based on ML-Agents of the present invention. DETAILED DESCRIPTION

[0055] In order to facilitate the understanding of the present invention, the present invention will be described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be pointed out that, for those skilled in the art, without departing from the inventive concept, the present invention can also make some variations and improvements, which all belong to the protection scope of the present invention.

[0056] Example 1

[0057] A multi-agent collaborative collision avoidance picking method based on digital twins and deep reinforcement learning, such as Figure 1 As shown, the following steps are included:

[0058] Step 1: Build a virtual orchard simulation environment based on digital twins, such as Figure 2 As shown. Real-world orchard environmental information is collected in real time through data collection devices such as sensors and cameras and transmitted to the cloud server. Combined with the real-time data and agronomic knowledge of the cloud server, the knowledge base and database of the digital orchard are constructed. The virtual scene model is constructed using 3DMax modeling software, and the model is optimized using LOD technology. The optimized three-dimensional model is imported into the Unity3D virtual simulation platform. At the same time, a real-time data-driven architecture is built on the Unity simulation platform, and the data communication between the cloud server and the simulation platform is established using the C# high-level programming language. The simulation platform accesses the target server through Http and obtains the server's response signal and returns data through the coroutine mechanism. The data-driven technology is used to drive the corresponding models or special effects (rainy days, foggy days, snowy days, etc.) in the simulation environment according to the real-time data of the real orchard in the server, so that each part of the model in the virtual environment maintains the same real-time state as the corresponding physical object in the real orchard. At the same time, the environmental state of the real orchard in a certain period of time can also be reproduced in combination with historical data;

[0059] Step 2: Establish a robot picking collision avoidance planning problem model based on Markov decision process, such as Figure 3 The Markov decision process is a mathematical model of sequential decision making, which is usually applicable to continuous action space. The present invention regards the robot collision avoidance picking process as a Markov decision process for problem modeling. For example, the state S of the picking robot at time t is t , execute action A according to the strategy t , and enter the state S at the next moment t+1 , and get reward R t Collected as a four-tuple and forming a set (S, A, R, S * ), find an optimal strategy π(a|s) to maximize the cumulative reward from time t. Where γ∈[0,1] represents the discount factor, r t Represents the reward value obtained by the agent at time t.

[0060]

[0061] Step 3: Build a multi-agent collaborative collision avoidance picking system based on deep reinforcement learning method, and establish APF-MADPPO learning model by combining multi-agent distributed proximal strategy optimization algorithm, such as Figure 4 As shown. Combined with the effective state information that can be obtained in the actual picking environment, the position observation variables of the end effector, picking point and obstacle of the picking robot are reasonably added. At the same time, the relative distance observation variables between the end effector and the picking point and the relative distance observation variables between each rotation axis and the obstacle are added; according to the physical structure characteristics of the actual picking robot, the action state and motion range of each rotation axis of the virtual robot arm are set; based on the idea of ​​artificial potential field method and collision cone collision detection method, a multi-agent collaborative picking avoidance reward function is constructed.

[0062] The reward function includes the picking point guidance reward function R guide , obstacle avoidance penalty function R obs , collision penalty function R between robots arm And the time penalty function R time Among them, both the reward coefficient and the penalty coefficient can be adjusted, and usually the reward coefficient is greater than the penalty coefficient.

[0063] The calculation method of the system cumulative reward value R is as follows:

[0064] R=R guide +R obs +R arm +R time

[0065] The picking point guides the reward function R guide The calculation method is to calculate the end effector of each robot arm and the target picking point position P goal =(x o ,y o , z o ) i=1.2.3...n is the number of agents, and the minimum distance at state t is obtained when When the distance between the target and the reward is gradually reduced, a low reward is given, and the low reward coefficient is k1, otherwise a penalty is given; when ... When it is 0, the maximum reward k2 is given and the round ends. The specific formula is as follows.

[0066]

[0067]

[0068]

[0069] Usually the lateral distance of the robot arm is smaller than the longitudinal distance. Therefore, in the obstacle avoidance penalty function R obs The calculation method only needs to consider the lateral distance between the rotation axis and the obstacle position. Assume that the radius of the obstacle is r, the radius of the obstacle warning area is R, and the horizontal distance between the rotation axes of the robot arm is L. When , the penalty function does not work; when When , a low penalty is given, and the penalty value is inversely proportional to the distance. k3 is represented by the low penalty coefficient; when When , the maximum penalty k4 is given and the round ends. The specific formula is as follows.

[0070]

[0071] The collision penalty function between robots is R arm The calculation method is to determine the spatial set of robots Spatial aggregation with neighboring robots Is there an intersection? If there is an intersection, it means that the adjacent robots have collided. At this time, the maximum penalty k5 is given and the round ends. If there is no intersection, it means that the adjacent robots have not collided. At this time, the penalty function does not have any effect and the penalty value is 0. The specific formula is as follows:

[0072]

[0073] The time penalty function R time The calculation method is based on the distance from each agent to the target picking point in the initial state. Set it up, k6 represents the time penalty coefficient. The specific formula is as follows:

[0074]

[0075] Step 4: Establish the APF-MADPPO algorithm network model, such as Figure 5As shown. The Actor network is responsible for outputting actions. During the training process, the system's camera will capture images of the simulated environment, and extract image feature values ​​through convolutional layers, pooling layers, and normalization. The image feature values, end effector position, obstacle position, target picking point position, and the closest distance between the end effector and the target picking point, the distance between the obstacle and each rotation axis of the robot, and the distance between each robot's rotation axis are used as environmental state information and input into the Actor network. According to the normal distribution parameters, an action is sampled to interact with the environment and generate a corresponding reward value. At the same time, the action probability ratio of the new and old strategies is used to represent the weight of the new strategy, limit the update amplitude of the new strategy, and update its parameters through back propagation. Finally, the output is the rotation angle of each rotation axis of each robot; the Critic network is mainly responsible for evaluating the quality of the action strategy. The input of the neural network is environmental state information. By continuously obtaining the state and reward value, the network parameters are back propagated and updated, so that the evaluation value is getting closer and closer to the reward value; the experience library is mainly responsible for storing exploration data. When the storage conditions are met, the Critic network starts to update the network parameters.

[0076] Step 5: Analyze the MADPPO algorithm network model training process. The algorithm contains a global network and multiple local networks, using centralized learning and decentralized execution. Figure 6 As shown in the figure. First, multiple local networks interact with the environment independently at the same time, collect data information, and then calculate the policy gradient and pass it into the shared gradient area for storage. When a certain number is met, the global network obtains the gradient information from the shared gradient area to uniformly update the Actor and Critic parameters and feed it back to each local network, and the cycle continues until the maximum number of training steps is reached.

[0077] Step 6: Build the simulation running environment and interactive control interface, such as Figure 7 As shown. Basic button functions such as start, run, pause and reset are designed to enhance the operability of the system. Users can set relevant parameters through the system control interface, such as the number and position of obstacles, the picking target position and the number of agents to conduct purposeful picking path collision avoidance planning simulation experiments. At the same time, relevant data user interfaces can also be set to facilitate fruit farmers to view the planned routes completed under different historical conditions, providing important basis and useful value for real picking robots.

[0078] Step 7: Set termination conditions, including round termination and training completion termination. Round termination conditions are when the robot reaches the target picking point, collides with an obstacle, the robot's movement exceeds the preset range, and fails to reach the target picking point for a long time; training completion termination conditions refer to stopping training when the training reaches the maximum number of training steps.

[0079] Step 8: Training process and results. Build a digital orchard and picking robot model based on the Unity virtual platform, set the number of random obstacles, use an orthogonal camera to evenly render the physical objects in the simulation environment, and improve the efficiency of state space feature information extraction. Use the ML-Agents plug-in to establish communication between the simulation environment and the Python API through the Socket communication mechanism, configure the training process parameters, and use the APF-MAPPO algorithm to train the robot to learn the picking strategy, such as Figure 8 After the training is completed, the training data results can be obtained and viewed through Tensorboard. At the same time, the system will integrate the learned strategy into a network model based on TensorFlow and automatically ensure it in the project.

[0080] Step 9: Result model verification and reasoning learning. The training result model can be used to safely, quickly and stably control the virtual robot to reach the random picking point. At the same time, the training result model is transferred to different environments for reasoning learning in combination with the transfer learning method, so that the robot can quickly and stably perform purposeful picking and collision avoidance path planning in a dynamic environment, providing a reference for the actual picking robot.

[0081] Step 10: Based on the autonomous picking path collision avoidance planning simulation experiment based on reasoning learning, the optimal picking path is obtained as a reference, and then the real robot is driven by control instructions to complete the automatic collision avoidance picking task in a complex and changeable environment.

[0082] The above descriptions are merely embodiments of the present invention, but the implementation methods of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention are equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A multi-agent collaborative collision avoidance picking method based on digital twins and deep reinforcement learning, characterized in that The steps include: (1) The data collection system collects real orchard environmental information in real time and transmits it to the cloud server, combining data analysis and mining technology with agronomic knowledge to establish a digital orchard database; (2) Establish data communication between the cloud server and the virtual simulation platform, use the real-time data of the cloud server to drive the three-dimensional model in the virtual simulation platform for intelligent simulation scheduling, build a twin orchard scene, and establish a virtual picking robot model; (3) Based on the Markov decision process framework, the environment modeling and simulation system of the virtual robot autonomous collision avoidance picking strategy problem are built to set the state space and action space; (4) Designing a reward function based on the artificial potential field method and establishing an APF-MADPPO learning model in combination with a multi-agent distributed proximal strategy optimization algorithm; the reward function includes a picking point guidance function, an obstacle collision penalty function, a collision penalty function between robots, and a time penalty function; The APF-MADPPO learning model includes state space, action space and reward function design; the state space includes the spatial position of the robot end effector , Spatial location of picking point , obstacle spatial position , the relative position of the robot end effector and the picking point , the relative position of the center point of each rotation axis of the robot and the obstacle , the distance between the robot end effector and the picking point , the distance between the center point of each rotation axis of the robot and the obstacle , the distance between the various rotation axes of the robots , denoted as ,in It is expressed as the number of intelligent agents, i.e., picking robots; The action space includes the changes in the rotation angles of each joint axis of each robot; The reward function includes: picking point guidance function , obstacle avoidance function , Collision penalty function between robots And the time penalty function ; The calculation method of the system cumulative reward value R is as follows: Among them, the picking point guidance function The calculation method is to calculate the end effector of each robot arm The target picking point The distance between , is the number of agents, and obtains the minimum distance at state t ;when When it gradually decreases, a low reward is given, and the low reward coefficient is , otherwise a penalty will be given; when the target distance When it is 0, the maximum reward is given And end this round, as shown in the following formula: Among them, the obstacle avoidance function The calculation method is to calculate the lateral distance between each robot rotation axis and the obstacle position ;when When it is greater than the lateral distance L of the rotation axis and the radius R of the obstacle warning area, the penalty function does not work; When the distance is greater than the sum of the obstacle radius r and the lateral distance L and less than the sum of the lateral distance L and the obstacle warning area radius R, a low penalty is given, and the penalty value is inversely proportional to the distance. Represents a low penalty coefficient; when When it is less than the sum of the obstacle radius r and the lateral distance L, the maximum penalty is given And end this round; as shown in the following formula: Among them, the collision penalty function between robots The calculation method is to determine the spatial set of robots Spatial aggregation with neighboring robots Is there an intersection? If there is an intersection, it means that the adjacent robots have collided, and the maximum penalty is given at this time. And the round ends; if there is no intersection, it means that there is no collision between adjacent robots. At this time, the penalty function does not have any effect and the penalty value is 0; the specific formula is as follows: Among them, the time penalty function The calculation method is based on the distance from each agent to the target picking point in the initial state. To set up, It is expressed as a time penalty coefficient; specifically as shown in the following formula: ； (5) Establish the APF-MADPPO network model and training process; (6) Setting the training method, including setting the termination condition, configuring the training parameters, setting the training process, and using the APF-MADPPO learning model for multi-agent picking and collision avoidance strategy learning and training; (7) Based on the transfer learning method, the training result model is transferred to a dynamic and unstructured environment to perform reasoning learning on picking collision avoidance path planning and plan an optimal picking path; (8) Using the optimal picking path as a reference, the real robot is driven by control instructions to complete the automatic collision avoidance picking task in a complex environment.

2. The multi-agent collaborative collision avoidance picking method according to claim 1 is characterized in that: In step (1), the data acquisition system includes a processor chip module, a LoRa communication module, a camera and a sensor module for collecting real-time status information of a real orchard; the sensor module includes an air temperature and humidity sensor, a soil moisture sensor, a soil pH sensor, a carbon dioxide concentration sensor and a light intensity sensor; the cloud server includes real-time data acquisition of the real orchard, data query of the database and client response and feedback; the database includes digital orchard operation status data storage.

3. The multi-agent collaborative collision avoidance picking method according to claim 1, characterized in that: In step (2), constructing the twin orchard scene refers to setting relevant state parameters according to the real orchard scene information, wherein the state parameters correspond to the model attributes of the digital twin orchard, and by obtaining the response signal from the server, the coroutine mechanism is used to transmit the real-time data to the simulation environment, and the intelligent simulation scheduling algorithm is used to drive the virtual platform models to construct the twin orchard scene.

4. The multi-agent collaborative collision avoidance picking method according to claim 1, characterized in that: The system is to model the environment of multi-agent collaborative collision avoidance picking strategy in a complex and dynamic wild environment, and use the Markov decision process to describe the random decision-making process of the interaction between the agent and the environment; the random decision-making process is defined as a four-tuple , specifically including: : is the state of the agent at time t, constituting the system state space; : is the action taken by the agent at time t, constituting the action space of the system; : is the reward value obtained by the agent at time t, which constitutes the reward function of the system; : is the state of the agent at time t+1.

5. The multi-agent collaborative collision avoidance picking method according to claim 1, characterized in that: In step (5), the establishment of the APF-MADPPO network model includes establishing an Actor network, a Critic network and an experience library; wherein the establishment of the Actor network is used to select an action strategy, and its input is camera image data, end effector position, obstacle position, target picking point position, the closest distance between the end effector and the target picking point, the distance between the obstacle and each rotation axis of the robot, and the distance between each rotation axis of the robot, and its output is the rotation angle information of each rotation axis of the robot; the establishment of the Critic network is used to evaluate the quality of the action strategy in the current state, and its input is the acquisition of camera image data, end effector position, obstacle position, target picking point position, the closest distance between the end effector and the target picking point, the distance between the obstacle and each rotation axis of the robot, and the distance between each rotation axis of the robot, and its output is the accumulated reward value obtained; the experience library is used to store exploration data.

6. The multi-agent collaborative collision avoidance picking method according to claim 1, characterized in that: In step (6), a human-machine collaborative training and learning method is adopted to design an interactive control interface. The control interface is used to intervene in the robot's picking collision avoidance strategy learning process in combination with human active behavior. This is used in dynamic scenarios where collisions frequently occur. At the same time, the position information and number of picking target points and obstacles can be set through the control interface to conduct purposeful autonomous collision avoidance picking simulation experiments.

Images