Multi-agent collaborative global coverage path planning method and system

Abstract

The invention discloses a multi-agent collaborative global coverage path planning method and system, and the method comprises the steps: carrying out the real-time data collection of a litchi orchard environment, and constructing a visual orchard real scene three-dimensional map; constructing a virtual orchard environment based on an orchard real scene three-dimensional map and an OpenAI Gym framework, and obtaining litchi fruit tree position information, a number point set, the number of robots and an initial position; based on a DDQN algorithm, an MA-DDQN algorithm is constructed; the MA-DDQN algorithm is trained, the shortest total path is selected, and path coordinate points of all robots are output; and based on the output global shortest path and the path coordinate point of each robot, driving each robot to complete a picking task along the respective picking path. According to the method, the learning efficiency of the algorithm and the precision of path planning are improved, and meanwhile, by designing the virtual orchard environment and dynamic data input, a plurality of intelligent agents can be trained and cooperatively work in the simulation environment, so that efficient path coverage and picking tasks are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent decision-making, and relates to a multi-agent collaborative global coverage path planning method and system. Background Art

[0002] With the rapid development of the fruit industry and the continuous growth of output, manual picking can no longer meet the needs of industrial development. Research on multi-robot collaborative operation systems that replace manual operations has become an important development direction for improving fruit picking efficiency. Multi-robot collaborative operation can significantly improve picking speed and coverage, while reducing the failure rate and work pressure of a single robot in a complex environment. This collaborative system can not only handle large-scale picking tasks more effectively, but also achieve optimal path planning and resource allocation through intelligent algorithms, thereby maximizing picking efficiency. Although existing coverage path planning methods have improved the coverage efficiency of multiple robots to a certain extent, they still have some limitations in practical applications. For example, these methods usually rely on pre-defined environmental models and are insufficient in coping with dynamic changes and complexity of the environment. In addition, multi-robot systems need to work together in real time in actual operation, which requires the system to be able to flexibly respond to environmental changes and task allocation to overcome uncertainty factors and improve the picking efficiency of multiple agents. Summary of the invention

[0003] The purpose of the present invention is to solve the problem that the multi-robot system in the prior art requires a pre-defined environmental model and is insufficient in coping with the dynamic changes and complexity of the environment; at the same time, the multi-robot system cannot flexibly respond to environmental changes and task allocation in actual operation, and to provide a global coverage path planning method and system for multi-agent collaboration.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A multi-agent collaborative global coverage path planning method, comprising:

[0006] Real-time data collection of litchi orchard environment;

[0007] Based on the collected environmental data, a visualized three-dimensional map of the real scene of the orchard is constructed;

[0008] Based on the real-scene 3D map of the orchard and the OpenAI Gym framework, a virtual orchard environment was constructed to obtain the location information, quantity point set, number of robots, and starting position of litchi trees;

[0009] Based on the DDQN algorithm, construct the MA-DDQN algorithm;

[0010] The MA-DDQN algorithm is trained through the location information of litchi trees, the number of points, the number of robots and the starting position, the shortest total path is selected, and the path coordinate points of each robot are output;

[0011] Based on the output global shortest path and the coordinate points of each robot's path, each robot is driven along its own picking path to complete the picking task.

[0012] A further improvement of the present invention is:

[0013] Furthermore, real-time data collection is performed on the litchi orchard environment, specifically: real-time data of the litchi orchard environment is collected through an unmanned aerial vehicle system; the unmanned aerial vehicle system includes a sensor module, a communication module, a camera module, an automatic path planning module, a data fusion module and a ground monitoring system; the automatic path planning module controls the unmanned aerial vehicle to scan the orchard along a predetermined route through a preset path planning algorithm to ensure the coverage of the unmanned aerial vehicle scanning path; the sensor module includes a lidar and an IMU for accurately measuring three-dimensional terrain and the shape of fruit trees; the communication module is used to monitor the data collection and safety of the unmanned aerial vehicle in real time; the camera module includes a depth camera for monitoring the position change, speed change, direction change of the unmanned aerial vehicle equipment, and calibrating and determining the absolute direction of the equipment; the data fusion module is based on FAST-LIO technology, and real-time integration of sensor data from the lidar and IMU; the ground monitoring system monitors the flight status of the unmanned aerial vehicle in real time, and transmits the monitored data to the automatic path planning module through the communication module.

[0014] Furthermore, based on the collected environmental data, a visualized three-dimensional map of the real orchard scene is constructed. Specifically, based on the three-dimensional point cloud reconstruction technology, the point cloud information is converted into a visualized three-dimensional map; the reconstruction technology is based on an onboard computer combined with the Point Cloud Library software to perform point cloud filtering, feature estimation and three-dimensional reconstruction.

[0015] Furthermore, after constructing the three-dimensional point cloud map, the point cloud data is preprocessed using the filtering algorithm of the Point Cloud Library software, including: point cloud filtering, point cloud registration, smoothing, surface reconstruction and post-processing output; the point cloud filtering includes: voxel grid filtering and statistical filtering, the voxel grid filtering reduces the amount of data while maintaining the overall geometric features; the statistical filtering removes abnormal outliers by calculating the number of neighboring points of each point in the point cloud;

[0016] The point cloud registration uses an Iterative Closest Point algorithm to accurately align point cloud data at different perspectives;

[0017] The smoothing process uses the Moving Least Squares algorithm to locally fit the point cloud surface to reduce the influence of noise and generate a smoother surface point cloud;

[0018] The surface reconstruction generates a continuous three-dimensional surface model based on the normal information of the point cloud through the Poisson reconstruction algorithm of the Point Cloud Library;

[0019] The post-processing output is exported in PLY format to establish a fruit tree distribution coordinate point set.

[0020] Furthermore, the location information of the litchi tree, the number point set, the number of robots and the starting position are obtained, specifically:

[0021] Based on the three-dimensional point cloud data obtained by drone scanning the orchard, the two-dimensional target point position coordinate array P in the orchard is extracted through dimensionality reduction processing. The dimension of the coordinate array P is N×2, where N is the total number of target points, representing the task points in the entire litchi orchard;

[0022] In the virtual environment, multiple autonomous agents are defined, each of which simulates an intelligent robot that performs tasks in the orchard. The initial position of each intelligent robot is represented by array p0, which contains the two-dimensional initial coordinates of all intelligent robots. The current position of the intelligent robot is pos. i (t) is updated at each time step t and is expressed as a two-dimensional coordinate [x, y];

[0023] Each intelligent robot i is assigned a target point set A. i , indicating the target point in the orchard that the intelligent robot needs to visit; the access status of the target point is determined by the variable v j Tracking, 1 means the point has been visited, 0 means it has not been visited.

[0024] Furthermore, based on the DDQN algorithm, the MA-DDQN algorithm is constructed. Specifically, the framework of the MA-DDQN algorithm includes a state space, an action space, an exploration strategy, a reward update algorithm, a reward function, and a priority experience replay mechanism based on TD-error;

[0025] The vector representation of the state space is:

[0026] State=[x1,y1,x2,y2,...,x n ,y n , v1, v2, ..., v m ]

[0027] where [x i ,y i ] represents the current position of the i-th intelligent robot, [vj ] represents the access status of the jth target point;

[0028] The action space is expressed as:

[0029] Action = [a1, a2, ..., a n ]

[0030] where a i Represents the target point index selected by the i-th intelligent robot at the current time step;

[0031] The exploration strategy includes the ε-greedy strategy, where the agent selects the best action in the current Q-value estimate based on the ε value, and the ε decay formula is as follows:

[0032]

[0033] where ∈ t is the exploration rate at the current time step t, ∈ start To set the initial test, ∈ end is the lower limit of the exploration rate, is the decay factor, n is the current cumulative number of training times;

[0034] The reward update algorithm combines the path length scaling factor with the immediate reward to make the Q-value update of each step more globally relevant; its formula is as follows:

[0035]

[0036] Among them, L st Represents the path length from the initial state to the current state, L path is the total length of the path; the scaling factor Used to adjust the amplitude of updating Q value; represents the proportional immediate reward accumulated along the entire path; r t is the immediate reward received at time step t;

[0037] The reward function R includes a negative reward mechanism, and the specific calculation formula is:

[0038]

[0039] Where P current and P target Represent the current position and target position coordinates of the intelligent robot respectively; In addition, illegal actions, i.e. repeated visits or illegal point selections, will be subject to a fixed -1 penalty to ensure compliance during the learning process. The -1 penalty means that if a visit is repeated or an illegal point is selected, the reward will be deducted by one point.

[0040] Furthermore, the TD-error-based priority experience replay mechanism includes Q-value update, TD-error indicator, sampling probability and sampling weight; the Q-value update is used to update the Q network on each agent; the update formula is as follows:

[0041] Q(s,a)↑Q(s,a)+α(r+γQ target (s′, arg max a ′Qlocal(s′,a′))-Q(s,a))

[0042] Where α is the learning rate, which controls the speed of integrating new information during the learning process; γ is the discount factor, which is used to measure the importance of future rewards; Qloca l The network is used to select action a′ in state s′, that is, to determine the best action; the Qtarget network is used to estimate the Q value of state s′ and action a′; s represents the current state; a represents the action taken in state s; r represents the immediate reward after taking action a; s′ represents the next state caused by action a;

[0043] The TD-error indicator is defined as:

[0044] TD-error=|r+γQ target (s′, arg max a ′Qlocal(s′,a′))-Q local (s, a)|

[0045] Where Q local is the network used to select actions (behavioral strategy); Q target is the network used to evaluate the value of actions (target policy); γ is the discount factor; s′ is the resulting state; arg max a 'Q local (s′, a′) represents the l The network determines the optimal action in state s′;

[0046] Based on TD- e rror, applied to the priority experience playback and stored in the priority buffer; the experience priority is set to:

[0047] Priority=(TD-error+∈) α

[0048] Among them, ∈ is a small constant to avoid the priority being zero; α adjusts TD- e The degree of impact of rror on priority;

[0049] The sampling probability comprises the formula:

[0050]

[0051] ∑ k Priorityk is the sum of all experience priorities stored in the experience replay buffer;

[0052] The sampling weight is defined as:

[0053]

[0054] Among them, N is the total number of experiences in the experience replay pool, and β controls the degree of weight adjustment, gradually approaching 1 as training progresses to reduce the impact of weights.

[0055] Furthermore, the training of the MA-DDQN algorithm includes an initialization phase, a training loop, a loop execution strategy, updating the Q network, a completion condition, and the end of the loop;

[0056] The initialization phase includes defining a point set P, where the numerical points in the point set P represent the targets that the intelligent robot wants to visit; initializing the environment LycheeOrchardEnv; setting the number M of intelligent robots; defining the action space A and the state space S; initializing two deep Q networks: one is the Q-main main network and the other is the Q-target target network; initializing a priority experience playback buffer D, setting its capacity and priority parameters, storing and replaying the experience generated during the training process; setting the optimizer O and the learning rate α, and setting the total number of training cycles T;

[0057] The training cycle includes a total of T cycles of training. At the beginning of each cycle, the environment is reset to obtain an initial state s0; the cumulative reward variable and the path length variable are initialized, and the total reward in the current cycle and the distance moved by the intelligent robot are recorded;

[0058] The loop execution strategy includes: t Use the ε-greedy strategy to select the corresponding action a for each intelligent robot; execute the selected action and observe the new state S that the intelligent robot reaches after executing the action t+1 and the reward r brought by this action t ; In each step, calculate the distance the current intelligent robot moves and update the path length; at the same time, update the cumulative reward and set the new reward r t Accumulate to the total reward; calculate the reward in proportion to the state, action, reward and new state (s t , a t , r t ,s t+1 ) is stored in the experience replay buffer;

[0059] The updating of the Q network includes using TD-error to perform priority sampling from the experience playback buffer D, selecting valuable experience to train the network; using the sampled experience to calculate the loss function L(θ), the loss function L(θ) represents the error between the actual reward and the reward predicted by the Q network; based on the loss function, the Q-main network is trained by the back propagation algorithm and its weight θ is updated t+1 ; Every C steps, copy the weights of the Q-main network to the Q-target network to maintain consistency between the main network and the target network;

[0060] The completion condition includes that when all points have been visited, a cycle in the training process ends and enters the next cycle; the cycle end includes repeating the above steps until the set T training cycles are completed; after the training is completed, the shortest total path is selected and the path coordinate points of each robot are output.

[0061] A multi-agent collaborative global coverage path planning system, comprising:

[0062] A collection module, which collects real-time data on the litchi orchard environment;

[0063] A first construction module, wherein the first construction module constructs a visualized three-dimensional map of a real scene of an orchard based on the collected environmental data;

[0064] An acquisition module, which constructs a virtual orchard environment based on a real-scene three-dimensional map of the orchard and an OpenAI Gym framework to obtain the location information, quantity point set, number of robots, and starting position of litchi trees;

[0065] A second building module, wherein the second building module builds an MA-DDQN algorithm based on the DDQN algorithm;

[0066] A training module, wherein the training module trains the MA-DDQN algorithm through the location information of the litchi trees, the quantity point set, the number of robots and the starting position, selects the shortest total path, and outputs the path coordinate points of each robot;

[0067] The driving module drives each robot along its own picking path to complete the picking task based on the output global shortest path and the coordinate points of each robot's path.

[0068] Compared with the prior art, the present invention has the following beneficial effects:

[0069] The present invention uses the collected real-time data and three-dimensional point cloud reconstruction technology to transform point cloud information into a visual three-dimensional map, and adopts the MA-DDQN algorithm, combined with the reward value update method and the priority setting playback mechanism based on the time difference error, to improve the learning efficiency of the algorithm and the accuracy of path planning. At the same time, by designing a virtual orchard environment and dynamic data input, multiple agents can be trained and work together in a simulated environment, thereby achieving efficient path coverage and picking tasks. The present invention optimizes path planning and also copes with the uncertainty and complexity in actual operations through dynamic adjustment strategies; and significantly improves operating efficiency and reduces redundant actions between robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0071] Figure 1 It is a flowchart of the global coverage path planning method of multi-agent collaboration of the present invention;

[0072] Figure 2 It is a structural schematic diagram of the multi-agent collaborative global coverage path planning system of the present invention;

[0073] Figure 3 A schematic diagram of the framework of the multi-agent collaborative global coverage path planning method of the present invention;

[0074] Figure 4 This is the FAST-LIO technical flow chart;

[0075] Figure 5 This is a schematic diagram of the process of building a 3D point cloud map based on FAST-LIO;

[0076] Figure 6 Schematic diagram of the MA-DDQN learning model;

[0077] Figure 7 Schematic diagram of MA-DDQN training process;

[0078] Figure 8 This is a schematic diagram of multi-agent collaborative autonomous picking according to the present invention. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0080] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0081] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0082] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0083] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0084] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0085] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0086] See also Figure 1 The present invention discloses a multi-agent collaborative global coverage path planning method, comprising:

[0087] S101, real-time data collection of litchi orchard environment;

[0088] Real-time data of the litchi orchard environment is collected through an unmanned aerial vehicle system; the unmanned aerial vehicle system includes a sensor module, a communication module, a camera module, an automatic path planning module, a data fusion module and a ground monitoring system; the automatic path planning module controls the unmanned aerial vehicle to scan the orchard along a predetermined route through a preset path planning algorithm to ensure the coverage of the unmanned aerial vehicle scanning path; the sensor module includes a laser radar and an IMU for accurately measuring three-dimensional terrain and the shape of fruit trees; the communication module is used to monitor the data collection and safety of the unmanned aerial vehicle in real time; the camera module includes a depth camera for monitoring the position change, speed change, direction change of the unmanned aerial vehicle equipment, and calibrating and determining the absolute direction of the equipment; the data fusion module is based on FAST-LIO technology, and real-time integration of sensor data from the laser radar and IMU; the ground monitoring system monitors the flight status of the unmanned aerial vehicle in real time, and transmits the monitored data to the automatic path planning module through the communication module.

[0089] S102, constructing a visualized three-dimensional map of the real scene of the orchard based on the collected environmental data;

[0090] Based on 3D point cloud reconstruction technology, point cloud information is converted into a visual 3D map; the reconstruction technology is based on an onboard computer combined with Point Cloud Library software to perform point cloud filtering, feature estimation and 3D reconstruction.

[0091] After constructing the three-dimensional point cloud map, the point cloud data is preprocessed using the filtering algorithm of the Point Cloud Library software, including: point cloud filtering, point cloud registration, smoothing, surface reconstruction and post-processing output; the point cloud filtering includes: voxel grid filtering and statistical filtering, the voxel grid filtering reduces the amount of data while maintaining the overall geometric characteristics; the statistical filtering removes abnormal outliers by calculating the number of neighborhood points of each point in the point cloud; the point cloud registration uses the Iterative Closest Point algorithm to accurately align point cloud data from different perspectives; the smoothing uses the Moving Least Squares algorithm to locally fit the point cloud surface to reduce the impact of noise and generate a smoother surface point cloud; the surface reconstruction uses the Poisson reconstruction algorithm of the Point Cloud Library to generate a continuous three-dimensional surface model based on the normal information of the point cloud; the post-processing output is exported in PLY format to establish a set of fruit tree distribution coordinate points.

[0092] S103, based on the real scene 3D map of the orchard and the OpenAI Gym framework, a virtual orchard environment is constructed to obtain the location information, quantity point set, number of robots and starting position of litchi trees;

[0093] Based on the three-dimensional point cloud data obtained by drone scanning the orchard, the two-dimensional target point position coordinate array P in the orchard is extracted through dimensionality reduction processing. The dimension of the coordinate array P is N×2, where N is the total number of target points, representing the task points in the entire litchi orchard;

[0094] In the virtual environment, multiple autonomous agents are defined, each of which simulates a drone or intelligent robot that performs tasks in the orchard. The initial position of each intelligent robot is represented by array p0, which contains the two-dimensional initial coordinates of all intelligent robots. The current position of the intelligent robot is pos. i (t) is updated at each time step t and is expressed as a two-dimensional coordinate [x, y];

[0095] Each intelligent robot i is assigned a target point set A. i , indicating the target point in the orchard that the intelligent robot needs to visit; the access status of the target point is determined by the variable v i Tracking, 1 means the point has been visited, 0 means it has not been visited.

[0096] S104, constructing a MA-DDQN algorithm based on the DDQN algorithm;

[0097] The framework of the MA-DDQN algorithm includes state space, action space, exploration strategy, reward update algorithm, reward function and TD-based e rror's priority experience playback mechanism;

[0098] The vector representation of the state space is:

[0099] State=[x1,y1,x2,y2,...,x n ,y n , v1, v2, ..., v m ]

[0100] where [x i ,y i ] represents the current position of the i-th intelligent robot, [v j ] represents the access status of the jth target point;

[0101] The action space is expressed as:

[0102] Action = [a1, a2, ..., a n ]

[0103] where ai Represents the target point index selected by the i-th intelligent robot at the current time step;

[0104] The exploration strategy includes the ε-greedy strategy, where the agent selects the best action in the current Q-value estimate based on the ε value, and the ε decay formula is as follows:

[0105]

[0106] where ∈ t is the exploration rate at the current time step t, ∈ start To set the initial test, ∈ end is the lower limit of the exploration rate, is the decay factor, n is the current cumulative number of training times;

[0107] The reward update algorithm combines the path length scaling factor with the immediate reward to make the Q-value update of each step more globally relevant; its formula is as follows:

[0108]

[0109] Among them, L st Represents the path length from the initial state to the current state, L path is the total length of the path; the scaling factor Used to adjust the amplitude of updating Q value; represents the proportional immediate reward accumulated along the entire path; r t is the immediate reward received at time step t;

[0110] The reward function R includes a negative reward mechanism, and the specific calculation formula is:

[0111]

[0112] Where P current and P target Represent the current position and target position coordinates of the intelligent robot respectively; in addition, illegal actions, i.e. repeated visits or illegal point selections, will be subject to a fixed -1 penalty to ensure compliance during the learning process.

[0113] The TD-error-based priority experience replay mechanism includes Q-value update, TD-error indicator, sampling probability and sampling weight; the Q-value update is used to update the Q network on each agent; the update formula is as follows:

[0114] Q(s,a)↑Q(s,a)+α(r+γQ target (s′, arg max a 'Q local (s′, a′))-Q(s, a))

[0115] Where α is the learning rate, which controls the speed of integrating new information during the learning process; γ is the discount factor, which is used to measure the importance of future rewards; Qloca l The network is used to select action a′ in state s′, that is, to determine the best action; the Qtarget network is used to estimate the Q value of state s′ and action a′; s represents the current state; a represents the action taken in state s; r represents the immediate reward after taking action a; s′ represents the next state caused by action a;

[0116] The TD- e The rror indicator is defined as:

[0117] TD-error=|r+γQ target (s′, arg max a 'Q local (s′, a′))-Q local (s, a)|

[0118] Where Q local is the network used to select actions (behavioral strategy); Q target is the network used to evaluate the value of actions (target policy); γ is the discount factor; s′ is the resulting state; arg max a 'Q local (s′, a′) represents the l The network determines the optimal action in state s′;

[0119] Based on TD- e rror, applied to the priority experience playback and stored in the priority buffer; the experience priority is set to:

[0120] Priority=(TD-error+∈) α

[0121] Among them, ∈ is a small constant to avoid the priority being zero; α adjusts the impact of TD-error on the priority;

[0122] The sampling probability comprises the formula:

[0123]

[0124] ∑ k Priorityk is the sum of all experience priorities stored in the experience replay buffer;

[0125] The sampling weight is defined as:

[0126]

[0127] Among them, N is the total number of experiences in the experience replay pool, and β controls the degree of weight adjustment, gradually approaching 1 as training progresses to reduce the impact of weights.

[0128] S105, training the MA-DDQN algorithm through the location information of the litchi trees, the number point set, the number of robots and the starting position, selecting the shortest total path, and outputting the path coordinate points of each robot;

[0129] Training the MA-DDQN algorithm includes the initialization phase, training loop, loop execution strategy, updating the Q network, completion conditions, and loop end;

[0130] The initialization phase includes defining a point set P, where the numerical points in the point set P represent the targets that the intelligent robot wants to visit; initializing the environment LycheeOrchardEnv; setting the number M of intelligent robots; defining the action space A and the state space S; initializing two deep Q networks: one is the Q-main main network and the other is the Q-target target network; initializing a priority experience playback buffer D, setting its capacity and priority parameters, storing and replaying the experience generated during the training process; setting the optimizer O and the learning rate α, and setting the total number of training cycles T;

[0131] The training cycle includes a total of T cycles of training. At the beginning of each cycle, the environment is reset to obtain an initial state s0; the cumulative reward variable and the path length variable are initialized, and the total reward in the current cycle and the distance moved by the intelligent robot are recorded;

[0132] The loop execution strategy includes: t Use the ε-greedy strategy to select the corresponding action a for each intelligent robot; execute the selected action and observe the new state s that the intelligent robot reaches after executing the action t+1 and the reward r brought by this action t ; In each step, calculate the distance the current intelligent robot moves and update the path length; at the same time, update the cumulative reward and set the new reward r t Accumulate to the total reward; calculate the reward proportionally by adding the state, action, reward and new state (s t , a t , r t ,s t+1 ) is stored in the experience replay buffer;

[0133] The updating of the W network includes using TD-error to perform priority sampling from the experience replay buffer D, selecting valuable experience to train the network; using the sampled experience to calculate the loss function L(θ), which represents the error between the actual reward and the reward predicted by the Q network; based on the loss function, training the Q-main network through the back propagation algorithm and updating its weight θ t+1 ; Every C steps, copy the weights of the Q-main network to the Q-target network to maintain consistency between the main network and the target network;

[0134] The completion condition includes that when all points have been visited, a cycle in the training process ends and enters the next cycle; the cycle end includes repeating the above steps until the set T training cycles are completed; after the training is completed, the shortest total path is selected and the path coordinate points of each robot are output.

[0135] S106, based on the output global shortest path and the coordinate points of each robot's path, drive each robot along its own picking path to complete the picking task.

[0136] See also Figure 2 The present invention discloses a multi-agent collaborative global coverage path planning system, comprising:

[0137] A collection module, which collects real-time data on the litchi orchard environment;

[0138] A first construction module, wherein the first construction module constructs a visualized three-dimensional map of a real scene of an orchard based on the collected environmental data;

[0139] An acquisition module, which constructs a virtual orchard environment based on a real-scene three-dimensional map of the orchard and an OpenAI Gym framework to obtain the location information, quantity point set, number of robots, and starting position of litchi trees;

[0140] A second building module, wherein the second building module builds an MA-DDQN algorithm based on the DDQN algorithm;

[0141] A training module, wherein the training module trains the MA-DDQN algorithm through the location information of the litchi trees, the quantity point set, the number of robots and the starting position, selects the shortest total path, and outputs the path coordinate points of each robot;

[0142] A driving module drives each robot along its own picking path to complete the picking task based on the output global shortest path and the coordinate points of each robot's path.

[0143] Example:

[0144] See also Figure 3The present invention provides a multi-agent collaborative global coverage path planning method, comprising:

[0145] Step 1: The drone based on FAST-LIO technology scans the real orchard environment and constructs a three-dimensional point cloud map. According to the geographic information of the orchard and the distribution of fruit trees, the drone scanning path is formulated. The flight parameters, including flight altitude, speed, and scanning range, are input through the ground station. The drone system includes a sensor module, a communication module, a camera module, an automatic path planning module, a data fusion module, and a ground monitoring system; the automatic path planning module controls the drone to scan the orchard along a predetermined route through a preset path planning algorithm to ensure the coverage of the drone scanning path; the sensor module includes a lidar and an IMU for accurately measuring three-dimensional terrain and the shape of fruit trees; the communication module is used to monitor the data collection and safety of the drone in real time; the camera module includes a depth camera for monitoring the position changes, speed changes, direction changes of the drone equipment, and calibrating and determining the absolute direction of the equipment; the ground monitoring system monitors the flight status of the drone in real time, and transmits the monitored data to the automatic path planning module through the communication module. See Figure 4 The data fusion module is based on FAST-LIO technology, which integrates sensor data from LiDAR and IMU in real time; LiDAR provides depth information of the environment, accumulates point clouds to form a set of continuous point clouds, and performs feature extraction to extract plane and edge feature points. The low-frequency inertial data provided by IMU is used to compensate for the attitude changes and acceleration changes during the flight of the drone. Next, forward propagation is performed to predict using IMU data and preliminarily estimate the motion trajectory of the sensor. Errors in sensor motion are compensated by backward propagation to ensure that the LiDAR point cloud is aligned with the motion information of the IMU. Then a submap is extracted from the map and the residual is calculated with the current measurement data, which is the difference between the actual measurement and the prediction. The system updates the state (pose, velocity, etc.) through an iterative Kalman filter and checks whether the filter converges. If converged, the update frequency is reduced. Finally, the map is updated using the extracted feature points to ensure that the new information can be used to further optimize the pose estimation. This process is repeated until the entire orchard is sampled.

[0146] See also Figure 5, Step 2: After the drone completes the orchard scan and generates a preliminary three-dimensional point cloud map, use PCL to further optimize the collected point cloud data. In the collected point cloud data, there may be redundant points and noise points caused by noise interference or sensor errors during the flight of the drone. The filtering algorithm provided by PCL is used to pre-process the point cloud data, mainly including: point cloud filtering, point cloud registration, smoothing, surface reconstruction and post-processing output. Point cloud filtering is due to the fact that in the collected point cloud data, there may be redundant points and noise points caused by noise interference or sensor errors during the flight of the drone. The filtering algorithm provided by PCL is used to pre-process the point cloud data, mainly including: voxel grid filtering, reducing the amount of data while maintaining the overall geometric features; statistical filtering, by calculating the number of neighborhood points of each point in the point cloud, removing abnormal outliers and improving the accuracy of the point cloud. Point cloud registration uses ICP (Iterative Filtering). e The Closest Point algorithm is used to accurately align point cloud data from different perspectives. The smoothing process uses the MLS (Moving Least Squares) algorithm to locally fit the point cloud surface, reduce the impact of noise, and generate a smoother surface point cloud. The surface reconstruction uses the PCL Poisson reconstruction algorithm to generate a continuous three-dimensional surface model based on the normal information of the point cloud. The post-processing output is exported in PLY format to establish the coordinate point set of the fruit tree distribution.

[0147] Step 3: Build a virtual orchard environment

[0148] In this step, based on the actual point cloud map of the litchi orchard, a virtual simulation environment LycheeOrchardEnv was created using the Python high-level programming language and the OpenAI Gym framework. The main steps include: point cloud data processing and target point generation, intelligent agent model design and parameter definition, task allocation and access status tracking. In point cloud data processing and target point generation, the three-dimensional point cloud data obtained by scanning the orchard by drone is used to extract the two-dimensional target point position coordinate array P in the orchard through dimensionality reduction and data processing. Each target point corresponds to a specific position in the orchard. The dimension of the coordinate array P is N×2, where N is the total number of target points, representing the task points in the entire litchi orchard. Each target point needs to be visited and processed in the intelligent robot coverage path planning, such as the location of the fruit tree or the mark point of important terrain. Next, the intelligent agent model design and parameter definition are carried out. In the virtual environment, multiple autonomous agents are defined, each of which simulates a drone or ground operation robot performing a task in the orchard. The initial position of each intelligent robot is represented by the array p0, which contains the two-dimensional initial coordinates of all intelligent robots. The current position of the intelligent robot pos i(t) is updated at each time step t and is represented as a two-dimensional coordinate [x, y]; these intelligent robots move in the orchard through a pre-set algorithm and visit the set of target points A assigned to them i Finally, task allocation and access status tracking are performed. In order to achieve efficient path planning and task allocation, a target point allocation mechanism is designed in the environment. Each intelligent robot i is assigned a target point set A i , indicating the target point in the orchard that the intelligent robot needs to visit; the access status of the target point is determined by the variable v j Tracking is performed, 1 indicates that the point has been visited, and 0 indicates that it has not been visited. This state tracking mechanism ensures that each target point is effectively covered by all intelligent robots during the simulation process, avoiding repeated visits and waste of resources.

[0149] Step 4: Build the MA-DDQN learning model

[0150] In this step, a reinforcement learning model based on MA-DDQN is designed and constructed for the problem of collaborative path planning of multiple intelligent robots. This model optimizes the path planning and task allocation of intelligent agents through collaborative learning of multiple agents in a virtual litchi orchard environment. The framework mainly includes state space, action space, exploration strategy, reward function update, and priority experience replay mechanism based on TD-error, as follows:

[0151] The state space contains the current position of the intelligent robot and the access status of the target point. For each intelligent robot, the two-dimensional coordinates [x, y] are used to represent its current position. Since the entire system involves multiple intelligent robots and multiple target points, if there are n intelligent robots, the position information is represented by 2n floating-point numbers. In addition, the state space also contains a vector indicating whether the target point has been visited. The length of this vector is the number of total target points, and each element is 0 or 1, where 0 means not visited and 1 means visited. Therefore, for a system with n intelligent robots and m target points, the dimension of the state space is 2n+m. The vector representation of the state space is:

[0152] State=[x1,y1,x2,y2,...,x n ,y n , v1, v2, ..., v m ]

[0153] Among them, [x i ,y i ] represents the current position of the i-th intelligent robot, [v j ] represents the access status of the jth target point.

[0154] The action space represents the set of actions that each intelligent robot can perform, including the target point selection assigned to each intelligent robot. Each intelligent robot can select a target point from its unvisited target point set at each time step and move along the optimal path.

[0155] The action space contains a multi-discrete space, where each discrete subspace corresponds to the action selection of an intelligent robot. For n intelligent robots and m target points, the action space is represented as a vector of length n, where each element is an integer from 0 to m-1, representing the index of the target point selected by the intelligent robot. Therefore, the action space can be represented as:

[0156] Action = [a1, a2, ..., a n ]

[0157] where a i Represents the target point index selected by the i-th intelligent robot at the current time step.

[0158] The exploration strategy includes an ε-greedy strategy. As the training progresses, the ε value gradually decays to a low value through exponential decay. The agent selects the optimal action in the current Q value estimation with the low value. The ε decay formula is as follows:

[0159]

[0160] where ∈ t is the exploration rate at the current time step t, ∈ start To set the initial test, ∈ end is the lower limit of the exploration rate (i.e. the lowest exploration rate after attenuation), is the decay factor, n is the current cumulative number of training times, and the decay factor can be changed dynamically.

[0161] The reward update algorithm includes a value function update algorithm that is different from the traditional value update function. The traditional value update function lacks global relevance and cannot effectively provide rewards. Therefore, the reward update algorithm proposes a method of distributing rewards proportionally, combining the proportional factor of the path length with the immediate reward, so that the Q value update of each step is more globally relevant. The formula is as follows:

[0162]

[0163] Among them, L st Represents the path length from the initial state to the current state, L path is the total length of the path. This scaling factor Used to adjust the amplitude of the updated Q value. stis the current path length, so this factor will actually amplify or reduce the update amplitude of the Q value. st Larger (close to L path ), the scale factor is close to 1, and the update amplitude is large; on the contrary, if L st The smaller the value, the larger the scale factor, and the smaller the update. This term represents the proportional immediate reward accumulated along the entire path. t is the immediate reward received at time step t. Normalize the instant rewards by path length to ensure that the distribution of rewards is proportional to the path length. This design can reasonably distribute rewards under different path lengths. A global perspective is introduced so that the Q-value update takes into account the path length from the starting point to the current state. This design allows the model to better understand the importance of different path segments in the overall planning, thereby more effectively transmitting reward information. This ensures that rewards are distributed reasonably according to the length of the path.

[0164] The reward function R includes a negative reward mechanism, and the specific calculation formula is:

[0165]

[0166] Where P current and P target Represent the current position and target position coordinates of the intelligent robot respectively. In addition, illegal actions (i.e. repeated visits or illegal point selections) will be subject to a fixed -1 penalty to ensure compliance during the learning process. The -1 penalty means that if a visit is repeated or an illegal point is selected, the reward will be deducted by one point.

[0167] See also Figure 6 , the TD-error-based priority experience replay mechanism includes Q-value update, TD-error indicator, sampling probability, and sampling weight. Among them, Q-value update is used to update the Q network on each agent, which is the basis of the entire TD-error-based priority experience replay mechanism. Traditional experience replay stores the experience data generated by the interaction between the agent and the environment through a fixed-size buffer (called the experience replay pool). Each data point is usually represented as a four-element experience component (s, a, r, s′), where: s represents the current state; a represents the action taken in state s; r represents the immediate reward obtained after taking action a; s′ represents the next state caused by action a. In each training iteration, a batch of samples are randomly selected from the experience pool with the same probability P1(i) for learning.

[0168]

[0169] Where N is the total number of experiences in the replay buffer.

[0170] The extracted samples are used to update the Q network of the agent, and the Q value update formula is as follows

[0171] Q(s,a)↑Q(s,a)+α(r+γQ target (s′, arg max a ′Qlocal(s′,a′))-Q(s,a))

[0172] Where α is the learning rate, which controls the speed of integrating new information during the learning process; γ is the discount factor, which is used to measure the importance of future rewards; Qloca l The network is used to select action a′ in state s′, that is, to determine the best action; Q target The network is used to estimate the Q-values ​​of states s′ and actions a′.

[0173] The TD- e The rror indicator is defined as:

[0174] TD-error=|r+γQ target (s′, arg max a 'Q local (s′, a′))-Q local (s, a)|

[0175] Where Q local is the network used to select actions (behavioral strategy); Q target is the network used to evaluate the value of actions (target policy); γ is the discount factor; r is the reward; s′ is the resulting state; arg max a 'Q local (s′, a′) represents the local The network determines the optimal action in state s′.

[0176] TD-error is usually expressed as the difference between the predicted Q-value of the current state-action pair and the observed reward plus the estimated Q-value of the future state. This difference measures the deviation between the actual gain and the expectation, indicating the accuracy of the current strategy and the potential for improvement. Based on the calculated TD-error, it is applied to the priority experience replay and stored in the priority buffer. The priority of the experience is set as:

[0177] Priority=(TD-error+∈) α

[0178] Among them, ∈ is a small constant to avoid the priority being zero; α adjusts the impact of TD-error on the priority.

[0179] The sampling probability contains a key formula

[0180]

[0181] >Priority k is the sum of all experience priorities stored in the experience replay buffer. P2(i)

[0182] is the improved sampling probability; P1(i) is the conventional sampling probability.

[0183] When training the agent, we sample from the experience replay buffer in a priority-weighted manner so that the TD- e Experiences with higher rror are sampled more frequently. Such a sampling strategy ensures that the agent can focus more on learning those experiences that are most critical to improving its performance.

[0184] The sampling weights are due to the fact that the sampling is no longer uniform, and the learning updates need to be adjusted by importance sampling weights (IS-Weights) to prevent the introduction of bias. These weights are defined as:

[0185]

[0186] Among them, N is the total number of experiences in the experience replay pool, and β controls the degree of weight adjustment, gradually approaching 1 as training progresses to reduce the impact of weights.

[0187] Step 5: Establish the MA-DDQN training process

[0188] See also Figure 7 and Figure Figure 8 In this step, a training process based on MA-DDQN (multi-agent dual-depth Q network) is constructed to optimize the collaborative path planning and task execution of the multi-agent system in the orchard environment. The training process includes the initialization phase, training loop, execution strategy, Q network update and loop end. At the beginning of training, the access target of the intelligent robot and the virtual environment LycheeOrchardEnv are first set, and the following initial parameters are configured: the initialization phase includes defining a point set P, which represents the target to be visited by the intelligent robot; initializing the environment LycheeOrchardEnv; setting the number of intelligent robots M; defining the action space A and the state space S; initializing two deep Q networks: one is Q-main (main network) and the other is Q-target (target network); initializing a priority experience playback buffer D, setting its capacity and priority parameters to store and playback the experience generated during training; setting the optimizer O and the learning rate α, and setting the total number of training cycles T.

[0189] The training cycle includes a total of T training cycles. At the beginning of each cycle, the environment is reset to obtain an initial state s0; the cumulative reward variable and the path length variable are initialized to record the total reward in the current cycle and the distance moved by the intelligent robot.

[0190] The loop execution strategy includes: t Use the ε-greedy strategy to select the corresponding action a for each intelligent robot; execute the selected action and observe the new state s that the intelligent robot reaches after executing the action t+1 and the reward r brought by this action t ; In each step, calculate the distance the current intelligent robot moves and update the path length. At the same time, update the cumulative reward and set the new reward r t Accumulate to the total reward; calculate the reward in proportion, and the reward may be distributed according to the cooperation between multiple agents or the completion of independent tasks; add the state, action, reward and new state (s t , a t , r t , S t+1 ) is stored in the experience replay buffer for use in subsequent training.

[0191] The updating of the Q network includes using TD-error to perform priority sampling from the experience replay buffer, selecting valuable experience to train the network; using the sampled experience, calculating the loss function L(θ), which represents the error between the actual reward and the reward predicted by the Q network. Based on the loss function, the Q-main network is trained by the back propagation algorithm and its weight θ is updated t+1 Every C steps, the weights of the Q-main network are copied to the Q-target network to maintain consistency between the main and target networks.

[0192] The completion condition includes that when all points are visited or some other conditions are met, a cycle in the training process ends and enters the next cycle.

[0193] The end of the loop shown includes repeating the above steps until the set T training cycles are completed.

[0194] After completing the training cycle, the path coordinates of each intelligent robot with the shortest total path are output.

[0195] Step 6: Implementing the path in a real scene

[0196] Based on the coordinates, control instructions are then used to drive the real robot to complete collaborative picking tasks in complex and changing environments.

[0197] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-agent collaborative global coverage path planning method, characterized in that: include: Real-time data collection of litchi orchard environment; Based on the collected environmental data, a visualized three-dimensional map of the real scene of the orchard is constructed; Based on the real-scene 3D map of the orchard and the OpenAI Gym framework, a virtual orchard environment was constructed to obtain the location information, quantity point set, number of robots, and starting position of litchi trees; Based on the DDQN algorithm, construct the MA-DDQN algorithm; The MA-DDQN algorithm is trained through the location information of litchi trees, the number of points, the number of robots and the starting position, the shortest total path is selected, and the path coordinate points of each robot are output; Based on the output global shortest path and the coordinate points of each robot's path, each robot is driven along its own picking path to complete the picking task.

2. The multi-agent collaborative global coverage path planning method according to claim 1, characterized in that: The real-time data collection of the litchi orchard environment is specifically as follows: real-time data of the litchi orchard environment is collected through an unmanned aerial vehicle system; the unmanned aerial vehicle system includes a sensor module, a communication module, a camera module, an automatic path planning module, a data fusion module and a ground monitoring system; the automatic path planning module controls the unmanned aerial vehicle to scan the orchard along a predetermined route through a preset path planning algorithm to ensure the coverage of the unmanned aerial vehicle scanning path; the sensor module includes a laser radar and an IMU for accurately measuring three-dimensional terrain and the shape of fruit trees; the communication module is used to monitor the data collection and safety of the unmanned aerial vehicle in real time; the camera module includes a depth camera for monitoring the position change, speed change, direction change of the unmanned aerial vehicle equipment, and calibrating and determining the absolute direction of the equipment; the data fusion module is based on FAST-LIO technology, and real-time integration of sensor data from the laser radar and the IMU; the ground monitoring system monitors the flight status of the unmanned aerial vehicle in real time, and transmits the monitored data to the automatic path planning module through the communication module.

3. The multi-agent collaborative global coverage path planning method according to claim 2, characterized in that: The method of constructing a visualized three-dimensional map of the real scene of the orchard based on the collected environmental data is specifically: based on the three-dimensional point cloud reconstruction technology, the point cloud information is converted into a visualized three-dimensional map; the reconstruction technology is based on an onboard computer combined with Point Cloud Library software to perform point cloud filtering, feature estimation and three-dimensional reconstruction.

4. The multi-agent collaborative global coverage path planning method according to claim 1, characterized in that: After constructing the 3D point cloud map, the point cloud data is preprocessed using the filtering algorithm of the Point Cloud Library software, including: point cloud filtering, point cloud registration, smoothing, surface reconstruction and post-processing output; the point cloud filtering includes: voxel grid filtering and statistical filtering. The voxel grid filtering reduces the amount of data while maintaining the overall geometric features; the statistical filtering removes abnormal outliers by calculating the number of neighboring points of each point in the point cloud; The point cloud registration uses an Iterative Closest Point algorithm to accurately align point cloud data at different perspectives; The smoothing process uses the Moving Least Squares algorithm to locally fit the point cloud surface to reduce the influence of noise and generate a smoother surface point cloud; The surface reconstruction generates a continuous three-dimensional surface model based on the normal information of the point cloud through the Poisson reconstruction algorithm of the Point Cloud Library; The post-processing output is exported in PLY format to establish a fruit tree distribution coordinate point set.

5. The multi-agent collaborative global coverage path planning method according to claim 4, characterized in that: The acquisition of the location information of the litchi tree, the quantity point set, the number of robots and the starting position is specifically as follows: Based on the three-dimensional point cloud data obtained by drone scanning the orchard, the two-dimensional target point position coordinate array P in the orchard is extracted through dimensionality reduction processing. The dimension of the coordinate array P is N×2, where N is the total number of target points, representing the task points in the entire litchi orchard; In the virtual environment, multiple autonomous agents are defined, each of which simulates an intelligent robot that performs tasks in the orchard. The initial position of each intelligent robot is represented by array p0, which contains the two-dimensional initial coordinates of all intelligent robots. The current position of the intelligent robot is pos. i (t) is updated at each time step t and is expressed as a two-dimensional coordinate [x, y]; Each intelligent robot i is assigned a target point set A. i , indicating the target point in the orchard that the intelligent robot needs to visit; the access status of the target point is determined by the variable v j Tracking, 1 means the point has been visited, 0 means it has not been visited.

6. The multi-agent collaborative global coverage path planning method according to claim 5, characterized in that: The MA-DDQN algorithm is constructed based on the DDQN algorithm, specifically: the framework of the MA-DDQN algorithm includes a state space, an action space, an exploration strategy, a reward update algorithm, a reward function, and a priority experience replay mechanism based on TD-error; The vector representation of the state space is: State=[x1,y1,x2,y2,...,x n ,y n ,v1,v2,...,v m ] where [x i ,y i ] represents the current position of the i-th intelligent robot, [v j ] represents the access status of the jth target point; The action space is expressed as: Action=[a1,a2,…,a n ] where a i Represents the target point index selected by the i-th intelligent robot at the current time step; The exploration strategy includes the ε-greedy strategy, where the agent selects the best action in the current Q-value estimate based on the ε value, and the ε decay formula is as follows: where ∈ t is the exploration rate at the current time step t, ∈ start To set the initial test, ∈ end is the lower limit of the exploration rate, is the decay factor, n is the current cumulative number of training times; The reward update algorithm combines the scaling factor of the path length with the immediate reward, making the Q-value update of each step more globally relevant; The formula is as follows: Among them, L st Represents the path length from the initial state to the current state, L path is the total length of the path; Scale Factor Used to adjust the amplitude of updating Q value; represents the proportional immediate reward accumulated along the entire path; r t is the immediate reward received at time step t; The reward function R includes a negative reward mechanism, and the specific calculation formula is: Where P current and P target Represent the current position and target position coordinates of the intelligent robot respectively; in addition, illegal actions, i.e. repeated visits or illegal point selections, will be subject to a fixed -1 penalty; the -1 penalty means that if a visit is repeated or an illegal point is selected, the reward will be deducted by one point.

7. The multi-agent collaborative global coverage path planning method according to claim 6, characterized in that: The TD-error-based priority experience replay mechanism includes Q value update, TD-error index, sampling probability and sampling weight; wherein the Q value update is used to update the Q network on each agent; the update formula is as follows: Q(s,a)←Q(s,a)+α(r+γQ target (s',argmax a' Q local (s',a'))-Q(s,a)) Where α is the learning rate, which controls the speed of integrating new information during the learning process; γ is the discount factor, which is used to measure the importance of future rewards; Q local The network is used to select action a' in state s', that is, to determine the best action; Q target The network is used to estimate the Q value of state s' and action a'; s represents the current state; a represents the action taken in state s; r represents the immediate reward after taking action a; s' represents the next state caused by action a; The TD-error indicator is defined as: TD-error=|r+γQ target (s',argmax a' Q local (s',a'))-Q local (s,a)| Where Q local is the network used to select actions, i.e., the behavior strategy; Q target is the network used to evaluate the value of the action, i.e., the target strategy; γ is the discount factor; s' is the result state; argmax a' Q local (s',a') represents the local The network determines the best action in state s'; Based on TD-error, it is applied to the priority experience playback and stored in the priority buffer; the priority of the experience is set to: Priority=(TD-error+∈) α Among them, ∈ is a small constant to avoid the priority being zero; α adjusts the impact of TD-error on the priority; The sampling probability comprises the formula: Σ k Priority k is the sum of all experience priorities stored in the experience replay buffer; The sampling weight is defined as: Among them, N is the total number of experiences in the experience replay pool, and β controls the degree of weight adjustment, gradually approaching 1 as training progresses to reduce the impact of weights.

8. The multi-agent collaborative global coverage path planning method according to claim 1, characterized in that: The training of the MA-DDQN algorithm includes an initialization phase, a training loop, a loop execution strategy, an update of the Q network, a completion condition, and a loop end; The initialization phase includes defining a point set P, where the numerical points in the point set P represent the targets that the intelligent robot wants to visit; initializing the environment LucjeePrcjardEmv; setting the number M of intelligent robots; defining the action space A and the state space S; initializing two deep Q networks: one is the Q-main main network and the other is the Q-target target network; initializing a priority experience playback buffer D, setting its capacity and priority parameters, storing and replaying the experience generated during the training process; setting the optimizer O and the learning rate α, and setting the total number of training cycles T; The training cycle includes a total of T cycles of training. At the beginning of each cycle, the environment is reset to obtain an initial state s0; the cumulative reward variable and the path length variable are initialized, and the total reward in the current cycle and the distance moved by the intelligent robot are recorded; The loop execution strategy includes: t Use the ε-greedy strategy to select the corresponding action a for each intelligent robot; execute the selected action and observe the new state s that the intelligent robot reaches after executing the action t+1 and the reward r brought by this action t ; In each step, calculate the distance the current intelligent robot moves and update the path length; at the same time, update the cumulative reward and set the new reward r t Accumulate to the total reward; calculate the reward proportionally by adding the state, action, reward and new state (s t ,a t ,r t ,s t+1 ) is stored in the experience replay buffer; The updating of the Q network includes using TD-error to perform priority sampling from the experience playback buffer D, selecting valuable experience to train the network; using the sampled experience to calculate the loss function L(θ), the loss function L(θ) represents the error between the actual reward and the reward predicted by the Q network; based on the loss function, the Q-main network is trained by the back propagation algorithm and its weight θ is updated t+1 ; Every C steps, copy the weights of the Q-main network to the Q-target network to maintain consistency between the main network and the target network; The completion condition includes that when all points have been visited, a cycle in the training process ends and enters the next cycle; the cycle end includes repeating the above steps until the set T training cycles are completed; after the training is completed, the shortest total path is selected and the path coordinate points of each robot are output.

9. A multi-agent collaborative global coverage path planning system, characterized in that: include: A collection module, which collects real-time data on the litchi orchard environment; A first construction module, wherein the first construction module constructs a visualized three-dimensional map of a real scene of an orchard based on the collected environmental data; An acquisition module, which constructs a virtual orchard environment based on a real-scene three-dimensional map of the orchard and an OpenAI Gym framework to obtain the location information, quantity point set, number of robots, and starting position of litchi trees; A second building module, wherein the second building module builds an MA-DDQN algorithm based on the DDQN algorithm; A training module, wherein the training module trains the MA-DDQN algorithm through the location information of the litchi trees, the quantity point set, the number of robots and the starting position, selects the shortest total path, and outputs the path coordinate points of each robot; A driving module drives each robot along its own picking path to complete the picking task based on the output global shortest path and the coordinate points of each robot's path.

Images