Path planning method of agricultural robot

Abstract

The invention discloses a path planning method for an agricultural robot, and the method comprises the steps: building a reinforcement learning interaction environment based on a prior map, and segmenting a global environment through an intermediate target point, and designing a DQN algorithm network architecture; designing a DQN algorithm reward function based on a forward bow-shaped operation mode of the agricultural robot, wherein the reward function is composed of a center line area reward function and a target point reward function; setting a DQN algorithm action space and an exploration strategy by combining a neighborhood exploration mode of an A * algorithm; a priority experience playback mechanism is adopted to access and transfer sample data to design an evaluation network and a target network, and path planning and decision making of the agricultural robot are carried out. And a forward bow-shaped obstacle avoidance path is planned between the starting point and the ending point in the scanned environment, so that the agricultural robot can perform autonomous navigation tracking control. The line feed operation position can be automatically adjusted after the operation environment is switched, and the problem that the environment self-adaptive adjustment capability is insufficient in the automatic navigation process is solved.

Description

Technical Field

[0001] The present application relates to the technical field of path planning, and in particular to a path planning method for an agricultural robot. Background Art

[0002] At present, in the agricultural planting and harvesting industry chain, the main working method is still based on manual driving operations. Not only is the work intensity high and the work efficiency low, but as the working time accumulates, the safety risks will gradually increase. Therefore, the solution to the problem of autonomous navigation operations of agricultural robots is imminent.

[0003] At present, the autonomous navigation of agricultural autonomous robots can be roughly divided into two categories when performing path planning: one is the global static path planning method. This type of algorithm is mostly used for global path planning in static environments, but the planned path has too many broken line segments and the broken line angles are too large. It is necessary to integrate other improved algorithms to perform secondary optimization and splicing on the path before it can be used. At the same time, after switching the working environment, it is necessary to re-search the working path and select the path optimization method according to the environment to smooth the path. This type of method has weak adaptability to the environment, poor applicability, and low path planning efficiency.

[0004] Therefore, how to improve the adaptive adjustment of robots to the environment during automatic navigation in agricultural operations is a technical problem that needs to be urgently solved in this field. Summary of the invention

[0005] In order to solve the above technical problems, this application proposes the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a path planning method for an agricultural robot, comprising:

[0007] Construct a reinforcement learning interactive environment based on the prior map, and use the intermediate target points to segment the global environment to design the DQN algorithm network architecture;

[0008] The DQN algorithm reward function is designed based on the forward bow-shaped operation mode of the agricultural robot, and the reward function consists of a midline area reward function and a target point reward function;

[0009] Based on the forward bow-shaped operation mode of the agricultural robot and the neighborhood exploration mode of the A* algorithm, the DQN algorithm action space and exploration strategy are set;

[0010] The priority experience replay mechanism is used to access and transfer sample data to design the evaluation network and target network for path planning and decision-making of the agricultural robot.

[0011] In a possible implementation, the reinforcement learning interactive environment is constructed based on the prior map, and the DQN algorithm network architecture is designed by using the intermediate target points to segment the global environment, including:

[0012] Use SLAM algorithm to scan the global environment to build a priori map, and build a deep reinforcement learning interactive environment based on the priori map. The environment is divided into crop row area and operation row area. The crop row area is an inaccessible area and the operation behavior is a passable area. At the same time, the crop row is expanded and fitted, and the left and right expansion widths are half of the robot width.

[0013] Combined with the forward bow-shaped operation mode of the agricultural robot, an intermediate target point is added between the starting point and the end point to cut the overall complex environment into short-range simple paths;

[0014] The deep reinforcement learning DQN algorithm is used to build the algorithm network architecture according to the working environment and working mode.

[0015] In a possible implementation, the forward bow-shaped operation mode of the agricultural robot is combined, an intermediate target point is added between the starting point and the end point, and the overall complex environment is cut into short-range simple paths, including:

[0016] Take the midline coordinates of the walking row and the crop row as the horizontal coordinates of the intermediate target point;

[0017] And take the vertical coordinate of the front end of the crop row and move it up by λd col Set the ordinate of the intermediate target point, move the odd-numbered crop rows up, and move the even-numbered crop rows down, and divide the complex overall path into multiple simple paths, where λ is the translation factor of the ordinate of the intermediate target point, d col It is the longitudinal distance between the crop row and the working boundary.

[0018] In a possible implementation, in order to prevent the robot from missing the intermediate target point, the robot is considered to have found the target point as long as it enters the target point area. The target point area is centered on the target point and is r area is the radius, and the area radius calculation formula is as follows:

[0019] r area =0.25d tr

[0020] Among them, r area is the area radius, d tr The spacing between walking rows;

[0021] The calculation formula of the ordinate of the intermediate target point is as follows:

[0022] y tp =y ep ±λd col

[0023] Among them, y tp is the ordinate of the intermediate target point, y ep It is the vertical coordinate of the front end point of the crop row.

[0024] In a possible implementation, the DQN algorithm reward function is designed based on the forward bow-shaped operation mode of the agricultural robot, including:

[0025] Construct a midline area reward function based on the distribution of crop rows and work rows to ensure that the robot drives in the midline area as much as possible when working between rows;

[0026] A target point reward function is constructed based on the intermediate target points. The navigation area is divided by the target points. Each area consists of a starting point and an end point, and the target point area is named after the area starting point. The starting point of the first target point area is the environment starting point, and the end point of the nth target point area is the environment end point.

[0027] In a possible implementation, the midline area reward function is constructed according to the distribution law of crop rows and operation rows to ensure that the robot drives in the midline area as much as possible when operating between rows, including:

[0028] Taking the centerline spacing between working rows as the reference, when the minimum value d in the laser radar scanning distance min In the midline area [0.4d tr , 0.6d tr ], the reward function is positive, and the closer the robot is to the walking center line, the higher the reward value; conversely, when the robot is driving outside the center line area, the reward value is negative, and the closer to the obstacles on both sides, the greater the penalty value. The calculation formula of the center line area reward function is as follows

[0029]

[0030] Among them, r mid is the midline area reward function, a is the midline reward factor, d tr is the row width, d min is the minimum scanning distance.

[0031] In a possible implementation, the Euclidean distance between the start and end points of a region is defined as the region distance, denoted by d reg , the Euclidean distance from the robot's current position to the end point of the area is the current distance, denoted as d cur , using the area distance d reg Distance d from current cur The square of the difference is used to calculate the target point reward function value, and b is used as the regional reward factor. If d reg -d cur >0, indicating that the current distance is less than the regional distance, that is, the robot is moving towards the end of the region. At this time, b>0 indicates that the target point reward value is positive; if d reg -d cur<0, indicating that the current distance is greater than the regional distance, that is, the robot is moving in the opposite direction. At this time, b<0 means that the target point reward value is negative; the target point reward function calculation formula is as follows:

[0032] r dis =b*(d cur -d reg ) 2

[0033] Among them, r dis is the target point reward function, b is the target point reward factor, d cur is the current distance, d reg is the regional distance.

[0034] In a possible implementation, the forward bow-shaped operation mode of the agricultural robot combined with the neighborhood exploration mode of the A* algorithm is used to set the DQN algorithm action space and exploration strategy, including:

[0035] The 8-neighborhood exploration mode of the traditional A* algorithm is changed to the forward 5-neighborhood exploration. The action space of the DQN algorithm is specified to be 5-dimensional. The current position is used as the reference center. According to the vehicle head direction of the robot's current position, action 2 is specified to be directly in front of the vehicle head. Actions 1 and 3 are respectively the left and right 45° turning directions, and actions 0 and 4 are respectively the left and right 90° turning directions. The robot's forward movement and steering are controlled by linear velocity and angular velocity. The formula for calculating the angular velocity by the action sequence number is as follows:

[0036]

[0037] Among them, w is the robot angular velocity, as_size is the length of the action space, a i is the action number, max_w is the maximum angular velocity;

[0038] Based on the idea of ​​greedy strategy, the action selection strategy is improved, and the greedy strategy exploration factor ε is set to dynamically adjust to adapt to the adjustment of exploration actions at different stages of training.

[0039] In a possible implementation, the action selection strategy is improved based on the greedy strategy idea, and a dynamic adjustment of the greedy strategy exploration factor ε is set to adapt to the adjustment of the exploration action at different stages of training, including:

[0040] In the early stage of training, ε is set to 1. At this stage, we need to quickly explore the unknown environment to accumulate experience, select the action with the maximum Q value, reduce invalid exploration and reward oscillation, and shorten the convergence process.

[0041] As the number of training times increases, the agent becomes more and more familiar with the environment, and accumulates more and more optimal exploration experience. At this time, the exploration factor ε is continuously reduced, and the way of selecting actions gradually changes from the maximum Q value action to the action that can obtain the maximum reward among the known actions, so that the agent can quickly understand the unknown environment and find the optimal path more easily, simplify the target point search process, shorten the navigation time, and dynamically adjust the exploration factor ε formula as follows:

[0042]

[0043] Among them, ε is the exploration factor, ε0 is the exploration factor per round, and ε min To explore the minimum value of the factor, ε max is the maximum value of the exploration factor, up_iter is the network update frequency, and ε_decay is the number of decay steps.

[0044] In a possible implementation, the method of using a priority experience replay mechanism to access and transfer sample data to design an evaluation network and a target network to perform path planning and decision-making for an agricultural robot includes:

[0045] (1) Generate the evaluation network (Eval Net) and target network (Target Net) through the DQN structure. The parameters of the evaluation network and target network are ω and ω respectively. - ;

[0046] (2) Based on the current state s, the greedy strategy is used to select action a, and then interact with the environment to obtain reward r, the next state s_, and the flag bit done to indicate whether the round is over, and form a transfer sample (s t , a t , r t ,s t+1 );

[0047] (3) For each transfer sample (s t , a t , r t ,s t+1 ) is marked with a priority mark δ t , according to the current Q value Q(s t , a t ;ω) and target Q value y t The difference between the TD error and the sample sampling priority p is calculated t , the priority calculation formula is as follows:

[0048]

[0049] Among them, δ t is the sample priority label, p tis the sample priority, τ is the sampling guarantee factor to ensure that the sample priority will not be 0;

[0050] According to the sampling priority p t Calculate the sampling probability P(t) of the sample transition, and use α as the weight factor to weigh uniform sampling and greedy sampling. If a transition has just been collected, it means that the data has not been used yet, then δ t Set it to the maximum value, give it the highest weight, and the sampling probability P(t) is calculated as follows:

[0051]

[0052] Among them, P(t) is the sample sampling probability, and α is the weight factor that weighs uniform sampling and greedy sampling;

[0053] (4) To prevent overfitting and speed up convergence, a bias mechanism is introduced. When calculating the loss function value during the training phase, a bias weight θ is added based on the TD error of the sample. t To improve the loss function, the bias weight θ t Characterizes the learning efficiency of the agent, θ t The larger the value is, the greater the gap between the predicted value and the TD target value at this time step is, and the greater the update amount of the agent is, indicating that the learning efficiency here is higher. At the same time, the importance sampling factor β is used to represent the priority experience replay weight. The deviation weight calculation formula is as follows:

[0054] θ t =(N*P(t)) -β (β∈(0,1),t∈[0,N])

[0055] Among them, t is the sample, N is the number of samples in the experience pool, β is the importance sampling factor, and the distribution range of β is (0, 1). The improved loss function calculation formula is as follows:

[0056]

[0057] Among them, m is the number of samples, θ t is the bias weight;

[0058] (5) To uniformly transfer the sample dimensions and reduce sample differences, the bias weight needs to be normalized. The normalized bias weight calculation formula is as follows:

[0059]

[0060] (6) Repeat (2), (3), (4) and (5) to evaluate the priority of each collected sample. At the same time, recalculate the sample priority of each training sample after each round of training until the training is completed.

[0061] In the embodiment of the present application, a forward bow-shaped obstacle avoidance path is planned between the starting point and the end point in the scanned environment for the agricultural robot to perform autonomous navigation tracking control. The line-changing operation position can be adjusted automatically after the operating environment is switched, solving the problem of insufficient adaptive adjustment ability to the environment during automatic navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A schematic diagram of a flow chart of a path planning method for an agricultural robot provided in an embodiment of the present application;

[0063] Figure 2 A schematic diagram of the operation direction of an odd-even tree row provided in an embodiment of the present application;

[0064] Figure 3 A schematic diagram of the global environment for segmenting intermediate target points provided in an embodiment of the present application;

[0065] Figure 4 A diagram of the DQN algorithm network architecture provided in an embodiment of the present application;

[0066] Figure 5 A target point area division diagram provided in an embodiment of the present application;

[0067] Figure 6 A schematic diagram of a five-neighborhood search action space provided in an embodiment of the present application;

[0068] Figure 7 This is a flowchart of the improved DQN algorithm training provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The present solution is described below in conjunction with the accompanying drawings and specific implementation methods.

[0070] See also Figure 1 The path planning method of the agricultural robot provided in this embodiment includes:

[0071] S101, build a reinforcement learning interactive environment based on the prior map, and use the intermediate target points to segment the global environment to design the DQN algorithm network architecture.

[0072] The SLAM algorithm is used to scan the global environment to build a prior map. A deep reinforcement learning interactive environment is constructed based on the prior map. The environment is divided into crop row area and work row area. The crop row area is an inaccessible area and the work behavior is a passable area. At the same time, the crop row is expanded and fitted, and the left and right expansion width is half of the robot width.

[0073] Combined with the forward bow-shaped operation mode of the agricultural robot, an intermediate target point is added between the starting point and the end point to cut the overall complex environment into a short-range simple path. The rules for setting the intermediate target point are as follows: the coordinates of the walking row and the midline of the crop row are taken as the horizontal coordinates of the intermediate target point, and the vertical coordinate of the end point of the crop row is shifted up by λd col Set the ordinate of the intermediate target point (odd-numbered crop rows move up, even-numbered crop rows move down), where λ is the translation factor of the intermediate target point ordinate. When the intermediate target point of the walking row is translated, λ is 0.25, and when the intermediate target point of the crop row is translated, λ is 0.5. In this way, the complex overall path is divided into multiple simple paths. At the same time, in order to prevent the robot from missing the intermediate target point, it is stipulated that the target point is found as long as the robot enters the target point area. The target point area is centered on the target point and is r area is the radius, and the area radius calculation formula is as follows:

[0074] r area =0.25d tr

[0075] Among them, r area is the area radius, d tr The spacing between walking rows;

[0076] The calculation formula of the ordinate of the intermediate target point is as follows:

[0077] y tp =y ep ±λd col

[0078] Among them, y tp is the ordinate of the intermediate target point, y ep is the ordinate of the end point of the crop row, λ is the translation factor of the ordinate of the intermediate target point, d col It is the longitudinal distance between the crop row and the working boundary.

[0079] join Figure 2 and Figure 3 Schematic diagram of the global environment for the operation direction of odd and even tree rows and the segmentation of intermediate target points, respectively.

[0080] The deep reinforcement learning DQN algorithm is adopted to build the algorithm network architecture according to the working environment and working mode. The input layer is set with 28 neurons, and the current posture, yaw angle, distance from the current position to the target point, and minimum obstacle distance of the agricultural robot are used as the input information of the neural network; two hidden layers are set in the middle, each with 256 neurons; the output layer is set with 5 neurons, and the output information is the Q value corresponding to the optional action. The network architecture diagram is shown in the figure Figure 4 shown.

[0081] S102, designing the DQN algorithm reward function based on the forward bow-shaped operation mode of the agricultural robot, wherein the reward function is composed of a midline area reward function and a target point reward function.

[0082] According to the distribution of crop rows and working rows, a reward function for the centerline area is constructed to ensure that the robot drives in the centerline area as much as possible when working between rows. Taking the centerline spacing between working rows as the benchmark, when the minimum value d in the laser radar scanning distance min In the midline area [0.4d tr , 0.6d tr ], the reward function is positive, and the closer the robot is to the walking center line, the higher the reward value; conversely, when the robot is driving outside the center line area, the reward value is negative, and the closer to the obstacles on both sides, the greater the penalty value. The calculation formula of the center line area reward function is as follows

[0083]

[0084] Among them, r mid is the midline area reward function, a is the midline reward factor, d tr is the row width, d min is the minimum scanning distance;

[0085] The target point reward function is constructed based on the intermediate target point. The navigation area is divided by the target point. Each area consists of a starting point and an end point. The target point area is named after the starting point of the area. The starting point of the first target point area is the environment starting point, and the end point of the nth target point area is the environment end point. The Euclidean distance between the starting point and the end point of the area is defined as the area distance, denoted by d reg , the Euclidean distance from the robot's current position to the end point of the area is the current distance, denoted as d cur , using the area distance d reg Distance d from current cur The square of the difference is used to calculate the target point reward function value, and b is used as the regional reward factor. If d reg -d cur >0, indicating that the current distance is less than the regional distance, that is, the robot is moving towards the end of the region. At this time, b>0 indicates that the target point reward value is positive; if d reg -d cur<0, indicating that the current distance is greater than the regional distance, that is, the robot is moving in the opposite direction. At this time, b<0 means that the target point reward value is negative. The target point reward function calculation formula is as follows:

[0086] r dis =b*(d cur -d reg ) 2

[0087] Among them, r dis is the target point reward function, b is the target point reward factor, d cur is the current distance, d reg is the area distance, and the target point area division diagram is as follows Figure 5 shown.

[0088] S103, based on the forward bow-shaped operation mode of the agricultural robot and combined with the neighborhood exploration mode of the A* algorithm, set the DQN algorithm action space and exploration strategy.

[0089] The 8-neighborhood exploration mode of the traditional A* algorithm is changed to the forward 5-neighborhood exploration. The action space of the DQN algorithm is specified to be 5-dimensional. The current position is used as the reference center. According to the vehicle head direction of the robot's current position, action 2 is specified to be directly in front of the vehicle head. Actions 1 and 3 are respectively the left and right 45° turning directions, and actions 0 and 4 are respectively the left and right 90° turning directions. The robot's forward movement and steering are controlled by linear velocity and angular velocity. The formula for calculating the angular velocity by the action sequence number is as follows:

[0090]

[0091] Among them, w is the robot angular velocity, as_size is the length of the action space, a i is the action number, and max_w is the maximum angular velocity. The five-neighborhood search action space is as follows Figure 6 shown.

[0092] Based on the greedy strategy idea, the action selection strategy is improved, and the greedy strategy exploration factor ε is set to dynamically adjust to adapt to the adjustment of exploration actions at different stages of training. The details are as follows: In the early stage of training, ε takes a value of 1. At this stage, it is necessary to quickly explore the unknown environment to accumulate experience, select the action with the maximum Q value as much as possible, reduce invalid exploration and reward oscillation, and shorten the convergence process; as the number of training times increases, the intelligent agent has a better understanding of the environment, and the accumulated optimal exploration experience is increasing. At this time, the exploration factor ε should be continuously reduced, and the way of selecting actions is gradually changed from the maximum Q value action to the action that can obtain the maximum reward among the known actions, so that the intelligent agent can quickly understand the unknown environment and find the optimal path more easily, simplify the target point search process, and shorten the navigation time. The formula for dynamically adjusting the exploration factor ε is as follows

[0093]

[0094] Among them, ε is the exploration factor, ε0 is the exploration factor per round, and ε min To explore the minimum value of the factor, ε max is the maximum value of the exploration factor, up_iter is the network update frequency, and ε_decay is the number of decay steps;

[0095] Improvement ideas for the DQN algorithm: Scan the working environment to build a priori map, set working rows (passable areas) and crop rows (impassable areas) in the map, fit the crop rows and add path boundaries to form a forward bow-shaped working corridor for the agricultural robot; use the intermediate target point to segment the overall environment, and divide the complex global environment into short-range simple navigation paths; improve the DQN algorithm reward function according to the forward bow-shaped working method of the agricultural robot. The reward function consists of two parts, namely the midline area reward function and the target point reward function; change the 8-neighborhood search of the traditional A* algorithm to a forward 5-neighborhood search, and use these 5 action directions to form the DQN algorithm action space; use the greedy strategy idea to improve the DQN algorithm exploration strategy. By dynamically adjusting the exploration factor, the environment is quickly explored in the early stage of training to accumulate experience. In the later stage of training, the action that can obtain the maximum reward among the known actions is selected to plan the optimal path, thereby improving the DQN algorithm.

[0096] S104, using the priority experience replay mechanism to access the transferred sample data to design the evaluation network and the target network, and perform path planning and decision-making for the agricultural robot.

[0097] (1) Generate the evaluation network (Eval Net) and target network (Target Net) through the DQN structure. The parameters of the evaluation network and target network are ω and ω respectively. - ;

[0098] (2) Based on the current state s, the greedy strategy is used to select action a, and then interact with the environment to obtain reward r, the next state s_, and the flag bit done to indicate whether the round is over, and form a transfer sample (s t , a t , r t ,s t+1 );

[0099] (3) For each transfer sample (s t , a t , r t ,s t+1 ) is marked with a priority mark δ t , according to the current Q value Q(s t , a t ;ω) and target Q value y t The difference between the TD error and the sample sampling priority p is calculatedt , the priority calculation formula is as follows:

[0100]

[0101] Among them, δ t is the sample priority label, p t is the sample priority, τ is the sampling guarantee factor to ensure that the sample priority will not be 0;

[0102] According to the sampling priority p t Calculate the sampling probability P(t) of the sample transition, and use α as the weight factor to weigh uniform sampling and greedy sampling. If a transition has just been collected, it means that the data has not been used yet, then δ t Set it to the maximum value and give it the highest weight. The calculation formula for sampling probability P(t) is as follows:

[0103]

[0104] Among them, P(t) is the sample sampling probability, and α is the weight factor that weighs uniform sampling and greedy sampling;

[0105] (4) To prevent overfitting and speed up convergence, a bias mechanism is introduced. When calculating the loss function value in the training phase, a bias weight θ is added according to the TD error of the sample. t To improve the loss function, the bias weight θ t Characterizes the learning efficiency of the agent, θ t The larger the value, the greater the gap between the predicted value and the TD target value at this time step, and the greater the update amount of the agent, which means that the learning efficiency here is higher. At the same time, the importance sampling factor β is used to represent the priority experience playback weight. The deviation weight calculation formula is as follows:

[0106] θ t =(N*P(t)) -β (β∈(0,1),t∈[0,N])

[0107] Where t is the sample, N is the number of samples in the experience pool, β is the importance sampling factor, β = 0 means that importance sampling is not used, but when β = 1, the effect of priority experience playback will also be offset, and the priority experience will degenerate into traditional experience playback, so the distribution range of β is (0, 1);

[0108] The improved loss function calculation formula is as follows:

[0109]

[0110] Among them, m is the number of samples, θ tis the bias weight;

[0111] (5) To uniformly transfer the sample dimensions and reduce sample differences, the bias weights need to be normalized. The normalized bias weight calculation formula is as follows:

[0112]

[0113] (6) Repeat (2), (3), (4), and (5) to evaluate the priority of each collected sample. At the same time, recalculate the sample priority of each training sample after each round of training until the training is completed. The improved DQN algorithm training is as follows: Figure 7 shown.

[0114] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0115] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A path planning method for an agricultural robot, characterized in that: include: Build a reinforcement learning interactive environment based on the prior map, and use the intermediate target points to segment the global environment to design the DQN algorithm network architecture; The DQN algorithm reward function is designed based on the forward bow-shaped operation mode of the agricultural robot, and the reward function consists of a midline area reward function and a target point reward function; Based on the forward bow-shaped operation mode of the agricultural robot and the neighborhood exploration mode of the A* algorithm, the DQN algorithm action space and exploration strategy are set; The priority experience replay mechanism is used to access and transfer sample data to design the evaluation network and target network for path planning and decision-making of the agricultural robot.

2. The path planning method for an agricultural robot according to claim 1, characterized in that: The method of constructing a reinforcement learning interactive environment based on a priori maps and using intermediate target points to segment the global environment and design the DQN algorithm network architecture includes: Use SLAM algorithm to scan the global environment to build a priori map, and build a deep reinforcement learning interactive environment based on the priori map. The environment is divided into crop row area and operation row area. The crop row area is an inaccessible area and the operation behavior is a passable area. At the same time, the crop row is expanded and fitted, and the left and right expansion widths are half of the robot width. Combined with the forward bow-shaped operation mode of the agricultural robot, an intermediate target point is added between the starting point and the end point to cut the overall complex environment into short-range simple paths; The deep reinforcement learning DQN algorithm is used to build the algorithm network architecture according to the working environment and working mode.

3. The path planning method for an agricultural robot according to claim 2, characterized in that: The method combines the forward bow-shaped operation mode of the agricultural robot, adds an intermediate target point between the starting point and the end point, and cuts the overall complex environment into short-range simple paths, including: Take the midline coordinates of the walking row and the crop row as the horizontal coordinates of the intermediate target point; And take the vertical coordinate of the front end of the crop row and move it up by λd col Set the ordinate of the intermediate target point, move the odd-numbered crop rows up, and move the even-numbered crop rows down, and divide the complex overall path into multiple simple paths, where λ is the translation factor of the ordinate of the intermediate target point, d col It is the longitudinal distance between the crop row and the working boundary.

4. The path planning method for an agricultural robot according to claim 3, characterized in that: To prevent the robot from missing the intermediate target point, the robot is considered to have found the target point once it enters the target point area. The target point area is centered on the target point and is r area is the radius, and the area radius calculation formula is as follows: r area ＝0.25d tr Among them, r area is the area radius, d tr The spacing between walking rows; The calculation formula of the ordinate of the intermediate target point is as follows: and tp =and ep ±λd col Among them, y tp is the ordinate of the intermediate target point, y ep It is the vertical coordinate of the front end point of the crop row.

5. The path planning method for an agricultural robot according to claim 1, characterized in that: The DQN algorithm reward function is designed based on the forward bow-shaped operation mode of the agricultural robot, including: Construct a midline area reward function based on the distribution of crop rows and work rows to ensure that the robot drives in the midline area as much as possible when working between rows; A target point reward function is constructed based on the intermediate target points. The navigation area is divided by the target points. Each area consists of a starting point and an end point, and the target point area is named after the area starting point. The starting point of the first target point area is the environment starting point, and the end point of the nth target point area is the environment end point.

6. The path planning method for an agricultural robot according to claim 5, characterized in that: The midline area reward function is constructed according to the distribution law of crop rows and operation rows to ensure that the robot drives in the midline area as much as possible when operating between rows, including: Taking the centerline spacing between working rows as the reference, when the minimum value d in the laser radar scanning distance min In the midline area [0.4d tr , 0.6d tr ], the reward function is positive, and the closer the robot is to the walking center line, the higher the reward value; conversely, when the robot is driving outside the center line area, the reward value is negative, and the closer to the obstacles on both sides, the greater the penalty value. The calculation formula of the center line area reward function is as follows Among them, r mid is the midline area reward function, a is the midline reward factor, d tr is the row width, d min is the minimum scanning distance.

7. The path planning method for an agricultural robot according to claim 5 or 6, characterized in that: The Euclidean distance between the starting point and the end point of the region is defined as the region distance, denoted by d reg , the Euclidean distance from the robot's current position to the end point of the area is the current distance, denoted as d cur , using the area distance d reg Distance d from current cur The square of the difference is used to calculate the target point reward function value, and b is used as the regional reward factor. If d reg -d cur >0, indicating that the current distance is less than the regional distance, that is, the robot is moving towards the end of the region. At this time, b>0 indicates that the target point reward value is positive; if d reg -d cur <0, indicating that the current distance is greater than the regional distance, that is, the robot is moving in the opposite direction. At this time, b<0 means that the target point reward value is negative; the target point reward function calculation formula is as follows: r dis =b*(d cur -d reg ) 2 Among them, r dis is the target point reward function, b is the target point reward factor, d cur is the current distance, d reg is the regional distance.

8. The path planning method for an agricultural robot according to claim 1, characterized in that: The DQN algorithm action space and exploration strategy are set based on the forward bow-shaped operation mode of the agricultural robot and combined with the neighborhood exploration mode of the A* algorithm, including: The 8-neighborhood exploration mode of the traditional A* algorithm is changed to the forward 5-neighborhood exploration. The action space of the DQN algorithm is specified to be 5-dimensional. The current position is used as the reference center. According to the vehicle head direction of the robot's current position, action 2 is specified to be directly in front of the vehicle head. Actions 1 and 3 are respectively the left and right 45° turning directions, and actions 0 and 4 are respectively the left and right 90° turning directions. The robot's forward movement and steering are controlled by linear velocity and angular velocity. The formula for calculating the angular velocity by the action sequence number is as follows: Where w is the angular velocity of the robot, a s_size is the length of the action space, a i is the action number, max_w is the maximum angular velocity; Based on the idea of ​​greedy strategy, the action selection strategy is improved, and the greedy strategy exploration factor ε is set to dynamically adjust to adapt to the adjustment of exploration actions at different stages of training.

9. The path planning method for an agricultural robot according to claim 8, characterized in that: The improved action selection strategy based on the greedy strategy idea and the setting of dynamically adjusting the greedy strategy exploration factor ε to adapt to the adjustment of the exploration action at different stages of training include: In the early stage of training, ε is set to 1. At this stage, we need to quickly explore the unknown environment to accumulate experience, select the action with the maximum Q value, reduce invalid exploration and reward oscillation, and shorten the convergence process. As the number of training times increases, the agent becomes more and more familiar with the environment, and accumulates more and more optimal exploration experience. At this time, the exploration factor ε is continuously reduced, and the way of selecting actions gradually changes from the maximum Q value action to the action that can obtain the maximum reward among the known actions, so that the agent can quickly understand the unknown environment and find the optimal path more easily, simplify the target point search process, shorten the navigation time, and dynamically adjust the exploration factor ε formula as follows: Among them, ε is the exploration factor, ε0 is the exploration factor per round, and ε min To explore the minimum value of the factor, ε max is the maximum value of the exploration factor, up_iter is the network update frequency, and ε_decay is the number of decay steps.

10. The path planning method for an agricultural robot according to claim 1, characterized in that: The method of using a priority experience playback mechanism to access and transfer sample data to design an evaluation network and a target network to perform path planning and decision-making for an agricultural robot includes: (1) Generate the evaluation network (Eval Net) and target network (Target Net) through the DQN structure. The parameters of the evaluation network and target network are ω and ω respectively. - ; (2) Based on the current state s, the greedy strategy is used to select action a, and then interact with the environment to obtain reward r, the next state s_, and the flag bit done to indicate whether the round is over, and form a transfer sample (s t , a t , r t ,s t+1 ); (3) For each transfer sample (s t , a t , r t ,s t+1 ) is marked with a priority mark δ t , according to the current Q value Q(s t , a t ;ω) and target Q value y t The difference between the TD error and the sample sampling priority p is calculated t , the priority calculation formula is as follows: Among them, δ t is the sample priority label, p t is the sample priority, τ is the sampling guarantee factor to ensure that the sample priority will not be 0; According to the sampling priority p t Calculate the sampling probability P(t) of the sample transition, and use α as the weight factor to weigh uniform sampling and greedy sampling. If a transition has just been collected, it means that the data has not been used yet, then δ t Set it to the maximum value, give it the highest weight, and the sampling probability P(t) is calculated as follows: Among them, P(t) is the sample sampling probability, and α is the weight factor that weighs uniform sampling and greedy sampling; (4) To prevent overfitting and speed up convergence, a bias mechanism is introduced. When calculating the loss function value during the training phase, a bias weight θ is added based on the TD error of the sample. t To improve the loss function, the bias weight θ t Characterizes the learning efficiency of the agent, θ t The larger the value is, the greater the gap between the predicted value and the TD target value at this time step is, and the greater the update amount of the agent is, indicating that the learning efficiency here is higher. At the same time, the importance sampling factor β is used to represent the priority experience replay weight. The deviation weight calculation formula is as follows: θ t =(N*P(t)) -β (β∈(0,1),t∈[0,N]) Among them, t is the sample, N is the number of samples in the experience pool, β is the importance sampling factor, and the distribution range of β is (0, 1). The improved loss function calculation formula is as follows: Among them, m is the number of samples, θ t is the bias weight; (5) To uniformly transfer the sample dimensions and reduce sample differences, the bias weight needs to be normalized. The normalized bias weight calculation formula is as follows: (6) Repeat (2), (3), (4) and (5) to evaluate the priority of each collected sample. At the same time, recalculate the sample priority of each training sample after each round of training until the training is completed.

Images