Coarse-to-fine agricultural aerial survey path generation method and system

Abstract

The invention relates to the technical field of agricultural automatic driving, in particular to a coarse-to-fine agricultural aerial survey path generation method and system. The method comprises the following steps: acquiring an orthographic aerial survey image generated by aerial survey of an unmanned aerial vehicle; carrying out image segmentation on the obtained orthographic aerial survey image, and carrying out crude extraction to obtain an operable area and a crop area; according to different operation types, an operation path and a transportation path are planned in the workable area; and the operation path and the transportation path are input into the agricultural robot, and full-autonomous operation is achieved. The method is extremely high in efficiency, and is very suitable for unmanned vehicle operation deployment in orchards or farmlands with ultra-large planting areas.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural automatic driving, and in particular to a method and system for generating a coarse-to-fine agricultural aerial survey path. Background Art

[0002] In the prior art, there are two main methods for agricultural unmanned vehicle path planning on the market: using satellite positioning dotting devices or the agricultural unmanned vehicle's own satellite positioning data, dotting is done at intervals on the path planned within the operating range to form a series of dot graphs. Then, the agricultural robot travels in a straight line between two adjacent points. Finally, a dot graph is formed to guide the agricultural unmanned vehicle to travel between points. This method is mostly used for orchard plant protection robots and automatic micro-tillage machines.

[0003] Similarly, satellite positioning dotting devices or the agricultural unmanned vehicle's own satellite positioning are used to mark the area around the operation with satellite points, and then the agricultural unmanned vehicle autonomously plans a full coverage path to cover the operation area. This method is mostly used for unmanned lawn mowing robots.

[0004] However, both of the above methods require people to hold a positioning terminal or control a robot to reach the required recorded points for marking operations. For large-scale operation scenarios, this method is inefficient, the workload of manual marking is large, and it takes a long time.

[0005] Therefore, there is an urgent need for a method and system for generating agricultural aerial survey paths from coarse to fine. Summary of the invention

[0006] In order to solve the above-mentioned problems, the present invention provides a method and system for generating a coarse-to-fine agricultural aerial survey path.

[0007] In a first aspect, the present invention provides a method for generating a coarse-to-fine agricultural aerial survey path, which adopts the following technical solution: A method for generating agricultural aerial survey paths from coarse to fine, comprising: Obtain orthophoto aerial images generated by drone aerial survey; Perform image segmentation on the acquired orthophoto aerial survey images, and obtain the workable area and crop area through rough extraction; According to different operation types, the operation path and transportation path are planned within the operable area; The operation path and transportation path are input into the agricultural robot to achieve fully autonomous operation.

[0008] Furthermore, the obtaining of orthophoto aerial images generated by drone aerial survey includes performing aerial survey of the operation scene by using a drone to generate high-precision orthophoto aerial images.

[0009] Furthermore, the acquired orthophoto aerial survey images are subjected to image segmentation, and the workable area and the crop area are obtained by rough extraction, including using an edge detection-based image segmentation algorithm and a deep learning feature detection algorithm to perform image segmentation on the orthophoto aerial survey images respectively, and dividing the images into crop areas and workable areas.

[0010] Furthermore, the orthophoto aerial survey image is segmented by using an edge detection-based image segmentation algorithm and a deep learning feature detection algorithm, respectively, including edge detection of the orthophoto aerial survey image using a Sobel operator, obtaining the approximate horizontal and vertical brightness difference values ​​of the orthophoto aerial survey image through edge detection, and finding the local maximum value through non-maximum suppression.

[0011] Furthermore, the orthophoto aerial survey images are segmented using an edge detection-based image segmentation algorithm and a deep learning feature detection algorithm, respectively, and the U-Net deep learning segmentation algorithm is used to perform fine segmentation on the crop images in the operation area, thereby obtaining the operation points one by one on the operation path, thereby realizing precise spraying of crops.

[0012] Furthermore, according to different operation types, operation paths and transportation paths are planned in the operable area, including for spraying operations, using a median algorithm to extract the path center line of the operable area so that the spraying equipment is operated on the center line; for weeding operations, a bow-shaped path coverage algorithm is used to achieve a bow-shaped full coverage path.

[0013] Furthermore, the operation path and transportation path planning are carried out in the operable area according to different operation types, and for transportation operations, the transportation nodes set by the dot device are included in the calculation list through the A* algorithm, the list is traversed to process the node data, and the feasible nodes are screened for planning to obtain the shortest path, so as to complete the transportation within the shortest path.

[0014] In the second aspect, a coarse-to-fine agricultural aerial survey path generation system is provided, comprising: The data acquisition module is configured to acquire orthophoto aerial survey images generated by drone aerial survey; An image segmentation module is configured to perform image segmentation on the acquired orthophoto aerial survey image, and obtain a workable area and a crop area through rough extraction; A path planning module is configured to plan operation paths and transportation paths within an operable area according to different operation types; The operation module is configured to input the operation path and the transportation path into the agricultural robot to achieve fully autonomous operation.

[0015] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, for example, the method for generating a coarse-to-fine agricultural aerial survey path.

[0016] In a fourth aspect, the present invention provides a terminal device including a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; and the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the method for generating a coarse-to-fine agricultural aerial survey path.

[0017] In summary, the present invention has the following beneficial technical effects: The problem solved by the present invention is that the marking work of existing agricultural robots during automatic operation deployment is relatively complicated and inefficient. The method proposed by the present invention is to use a drone to conduct a large-scale aerial survey of the operation area, such as an orchard, and then identify the robot's operable area in the farmland based on high-precision aerial survey images in combination with an AI image recognition algorithm, and then further generate a more accurate reciprocating operation path in the operable area. At the same time, manual calibration can also be used to calibrate unidentified obstacle information, such as wells, wires, etc., so that the generated path bypasses the calibrated obstacles, thereby quickly generating an agricultural unmanned vehicle operation path, solving the problem of complex deployment of unmanned vehicles in large-scale agricultural scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of a method for generating a coarse-to-fine agricultural aerial survey path according to Embodiment 1 of the present invention; Figure 2 It is the original image of the drone aerial survey orthophoto according to Embodiment 1 of the present invention; Figure 3 The drivable area map after the rough extraction step of Example 1 of the present invention; Figure 4 The path planning driving area map after the detailed extraction step of embodiment 1 of the present invention; Figure 5 U-Net network structure diagram of embodiment 1 of the present invention; Figure 6 A simplified diagram of the A* algorithm path planning in Example 1 of the present invention. DETAILED DESCRIPTION

[0019] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0020] Example 1 Reference Figure 1 The real-time process of this embodiment is as follows: Figure 1As shown in the figure, firstly, a UAV is used to conduct an aerial survey of the working environment to generate an aerial survey orthophoto; then, the image is imported into the automatic path planning and recognition system, and an edge detection-based image segmentation algorithm is used to segment the UAV aerial image, and the crop area and the workable area are identified and marked; then, the area output by the automatic recognition system is combined with the manually marked obstacles, and a complete planned working path is generated according to different working types; finally, the generated working path is imported into the agricultural unmanned vehicle control system, and the agricultural unmanned vehicle is guided to perform fully automatic operations in the orchard.

[0021] Specifically: 1. First, in the operation scene to be deployed, such as orchards and farmlands, use drone aerial surveys to generate high-precision orthophoto images. Figure 2 shown.

[0022] 2. The acquired orthophoto is then imported into the process designed by the present invention, and then the fruit trees or other crop rows are segmented from the aerial images using an edge detection-based image segmentation algorithm and AI deep learning feature detection. This is a coarse extraction step. The edge detection image segmentation algorithm attempts to solve the segmentation problem by detecting the edges of different regions. Usually, the grayscale values ​​of pixels on the boundaries of different regions vary dramatically. If the image is transformed from the spatial domain to the frequency domain through Fourier transform, the edge corresponds to the high-frequency part, so that fruit trees or other crop areas with a large difference in color from the land can be extracted. This step realizes the coarse extraction and segmentation of a larger area of ​​aerial survey images into crop areas and workable areas. Figure 3 shown.

[0023] Among them, the AI ​​deep learning feature detection method requires pre-collecting a certain number of orthophotos and annotating the fruit trees and other crops therein, and then training the annotated data to obtain the corresponding detection model. In order to obtain a higher model accuracy, a certain amount of annotated data needs to be guaranteed. This technical invention uses YoloV8 as the basic target detection algorithm.

[0024] 1. The specific edge detection implementation plan is as follows: (1) First, image preprocessing is performed: denoising and smoothing are performed on the aerial images to reduce the impact of noise on edge detection; For the current pixel point (x, y) to be processed, select a template composed of several of its neighboring pixels, calculate the mean of all pixels in the template, and then assign the mean to the current pixel point (x, y) as the grayscale g(x, y) of the processed image at this point, that is, g(x, y)=∑f(x, y) / m, where m is the total number of pixels in the template including the current pixel.

[0025] (2) Select an edge detection operator: such as Roberts operator, Prewitt operator, Sobel operator or Canny operator, including using a Gaussian filter to smooth the input image to remove noise; (You can add some formulas for Canny or Sobel operators, it is recommended to choose Sobel or Canny): The Sobel operator is mainly used for edge detection. Technically, it is a discrete difference operator that is used to calculate the grayscale approximation of the image brightness function. Using this operator at any point in the image will generate the corresponding grayscale vector or its normal vector.

[0026] The operator contains two sets of 3x3 matrices, one for horizontal and one for vertical. By performing a plane convolution with the image, the approximate values ​​of the horizontal and vertical brightness differences can be obtained. If A represents the original image, Gx and Gy represent the grayscale values ​​of the image after horizontal and vertical edge detection, respectively, the formula is as follows: (3) Calculate the gradient magnitude and angle image of the smoothed image, apply non-maximum suppression to the gradient magnitude image, find the local maximum, and set the local non-maximum to 0 to refine the edge; (add a formula for calculating the image gradient) Image gradient: G(x,y) = dx(i,j) + dy(i,j); dx(i,j) = I(i+1,j) - I(i,j); dy(i,j) = I(i,j+1) - I(i,j); Among them, I is the value of the image pixel (such as RGB value), and (i, j) is the coordinate of the pixel.

[0027] (4) Perform double threshold processing, set a high threshold and a low threshold, set points less than the low threshold as false edges and set them to 0, set points greater than the high threshold as strong edges and set them to 1, and determine whether to retain points between the two thresholds based on their connection with the strong edge.

[0028] (5) The U-Net network is used for fine-grained crop segmentation. The network can locate the pixels of the image, classify each pixel in the image, and finally output the image segmented according to the category of the pixel. The operation process of U-Net is similar to a "U" shape, and its specific structure is as follows: Figure 5 shown.

[0029] like Figure 5As shown in the figure, the U-Net network structure has four layers, which downsample and upsample the image four times respectively. The left half of the network is a four-time downsampling process, where the input size is downsampled from 572×572 to 32×32; the right half of the network is a four-time upsampling process, where the feature scale is restored from 28×28 to 392×392 after four upsamplings. In order to utilize the features of the left half of the downsampling process, the network crops the downsampled feature matrix into a feature matrix of the same size as the right half of the upsampling process, and then concatenates and fuses the features of the downsampling stage with the features of the upsampling stage. The U-Net network can output the results of fine crop segmentation end-to-end.

[0030] 3. After obtaining the images of the operable area and crop area, further, according to the type of operation required, such as spraying, weeding, transportation and other different operation types, the algorithm automatically plans the operation path suitable for the operation type in the operation area. For example, for spraying, the algorithm will use the median algorithm to extract the center line of the path in the operable area, so that the spraying equipment can run on the center line and apply the pesticide more accurately; for weeding operations, the algorithm will cover the entire operable area through a bow-shaped full coverage path, making the weeding operation more efficient and reducing the omission rate; for transportation operations, a traffic network map will be planned, so that the unmanned vehicle can complete the transportation operation along the shortest path.

[0031] Specifically, (1) The median algorithm extracts the path centerline of the workable area: First, find the two short sides of the rectangular area, find the pixel points (x1, y1) and (x2, y2) at the midpoints of the two short sides, and then find the pixel point (x3, y3) at the intersection of the two diagonals of the rectangular area. Connect the three points to extract the path centerline of the workable area.

[0032] (x1, y1) are the coordinates of the midpoint pixel of the short side, (a1, b1) and (a2, b2) are the coordinates of the vertices of the short side, and the method for calculating the coordinates of (x2, y2) is the same as above.

[0033] (x3, y3) are the coordinates of the pixel point in the middle of the short side, (a1, b1) and (a3, b3) are the coordinates of the diagonal vertices.

[0034] (2) Bow-shaped path coverage algorithm: Determine the area, then translate the straight line generated by parallel endpoints 0 and 1 to generate a fully covered path, then determine whether there is an obstacle avoidance area, determine the intersection of the path and the area, bypass the area and continue to generate paths until the area is fully covered, and connect the generated curves in an arc to form an arc-shaped path coverage.

[0035] 4. After completing the operation path planning, the path is loaded into the agricultural unmanned vehicle control system to control the agricultural unmanned vehicle to operate efficiently and accurately along the operation path.

[0036] By A * To optimize the shortest path problem, the position marked by the dotting device needs to be set as the node coordinates, where the origin coordinates and the initial coordinates of the work vehicle are set to (x0, y0), and the end point coordinates are set to (x k ,y k ), Euler's formula is used to calculate the Euclidean distance: in represents the cost estimate from the initial state via state n to the target state, represents the actual cost from the initial state to state n, Represents the estimated cost of the best path from state n to the goal state.

[0037] A * The algorithm needs to maintain two state tables, called Open_List and Close_List. The Open_List consists of nodes to be examined, and the Close_List consists of nodes that have been examined.

[0038] like Figure 6 As shown, starting from the starting point A, define A as the parent node and add it to the Open_List. There are 8 nodes around the parent node A, which are defined as child nodes. The actual cost of the child nodes is and estimated cost The information is put into Open_List and becomes the object to be examined. The moving costs of the remaining 8 child nodes in the current Open_List are calculated respectively, and the node F with the minimum cost is selected and moved to Close_List. And the moving cost of the child nodes adjacent to node A is selected The smallest node, taking node I shown in the figure as an example, calculate its cumulative moving cost And repeat the above steps with I as the parent node of the new round.

[0039] Following the above steps, traverse the list values ​​and perform optimal planning to obtain the shortest path, thereby completing the transportation within the shortest path.

[0040] The agricultural unmanned vehicle path planning method based on drone aerial survey images proposed in the present invention can greatly simplify the current agricultural unmanned vehicle deployment work and greatly improve the deployment efficiency. The previous technology uses a handheld dotting device or unmanned vehicle positioning to calibrate the operation point, and the operation efficiency is basically 300-500 acres per day. The agricultural unmanned vehicle path planning method based on drone aerial survey images proposed in the present invention can complete aerial photography of a 1,000-acre orchard within 1 hour using drone aerial survey images. The edge detection-based segmentation method is used to segment rows of fruit trees from the image within 30 minutes. The entire path planning of the agricultural unmanned vehicle in a 1,000-acre orchard is completed within 2 hours. It can be seen that this method is extremely efficient and is very suitable for the deployment of unmanned vehicle operations in orchards or farmlands with extremely large planting areas (more than 5,000 acres).

[0041] Example 2 This embodiment provides a coarse-to-fine agricultural aerial survey path generation system, including: The data acquisition module is configured to acquire orthophoto aerial survey images generated by drone aerial survey; An image segmentation module is configured to perform image segmentation on the acquired orthophoto aerial survey image, and obtain a workable area and a crop area through rough extraction; A path planning module is configured to plan operation paths and transportation paths within an operable area according to different operation types; The operation module is configured to input the operation path and the transportation path into the agricultural robot to achieve fully autonomous operation.

[0042] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor of a terminal device and executing the method described.

[0043] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; and the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the described method.

[0044] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating agricultural aerial survey paths from coarse to fine, characterized in that: include: Obtain orthophoto aerial images generated by drone aerial survey; Perform image segmentation on the acquired orthophoto aerial survey images, and obtain the workable area and crop area through rough extraction; According to different operation types, the operation path and transportation path are planned within the operable area; The operation path and transportation path are input into the agricultural robot to achieve fully autonomous operation.

2. The method for generating a coarse-to-fine agricultural aerial survey path according to claim 1, characterized in that: The obtaining of orthophoto aerial images generated by drone aerial survey includes performing aerial survey of the operation scene by using the drone to generate high-precision orthophoto aerial images.

3. The method for generating a coarse-to-fine agricultural aerial survey path according to claim 2, characterized in that: The acquired orthophoto aerial survey images are segmented to obtain the workable area and the crop area through rough extraction, including using an edge detection-based image segmentation algorithm and a deep learning feature detection algorithm to segment the orthophoto aerial survey images respectively, and divide the images into the crop area and the workable area.

4. The method for generating a coarse-to-fine agricultural aerial survey path according to claim 3, characterized in that: The orthophoto aerial survey images are segmented using an edge detection-based image segmentation algorithm and a deep learning feature detection algorithm, respectively, including edge detection of the orthophoto aerial survey images using a Sobel operator, obtaining approximate horizontal and vertical brightness difference values ​​of the orthophoto aerial survey images through edge detection, and finding local maxima through non-maximum suppression.

5. The method for generating a coarse-to-fine agricultural aerial survey path according to claim 4, characterized in that: The orthophoto aerial survey images are segmented using an edge detection-based image segmentation algorithm and a deep learning feature detection algorithm, respectively. The U-Net deep learning segmentation algorithm is also used to perform fine segmentation on the crop images in the operation area, thereby obtaining the operation points one by one on the operation path and realizing precise spraying operations.

6. The method for generating a coarse-to-fine agricultural aerial survey path according to claim 5, characterized in that: According to different operation types, the operation path and transportation path planning are carried out in the operable area, including for the pesticide spraying operation, using the median algorithm to extract the path center line of the operable area so that the pesticide spraying equipment is carried out on the center line; for the weeding operation, a bow-shaped path coverage algorithm is used to achieve a bow-shaped full coverage path.

7. The method for generating a coarse-to-fine agricultural aerial survey path according to claim 6, characterized in that: The operation path and transportation path planning are performed in the operable area according to different operation types, and for the transportation operation, A * The algorithm calculates the current cost distance between each transportation node, and uses the Euler formula to calculate the estimated cost distance to the target. The two cost distance functions are calculated and added to the list. The list values ​​are traversed and optimal planning is performed to obtain the shortest path, thereby completing the transportation within the shortest path.

8. A coarse-to-fine agricultural aerial survey path generation system, characterized in that: include: The data acquisition module is configured to acquire orthophoto aerial survey images generated by drone aerial survey; An image segmentation module is configured to perform image segmentation on the acquired orthophoto aerial survey image, and obtain a workable area and a crop area through rough extraction; A path planning module is configured to plan operation paths and transportation paths within an operable area according to different operation types; The operation module is configured to input the operation path and the transportation path into the agricultural robot to achieve fully autonomous operation.

9. A computer-readable storage medium storing a plurality of instructions, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executed by a method for generating a coarse-to-fine agricultural aerial survey path as claimed in claim 1.

10. A terminal device, comprising a processor and a computer-readable storage medium, wherein the processor is used to implement each instruction; and the computer-readable storage medium is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the method for generating a coarse-to-fine agricultural aerial survey path as claimed in claim 1.

Images