A field navigation method for a phenotypic robot based on ground-air collaboration

Abstract

The present invention discloses a field navigation method of a phenotyping robot based on ground-air collaboration, by using a drone equipped with real-time motion compensation RTK technology and a camera module, a digital orthophoto DOM is constructed for a target crop area, and a semantic segmentation model is used to obtain a semantic segmentation map of the crop row body in combination with artificial intelligence technology, and the image is preprocessed by an image processing method, and the key points are obtained by combining the Skeletonize algorithm, Douglas-Peucker algorithm, Shi-Tomasi method and linear extrapolation method, and the key points are grouped and sorted according to appropriate strategies, and then imported into a navigation system to generate a coherent navigation path, and a pure tracking algorithm controller is used to track the path to complete the navigation task. The present invention is used for the automated collection of field crop phenotypes and continuous monitoring of the growth process, more finely covering the entire operating area, improving the level of control visualization, and helping to improve the accuracy and efficiency of crop phenotyping data collection.

Description

Technical Field

[0001] The present invention relates to the fields of smart agricultural equipment technology, artificial intelligence technology and field automation navigation, and in particular to a field navigation method of a phenotypic robot based on ground-air collaboration. Background Art

[0002] The autonomous navigation technology of the phenotyping robot has important application significance in the collection of crop phenotyping data, and provides basic support for accurate crop detection, crop phenotyping collection and yield prediction. In the past, the manual data collection method was time-consuming, labor-intensive and inefficient. With the rapid development of remote sensing technology, drones equipped with RTK high-precision positioning modules have unique aerial advantages, such as efficient monitoring capabilities and flexible maneuverability. They have become an important tool for crop phenotyping monitoring. Drones have shown great potential in crop growth monitoring, pest and disease detection, and phenotypic data collection. However, despite the excellent performance of drones in aerial monitoring, they still have limitations in data collection and accuracy. For example, when drones collect data in complex terrain and high-density crop coverage areas, the limited field of view will lead to discontinuous and missing data.

[0003] When doing path planning based on drone high-precision maps, compared with coverage path planning in 3D space, traditional 2D coverage path planning cannot take terrain factors into account. Hamid et al. performed 3D projection of the digital elevation model (DEM) of farmland terrain, expanded the traditional 2D coverage path planning to 3D space, and used complex algorithms to focus on solving the problem of how to minimize omissions and overlapping areas when agricultural robots are planning coverage paths. However, complex algorithms require a lot of computing power and are relatively resource-intensive.

[0004] Therefore, there is an urgent need for a method that can generate reliable coverage navigation methods based on actual crop growth distribution and limited resources. Summary of the invention

[0005] The technical problems to be solved by the present invention include but are not limited to the following: traditional farm robot navigation generally relies on its own GNSS module for positioning and tracking, is difficult to adapt to changes in crop layout and growth, is difficult to adapt to irregular crop distribution and curved crop rows, and does not have visual map monitoring.

[0006] In order to solve the above problems, the present invention provides the following technical solutions.

[0007] A field navigation method of a phenotypic robot based on ground-air collaboration, characterized in that it comprises the following steps:

[0008] Step 1: Use a drone equipped with real-time motion compensation RTK technology and a camera module to take aerial photos of the target crop area in the test field, construct a digital orthophoto DOM, and store it in GeoTIFF data format;

[0009] Step 2: crop the DOM so that the cropped area covers the crop row area and is converted into a JPG image data format. An XOY coordinate system is established, where the Y axis points to the flight direction of the drone, i.e., the direction of the crop row. Semantic segmentation prediction and filtering are performed on the cropped image data to obtain a denoised crop row image.

[0010] Step 3: The Skeletonize algorithm is used to extract the skeleton line of the denoised crop row image, and the Douglas-Peucker algorithm is used to perform straight line fitting on the skeleton line to obtain the straight line fitting line of each crop row area;

[0011] Step 4, fitting a straight line in each crop row area and extracting endpoints and inflection points;

[0012] Step 5: Use the GDAL library to convert the pixel coordinates of the endpoints and inflection points into geographic coordinates, that is, map them back to the DOM in step 1, match the actual longitude and latitude coordinates, and use the matched longitude and latitude coordinates as key control points.

[0013] Step 6, segmenting the DOM stored in the GeoTIFF data format in step 1 to obtain tile images, superimposing the segmented tile images on the satellite map, performing coordinate correction, and obtaining a high-resolution map;

[0014] Step 7, using the snake sorting algorithm to import the longitude and latitude coordinates of the key control points into the phenotypic robot navigation system, and then connecting these key points on the high-precision map generated in step 6 according to the order from the entrance to the exit, to generate a coherent navigation path;

[0015] Step 8, the navigation system of the phenotype robot is calculated through a pure tracking algorithm controller, outputs the steering angle of the phenotype robot, tracks the navigation path, and completes the navigation task.

[0016] Preferably, in step 1, aerial photography of the target crop area is performed to construct a digital orthophoto DOM, which includes the following steps:

[0017] Step S11, select the working field, determine the take-off and landing location of the UAV, arrange the ground control flight point, set the image photography overlap rate, perform orthophotography of the working field and collect geographic coordinates;

[0018] Step S12, analyzing multiple images of overlapping areas by aerial triangulation, determining the geographic coordinate difference of the same landmark in different overlapping images, and calculating the true value of the geographic coordinate data of the landmark;

[0019] Step S13 , performing two-dimensional reconstruction according to the flight altitude, flight direction and photography overlap rate parameters of the UAV combined with the real value of the geographic coordinate data, generating a digital orthophoto DOM, and storing it in GeoTIFF data format.

[0020] Preferably, in step 2, the filtering process specifically comprises the following steps:

[0021] Step S21, binarizing the semantically segmented image, setting the background area to 0, assigning the crop row area to 1, and then blurring and resizing the crop row area to obtain a pre-processed image;

[0022] Step S22, using the findContours method to extract the preprocessed image and identify the contour lines of the crop row area;

[0023] In step S23, the value of the extracted contour line is assigned to 1, merged into the crop row area, and then the image size is reduced to the original size by resizing to obtain a denoised crop row image.

[0024] Preferably, in step 4, the endpoint and inflection point extraction strategy is specifically:

[0025] Step S41, setting a 3×3 convolution kernel, traversing all pixel points of the skeleton line, identifying the number of adjacent points of each point, and if the number of adjacent points of a point is 1, marking it as a temporary endpoint;

[0026] Step S42, for each crop row region straight line fitted by the Douglas-Peucker algorithm, all pixel points on each straight line are traversed by the Shi-Tomasi method to detect its minimum eigenvalue. Only when the minimum eigenvalue exceeds a set specific threshold, the point is identified as an inflection point;

[0027] Step S43, starting from the temporary endpoint according to the length of the phenotype robot, find the nearest point in the same area as the temporary endpoint, extend the line along the direction of the two points, i.e., perform linear extrapolation, and determine new pixel points, which are regarded as endpoints;

[0028] Step S44, based on the endpoints and inflection points, grouping and sorting are performed according to the crop row areas to which they belong, grouping the points belonging to the same crop row area into one group, and sorting the points in the same crop row area in order of the size of the Y-axis coordinate values.

[0029] Preferably, in step S43, the endpoints are extracted by linear extrapolation, and the specific extraction formula is:

[0030]

[0031]

[0032]

[0033]

[0034]

[0035] Wherein, U is the unit direction vector, P2-P1 is the vector calculated from P1(x1, y1) to P2(x2, y2), |P2-P1| is the modulus of the vector calculated by the Euclidean distance formula, P1(x1, y1) is the temporary endpoint extracted according to the convolution kernel traversal operation in step S41, P2(x2, y2) is the nearest point found as described in step S43, D is the distance the temporary endpoint extends outward along the direction of U, W is the length of the phenotype robot, S is the reserved safety distance, and U x and U y They are the components of the unit direction vector U along the x-axis and y-axis at the endpoint P(x,y), and x and y are the horizontal and vertical coordinates that determine the endpoint P(x,y).

[0036] Preferably, the specific method of grouping and sorting in step S44 is: setting the minimum circumscribed rectangle of the straight line fitting line of each crop row area, grouping the endpoints and inflection points in the same rectangle into one group, and then sorting the points in the same group in order according to the size of the y-axis coordinate value.

[0037] Preferably, the pure tracking algorithm controller in step 8 is specifically formulated as follows:

[0038]

[0039] Among them, α represents the deviation between the heading of the phenotypic robot and the heading of the target coordinate point, L is the wheelbase of the phenotypic robot, and L d It represents the distance from the current coordinate point of the phenotype robot to the target coordinate point, and δ is the steering angle of the wheels of the phenotype robot.

[0040] Compared with the prior art, the field navigation method of a phenotypic robot based on ground-air collaboration provided by the present invention has significant positive technical effects, which are specifically manifested in the following aspects.

[0041] (1) In the prior art, the robot navigation method in farmland generally relies on its own GNSS module for positioning and tracking, that is, the robot equipped with the GNSS module will first walk through the area to be navigated and record the trajectory, so as to track according to the pre-recorded path later. However, the layout of crops may change as they grow, resulting in insufficient adaptability of this navigation method. The present invention utilizes the high-altitude vision characteristics of the drone equipped with real-time motion compensation RTK technology and camera module to instantly shoot the growing crop scene, construct digital orthophotos, provide precise positioning, and extract the navigation path based on image processing technology, thereby enhancing the adaptability of farmland robot navigation.

[0042] (2) In the prior art, when performing path planning based on a two-dimensional map, a linear navigation path is generally designed according to the crop area. Although this method is easy to implement, it has the limitation of not being able to fully cover the entire operating area and is difficult to adapt to irregular crop distribution and curved crop rows. It is not universal in practical application. The present invention applies an endpoint and inflection point extraction strategy, which are called key points. The extraction strategy takes into account the distribution of crops and the actual direction of crop rows. By navigating in this way, the entire operating area can be covered more finely.

[0043] (3) In the prior art, during the navigation process of farmland robots, data is usually directly input into the robot navigation system. This method lacks intuitive information display, making it impossible for the operator to intuitively see the relative position and predetermined path of the robot in the actual farmland environment, thereby limiting the operator's ability to monitor the robot during navigation. The present invention divides, corrects and overlays DOM data with satellite maps to generate a high-resolution map, and connects the extracted key control points on the map to generate a coherent navigation path. The robot tracks the path through a control algorithm and displays the tracking effect in real time, which effectively improves the user's control experience and facilitates the acquisition of navigation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The present invention is a flow chart of the method.

[0045] Figure 2 A DOM image stored in GeoTIFF data format.

[0046] Figure 3 It is the semantic segmentation map of the crop row body.

[0047] Figure 4 Schematic diagram and flow chart for extracting inflection points using the Shi-Tomasi method.

[0048] Figure 5 The generated navigation path map.

[0049] Figure 6 Visualize the results of the path tracing. DETAILED DESCRIPTION

[0050] The present invention is further described below in conjunction with the accompanying drawings and implementation examples.

[0051] like Figure 1 As shown in the figure, a preferred embodiment of the present invention is a field navigation method of a phenotypic robot based on ground-air collaboration, comprising the following steps:

[0052] Step 1: Take aerial photos of the target crop area and construct a digital orthophoto DOM, which includes the following sub-steps:

[0053] (1) Select the working field, determine the take-off and landing location of the UAV, arrange the ground control flight point, set the image photography overlap rate to 80%, and perform orthophotography and geographic coordinate collection of the working field;

[0054] (2) Analyze multiple images of overlapping areas through aerial triangulation to determine the geographic coordinate differences of the same landmark in different overlapping images and calculate the true value of the geographic coordinate data of the landmark;

[0055] (3) According to the flight altitude, flight direction and photography overlap rate parameters of the UAV, combined with the real value of the geographic coordinate data, a two-dimensional reconstruction is performed to generate a digital orthophoto DOM, which is stored in the GeoTIFF data format. The DOM is as follows: Figure 2 shown.

[0056] Step 2, semantic segmentation prediction and filtering are performed on the cropped image data, which includes the following sub-steps:

[0057] (1) Dataset preparation: The crop row area in the JPG image data obtained in step 2 is segmented into 512 × 512 dimensions. All crop row main data sets after segmentation are labeled using the bounding box annotation method using the labelme software.

[0058] (2) Model training: Use the open source semantic segmentation model padlepadle-liteseg to train the dataset annotated in (1) to obtain a semantic segmentation model that can segment the main body of the crop row;

[0059] (3) Model prediction: The image containing the target crop row area is imported into the semantic segmentation model in (2) for prediction, and a semantic segmentation map containing the main crop row is obtained. The semantic segmentation map is as follows: Figure 3 As shown;

[0060] (4) The filtering process includes the following sub-steps.

[0061] 1) Preprocessing: Binarize the semantic segmentation map, set the background area to 0, assign the crop row area to 1, and then blur and resize the crop row area to obtain the preprocessed image;

[0062] 2) Contour extraction: Use the findContours method to extract the preprocessed image and identify the contour lines of the crop row area;

[0063] 3) Contour smoothing: The value of the extracted contour line is assigned to 1, merged into the crop row area, and then the image size is reduced to the original size by resizing to obtain the denoised crop row image.

[0064] Step 3, obtaining the straight line fitting line of each crop row area includes the following sub-steps:

[0065] (1) Crop row region skeleton extraction: The denoised crop row image is skeletonized using the Skeletonize algorithm, and the single-element crop row shape is obtained by combining logical sum operations.

[0066] (2) Skeleton line fitting: The Douglas-Peucker algorithm is used to perform straight line fitting on the skeleton to obtain the straight line fitting line of each crop row area.

[0067] Step 4, endpoint and inflection point extraction strategy includes the following sub-steps:

[0068] (1) Set a 3×3 convolution kernel, traverse all the pixels of the skeleton line, identify the number of adjacent points of each point, and if the number of adjacent points of a point is 1, mark it as a temporary endpoint;

[0069] (2) For each crop row area fitted by the Douglas-Peucker algorithm, all pixel points on each straight line fitting line are traversed by the Shi-Tomasi method, as follows: Figure 4 As shown, the Sobel operator is used to calculate the gradient and autocorrelation matrix of each pixel point, and then the eigenvalue of the autocorrelation matrix is ​​obtained, and the minimum eigenvalue of the pixel point is detected. Only when the minimum eigenvalue exceeds the set specific threshold, the point is identified as an inflection point;

[0070] (3) According to the length of the phenotype robot, starting from the temporary endpoint, find the nearest point in the same area as the temporary endpoint, extend it in a straight line along the direction of the two points, that is, linear extrapolation, and determine new pixel points. These new pixel points are regarded as endpoints. The specific formula of linear extrapolation is:

[0071]

[0072]

[0073]

[0074]

[0075]

[0076] Wherein, U is the unit direction vector, P2-P1 is the vector calculated from P1(x1, y1) to P2(x2, y2), |P2-P1| is the modulus of the vector calculated by the Euclidean distance formula, P1(x1, y1) is the temporary endpoint extracted according to the convolution kernel traversal operation in step S41, P2(x2, y2) is the nearest point found as described in step S43, D is the distance the temporary endpoint extends outward along the direction of U, W is the length of the phenotype robot, S is the reserved safety distance, and U x and U y They are the components of the unit direction vector U along the x-axis and y-axis at the endpoint P(x,y), and x and y are the horizontal and vertical coordinates that determine the endpoint P(x,y).

[0077] (4) Based on the endpoints and inflection points, the points are grouped and sorted according to the crop row areas to which they belong. The minimum circumscribed rectangle of the straight line fitting line of each crop row area is set, and the endpoints and inflection points in the same rectangle are grouped together. Then, the points in the same group are sorted in order according to the size of the y-axis coordinate value.

[0078] Step 5: Use the GDAL library to convert the pixel coordinates of the endpoints and inflection points into geographic coordinates, that is, map them back to the DOM in step 1, match the actual longitude and latitude coordinates, and use the matched longitude and latitude coordinates as key control points.

[0079] Step 6, segment the DOM stored in the GeoTIFF data format in step 1 to obtain tile images, overlay the segmented tile images on the corresponding positions of the satellite map according to the coordinate information, and then perform coordinate correction according to the known coordinate points to obtain a high-resolution map;

[0080] Step 7, the longitude and latitude coordinates of the key control points are imported into the phenotypic robot navigation system using the snake sorting algorithm, and then these key points are connected on the high-precision map generated in step 6 according to the order from the entrance to the exit to generate a coherent navigation path, such as Figure 5 As shown;

[0081] Step 8: The navigation system of the phenotype robot calculates through the pure tracking algorithm controller, outputs the steering angle of the phenotype robot, tracks the navigation path, and realizes the navigation task. The specific formula of the pure tracking algorithm controller is:

[0082]

[0083] Among them, α represents the deviation between the heading of the phenotypic robot and the heading of the target coordinate point, L is the wheelbase of the phenotypic robot, and L d It represents the distance from the current coordinate point of the phenotype robot to the target coordinate point, and δ is the steering angle of the wheels of the phenotype robot.

[0084] like Figure 6 As shown, the navigation effect of the pure tracking algorithm controller meets the requirements and ensures that the phenotypic robot can stably travel along the predetermined path. At the same time, information visualization makes the entire navigation process more intuitive. At the same time, the lateral error between the position of the phenotypic robot and the predetermined path during the navigation process is recorded. In this embodiment, the maximum absolute lateral error and the average absolute lateral error are used as two indicators of the evaluation method. The maximum absolute error is 6.40 cm, and the average absolute error is 2.83 cm. It can be seen that the scheme of the present invention has good navigation accuracy.

[0085] The specific examples described in the application are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific examples described in the present invention, or replace them in a similar manner, but they will not deviate from the spirit of the present invention or exceed the scope defined by the attached claims.

Claims

1. A field navigation method for a phenotypic robot based on ground-air collaboration, It is characterized in that The following steps are involved: Step 1: Use a drone equipped with real-time motion compensation RTK technology and a camera module to take aerial photos of the target crop area in the test field, construct a digital orthophoto DOM, and store it in GeoTIFF data format; Step 2: crop the DOM so that the cropped area covers the crop row area and is converted into a JPG image data format. An XOY coordinate system is established, where the Y axis points to the flight direction of the drone, i.e., the direction of the crop row. Semantic segmentation prediction and filtering are performed on the cropped image data to obtain a denoised crop row image. Step 3: The Skeletonize algorithm is used to extract the skeleton line of the denoised crop row image, and the Douglas-Peucker algorithm is used to perform straight line fitting on the skeleton line to obtain the straight line fitting line of each crop row area; Step 4, fitting a straight line in each crop row area and extracting endpoints and inflection points; Step 5: Use the GDAL library to convert the pixel coordinates of the endpoints and inflection points into geographic coordinates, that is, map them back to the DOM in step 1, match the actual longitude and latitude coordinates, and use the matched longitude and latitude coordinates as key control points. Step 6, segmenting the DOM stored in the GeoTIFF data format in step 1 to obtain tile images, superimposing the segmented tile images on the satellite map, performing coordinate correction, and obtaining a high-resolution map; Step 7, using the snake sorting algorithm to import the longitude and latitude coordinates of the key control points into the phenotypic robot navigation system, and then connecting these key points on the high-precision map generated in step 6 according to the order from the entrance to the exit, to generate a coherent navigation path; Step 8, the navigation system of the phenotype robot calculates through a pure tracking algorithm controller, outputs the steering angle of the phenotype robot, tracks the navigation path, and realizes the navigation task; Among them, in step 4, the endpoint and inflection point extraction strategy is specifically as follows: Step S41, setting a 3×3 convolution kernel, traversing all pixel points of the skeleton line, identifying the number of adjacent points of each point, and if the number of adjacent points of a point is 1, marking it as a temporary endpoint; Step S42, for each crop row region straight line fitted by the Douglas-Peucker algorithm, all pixel points on each straight line are traversed by the Shi-Tomasi method to detect its minimum eigenvalue. Only when the minimum eigenvalue exceeds a set specific threshold, the pixel point is identified as an inflection point; Step S43, starting from the temporary endpoint according to the length of the phenotype robot, find the nearest point in the same area as the temporary endpoint, extend the line along the direction of the two points, i.e., perform linear extrapolation, and determine new pixel points, which are regarded as endpoints; Step S44, grouping and sorting the points according to the crop row regions based on the endpoints and inflection points, grouping the points belonging to the same crop row region into one group, and sorting the points in the same crop row region in order of Y-axis coordinate values; Among them, in step S43, the endpoint is extracted by linear extrapolation, and the specific extraction formula is: Where U is the unit direction vector, P 2 -P 1 Is to calculate P 1 (x 1 ,y 1 ) points to P 2 (x 2 ,y 2 ), |P 2 -P 1 | is the modulus of the vector calculated by the Euclidean distance formula, P 1 (x 1 ,y 1 ) is the temporary endpoint extracted according to the convolution kernel traversal operation in step S41, P 2 (x 2 ,y 2 ) is the nearest point to be found as described in step S43, D is the distance of the temporary endpoint extending outward along the direction of U, W is the length of the phenotype robot, S is the reserved safety distance, and U x and U y They are the components of the unit direction vector U along the x-axis and y-axis at the endpoint P(x,y), and x and y are the horizontal and vertical coordinates that determine the endpoint P(x,y).

2. The field navigation method of the phenotypic robot based on ground-air collaboration according to claim 1, Features: In step 1, aerial photography of the target crop area is performed to construct a digital orthophoto DOM, which includes the following steps: Step S11, select the working field, determine the take-off and landing location of the UAV, arrange the ground control flight point, set the image photography overlap rate, perform orthophotography of the working field and collect geographic coordinates; Step S12, analyzing multiple images of overlapping areas by aerial triangulation, determining the geographic coordinate difference of the same landmark in different overlapping images, and calculating the true value of the geographic coordinate data of the landmark; Step S13 , performing two-dimensional reconstruction according to the flight altitude, flight direction and photography overlap rate parameters of the UAV combined with the real value of the geographic coordinate data, generating a digital orthophoto DOM, and storing it in GeoTIFF data format.

3. The field navigation method of the phenotypic robot based on ground-air collaboration according to claim 1, It is characterized in that In step 2, the filtering process specifically includes the following steps: Step S21, binarizing the semantically segmented image, setting the background area to 0, assigning the crop row area to 1, and then blurring and resizing the crop row area to obtain a pre-processed image; Step S22, using the findContours method to extract the preprocessed image and identify the contour lines of the crop row area; In step S23, the value of the extracted contour line is assigned to 1, and the contour line is merged into the crop row area. Then, the image size is reduced to the original size by resizing to obtain a denoised crop row image.

4. The field navigation method of the phenotypic robot based on ground-air collaboration according to claim 3, It is characterized in that The specific method of grouping and sorting in step S44 is: setting the minimum circumscribed rectangle of the straight line fitting line of each crop row area, grouping the endpoints and inflection points in the same rectangle into one group, and then sorting the points in the same group in order according to the size of the y-axis coordinate value.

5. The field navigation method of the phenotypic robot based on ground-air collaboration according to claim 1, It is characterized in that The pure tracking algorithm controller in step 8 is controlled by the following formula: Among them, α represents the deviation between the heading of the phenotypic robot and the heading of the target coordinate point, L is the wheelbase of the phenotypic robot, and L d It represents the distance from the current coordinate point of the phenotype robot to the target coordinate point, and δ is the steering angle of the wheels of the phenotype robot.

Images