Method for generating navigation path of harvester

Abstract

A method for generating a navigation path of a harvester comprises the following steps: generating a foreground mask (eliminating a redundant background) through precise calibration of an ATX laser radar and an industrial RGB camera and segmentation of an SAM large model, realizing precise extraction of a land parcel boundary by combining clustering, collecting sample data of different weather and time periods, and strengthening anti-illumination and anti-fuzzy capabilities through data enhancement and model fine tuning; a downsampling + filtering algorithm is adopted to eliminate point cloud noise, a clustering algorithm is adopted to accurately distinguish harvested / unharvested areas through height difference, the method has the advantages of high boundary detection precision, high environmental adaptability and the like, and high-precision boundary recognition, complex environmental adaptability and real-time navigation path output can be considered at the same time; under the complex working condition, the operation area is stably recognized, the navigation route is generated in real time, and the operation precision and reliability of the automatic navigation harvester are improved.

Description

Technical Field

[0001] This invention relates to the field of harvester navigation technology, and in particular to a method for generating harvester navigation paths. Background Technology

[0002] Existing harvesting navigation technologies mainly include vision-based work area identification and GNSS-based path planning. However, visual identification is susceptible to changes in lighting, weed interference, and dust obstruction, leading to unstable image features and difficulty in maintaining reliable boundary detection and route identification in complex field environments. GNSS-based navigation schemes experience decreased positioning accuracy under conditions of electromagnetic interference and terrain undulations, and typically rely on preset paths, failing to dynamically generate navigation lines based on real-time changes in the harvesting area. In recent years, LiDAR has been applied in agricultural machinery scenarios, but the point cloud data volume is enormous, preprocessing algorithms are complex, real-time algorithm performance is difficult to guarantee, and it is not compatible with vehicle-mounted edge devices. Therefore, existing technologies struggle to simultaneously achieve high-precision boundary identification, adaptability to complex environments, and real-time navigation path output. Based on this, a path generation method is needed that integrates multi-source sensing information, can stably identify work areas and generate navigation routes in real time under complex working conditions, to improve the operational accuracy and reliability of automated guided harvesters. Summary of the Invention

[0003] The purpose of this invention is to provide a method for generating navigation paths for harvesters, which can stably identify the work area and generate navigation routes in real time under complex working conditions.

[0004] The technical solution adopted by this invention to solve the above-mentioned technical problems is: a method for generating a harvester navigation path, comprising the following steps: Step 1: Install the industrial RGB camera and ATX LiDAR at the front of the harvester cab, and use Zhang Zhengyou's checkerboard calibration method to correct image distortion of the industrial RGB camera. Step 2: Synchronously acquire RGB images of the calibration board and point cloud data of the ATX LiDAR, and establish the LiDAR point cloud coordinate system using the PnP algorithm. The mapping formula to the camera pixel coordinate system (u, v) is as follows: ; ; Where K is the camera intrinsic parameter matrix, R is the lidar extrinsic parameter rotation matrix, and T is the lidar extrinsic parameter translation vector; Step 3: Simultaneously acquire RGB image data and point cloud data of rice and wheat in the harvester's working area using the industrial RGB camera and ATX LiDAR installed in Step 1. After distortion correction of the RGB images, label the RGB images using a labeling tool to delineate the harvested and unharvested areas. Then clean the dataset and finally export a JSON format labeling file containing area category and area coordinate information. Step 4: Construct the dataset using the JSON format annotation file from Step 3 and divide it into training, validation, and test sets. Then, perform rice and wheat scene adaptation training on the SAM basic model. In the model decoding part, introduce multi-scale and local attention mechanisms, fuse low-level high-resolution features, and add small parallelogram priors to enhance small target feature extraction and anti-interference capabilities. During training, use geometric and color enhancement, edge / IoU weighted loss, and contrastive learning to improve the segmentation accuracy and robustness of small targets. Generate a binary foreground mask corresponding to the RGB image through the model output after training. Step 5: Convert the ATX LiDAR point cloud coordinates to pixel coordinates, filter the point cloud coordinates using the boundary coordinates of the harvested and unharvested areas, and compare them with the binary foreground mask to remove invalid point clouds that exceed the area boundaries, thus obtaining a set of valid point clouds. Step 6: Perform downsampling on the effective point cloud set and use a filtering algorithm to remove isolated points. Then, use a clustering algorithm to extract point cloud clusters of three different height feature regions from the effective point cloud set, namely, unharvested normal height point cloud clusters, unharvested middle and lower part point cloud clusters, and harvested area point cloud clusters. Keep the unharvested normal height point cloud clusters as the edge point set of the effective point cloud. Step 7: Use the latitude and longitude scanning method to extract boundary points from the edge point set, generating latitude scans parallel to the x-axis and longitude scans parallel to the y-axis. The latitude scan filters the corresponding y-coordinate point set and sorts them by x to obtain the first and last extreme points. The longitude scan filters the corresponding x-coordinate point set and sorts them by y to obtain the first and last extreme points. The two are then merged to obtain the initial boundary point set. Step 8: Fit the initial boundary point set with a third-order B-spline curve, then use a linear interpolation algorithm to generate a boundary curve using the fitted boundary points, and then superimpose the boundary curve onto the real-time RGB image. Step 9: Generate the corresponding navigation path based on the coordinates of the point cloud boundary points. Transform the navigation path from the pixel coordinate system into a sequence of coordinate points in the world coordinate system. Use a sliding window filtering algorithm to smooth the coordinate point sequence and remove inflection points in the path, thus completing the generation of the harvester navigation path.

[0005] Preferably, the image distortion correction method in step 1 is as follows: using a 7×5 checkerboard calibration board, at least 20 images of the calibration board are captured to ensure that all calibration board images cover the entire field of view of the camera. Then, the OpenCV's findChessboardCorners function is used to detect the corner points within the checkerboard, the cornerSubPix function is used for sub-pixel refinement, and the calibrateCamera function is used to calculate the camera intrinsic parameter matrix K and distortion coefficients D, expressed as: , where f x f y c is the focal length. x c y Principal point coordinates; , where k1, k2, and k3 are radial distortion coefficients, and p1 and p2 are tangential distortion coefficients; Then, the original camera coordinate system Distortion correction is performed to correct the distortion. The correction formula is: ,in .

[0006] Preferably, in step 2, R is a 3×3 rotation matrix, expressed as: T is a 3×1 translation vector, expressed as: .

[0007] Preferably, in step 2, the ATX LiDAR point cloud is projected onto the RGB image using a mapping formula, and the reprojection error between the calibration board point cloud contour and the calibration board edge in the image is calculated. When the average error exceeds the threshold, the installation positions of the industrial RGB camera and the ATX LiDAR are readjusted, or the number of calibration images is increased, and then step 2 is repeated until the reprojection error meets the threshold requirement.

[0008] Preferably, in step 3, the Labelme annotation tool is used to annotate 1000+ rice and wheat images after distortion correction, and the training set, validation set and test set are divided in a ratio of 8:1:1.

[0009] Preferably, in the binary foreground mask described in step 4, the value "1" corresponds to the target area and the value "0" corresponds to the background redundancy area.

[0010] Preferably, in step 6, a voxel lattice downsampling algorithm is used, with a voxel size of 1cm×1cm×1cm, and the number of output point clouds is 1 / 5 of the number of original point clouds.

[0011] Preferably, in step 6, the K-Means clustering algorithm is used, with the number of clusters K=3. The height characteristics of the three types of regions are as follows: unharvested normal height ≥ 0.5m, 0.15m < unharvested middle and lower part height ≤ 0.5m, and harvested area height ≤ 0.15m.

[0012] Preferably, in step 8, any two adjacent boundary points and The equation of the straight line between them is: .

[0013] Preferably, in step 9, when generating the navigation path, the distance R between the sensor and the boundary of the harvester and the overlap amount r are set according to the working width W of the harvester. The distance between the first path and the sensor is between (Rr) and R, and the distance between adjacent paths is (Wr).

[0014] According to the above technical solution, the beneficial effects of the present invention are: High boundary detection accuracy By accurately calibrating ATX LiDAR and industrial RGB cameras, generating foreground masks through SAM large model segmentation (removing redundant backgrounds), and combining clustering, accurate extraction of plot boundaries is achieved, effectively solving the problem of boundary misjudgment caused by uneven crop density and changes in light intensity in traditional methods.

[0015] Strong environmental adaptability We collect sample data under different weather conditions and at different times, and enhance the anti-lighting and anti-blurring capabilities through data augmentation and model fine-tuning. Downsampling and filtering algorithms remove point cloud noise, and clustering algorithms accurately distinguish between harvested and unharvested areas based on height differences.

[0016] In summary, this invention has many advantages, such as high boundary detection accuracy and strong environmental adaptability. It can simultaneously achieve high-precision boundary recognition, adaptability to complex environments, and real-time navigation path output. It can stably identify the working area and generate navigation routes in real time under complex working conditions, thereby improving the working accuracy and reliability of the automatic navigation harvester. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0018] like Figure 1 As shown, a method for generating a harvester navigation path includes the following steps: Step 1: Install the industrial RGB camera and ATX LiDAR at the front of the harvester cab, and use Zhang Zhengyou's checkerboard calibration method to correct image distortion of the industrial RGB camera.

[0019] The image distortion correction method involves using a 7×5 checkerboard calibration board with corner points, capturing at least 20 images of the calibration board to ensure that all images cover the entire field of view of the camera. Then, the `findChessboardCorners` function of OpenCV is used to detect the corner points within the checkerboard, and the `cornerSubPix` function is used for sub-pixel refinement. Finally, the `calibrateCamera` function is used to calculate the camera intrinsic parameter matrix K and distortion coefficients D, expressed as follows: , where f x f y c is the focal length. x c y The coordinates of the main point.

[0020] , where k1, k2, and k3 are radial distortion coefficients, and p1 and p2 are tangential distortion coefficients.

[0021] Then, the original camera coordinate system Distortion correction is performed to correct the distortion. The correction formula is: ,in .

[0022] Step 2: Synchronously acquire RGB images of the calibration board and point cloud data of the ATX LiDAR, and establish the LiDAR point cloud coordinate system using the PnP algorithm. The mapping formula to the camera pixel coordinate system (u, v) is as follows: ; .

[0023] Where K is the camera intrinsic parameter matrix, and R is the lidar extrinsic parameter rotation matrix, using a 3×3 rotation matrix, expressed as: T is the extrinsic translation vector of the lidar, using a 3×1 translation vector, expressed as: .

[0024] The ATX LiDAR point cloud is projected onto the RGB image using a mapping formula. The reprojection error between the calibration board point cloud contour and the calibration board edge in the image is calculated. If the average error is less than 0.5 pixels, it indicates that the calibration result has reached a high accuracy requirement and the calibration is successful. If the average error is greater than 0.5 pixels, it indicates that there is an error in the calibration process. The installation positions of the industrial RGB camera and ATX LiDAR should be readjusted, or the number of calibration images should be increased. Then step 2 should be repeated until the reprojection error meets the threshold requirement.

[0025] Step 3: Simultaneously acquire RGB image data and point cloud data of rice and wheat in the harvester's working area using the industrial RGB camera and ATX LiDAR installed in Step 1. After distortion correction of the RGB images, use the Labelme annotation tool to annotate more than 1,000 distortion-corrected rice and wheat images to delineate the harvested and unharvested areas. Then clean the dataset and finally export a JSON format annotation file containing area category and area coordinate information.

[0026] Step 4: Use the JSON format annotation file from Step 3 to construct the dataset and divide it into training, validation, and test sets in an 8:1:1 ratio. Then, train the SAM basic model to adapt it to the rice and wheat scene.

[0027] In the model decoding part, a multi-scale and local attention mechanism is introduced, which integrates the underlying high-resolution features and adds small parallelogram priors to enhance the feature extraction and anti-interference ability of small targets. During the training process, geometric and color enhancement, edge / IoU weighted loss and contrastive learning are used to improve the segmentation accuracy and robustness of small targets.

[0028] The trained model output generates a binary foreground mask corresponding to the RGB image, where a "1" value corresponds to the target region and a "0" value corresponds to the background redundancy region.

[0029] Step 5: Convert the ATX LiDAR point cloud coordinates to pixel coordinates, filter the point cloud coordinates using the boundary coordinates of the harvested and unharvested areas, and compare them with the binary foreground mask. Only retain the point cloud with the corresponding value of "1" in the mask, that is, the point cloud of the parallelogram area composed of the harvested and unharvested areas in the rice and wheat fields. This effectively removes redundant background point clouds such as the sky and distant trees that exceed the area boundary, reducing the amount of computation.

[0030] Step 6: Perform a voxel lattice downsampling operation on the effective point cloud set. The voxel size is 1cm×1cm×1cm, and the number of output point clouds is 1 / 5 of the number of original point clouds.

[0031] An isolated point was removed using a filtering algorithm. Then, the K-Means clustering algorithm was used to extract three types of point cloud clusters with different height features from the effective point cloud set, namely, point cloud clusters with unharvested normal height, point cloud clusters with unharvested lower and middle parts, and point cloud clusters with harvested areas. The point cloud clusters with unharvested normal height were retained as the edge point set of the effective point cloud.

[0032] The height characteristics of the three types of areas are as follows: unharvested normal height ≥ 0.5m, unharvested middle and lower part height ≤ 0.5m, and harvested area height ≤ 0.15m.

[0033] Step 7: Use the latitude and longitude scanning method to extract boundary points from the edge point set, generating latitude scans parallel to the x-axis and longitude scans parallel to the y-axis. The latitude scan filters the corresponding y-coordinate point set and sorts them by x to obtain the first and last extreme points. The longitude scan filters the corresponding x-coordinate point set and sorts them by y to obtain the first and last extreme points. The two are then merged to obtain the initial boundary point set.

[0034] Step 8: Fit the initial boundary point set with a third-order B-spline curve, and then use a linear interpolation algorithm to generate the boundary curve using the fitted boundary points. Any two adjacent boundary points... and The equation of the straight line between them is: .

[0035] In the image, this calculation is performed on all adjacent boundary points to generate a continuous curve, which is then superimposed onto the real-time RGB image as a solid red line.

[0036] Step 9: Generate the corresponding navigation path based on the coordinates of the point cloud boundary points. Transform the navigation path from the pixel coordinate system into a sequence of coordinate points in the world coordinate system. Use a sliding window filtering algorithm to smooth the coordinate point sequence and remove inflection points in the path, thus completing the generation of the harvester navigation path.

[0037] When generating navigation paths, the distance R between the sensor and the boundary of the harvester and the overlap r are set according to the harvester's working width W. The distance between the first path and the sensor is between (Rr) and R, and the spacing between adjacent paths is (Wr). For example, if the working width W = 200cm, this value can be precisely set according to the harvester's header width, with a preset overlap of ≥5cm, and the sensor distance from the boundary of the harvester R = 120cm. The distance between the first path and the sensor = R - 5 = 115cm, and the spacing between adjacent paths ≤ W - 5 = 195cm, ensuring complete path coverage without repetition.

Claims

1. A method for generating a harvester navigation path, characterized in that, Includes the following steps: Step 1: Install the industrial RGB camera and ATX LiDAR at the front of the harvester cab, and use Zhang Zhengyou's checkerboard calibration method to correct image distortion of the industrial RGB camera. Step 2: Synchronously acquire RGB images of the calibration board and point cloud data of the ATX LiDAR, and establish the LiDAR point cloud coordinate system using the PnP algorithm. The mapping formula to the camera pixel coordinate system (u, v) is as follows: ； ； Where K is the camera intrinsic parameter matrix, R is the lidar extrinsic parameter rotation matrix, and T is the lidar extrinsic parameter translation vector; Step 3: Simultaneously acquire RGB image data and point cloud data of rice and wheat in the harvester's working area using the industrial RGB camera and ATX LiDAR installed in Step 1. After distortion correction of the RGB images, label the RGB images using a labeling tool to delineate the harvested and unharvested areas. Then clean the dataset and finally export a JSON format labeling file containing area category and area coordinate information. Step 4: Construct the dataset using the JSON format annotation file from Step 3 and divide it into training, validation, and test sets. Then, perform rice and wheat scene adaptation training on the SAM basic model. In the model decoding part, introduce multi-scale and local attention mechanisms, fuse low-level high-resolution features, and add small parallelogram priors to enhance small target feature extraction and anti-interference capabilities. During training, use geometric and color enhancement, edge / IoU weighted loss, and contrastive learning to improve the segmentation accuracy and robustness of small targets. Generate a binary foreground mask corresponding to the RGB image through the model output after training. Step 5: Convert the ATX LiDAR point cloud coordinates to pixel coordinates, filter the point cloud coordinates using the boundary coordinates of the harvested and unharvested areas, and compare them with the binary foreground mask to remove invalid point clouds that exceed the area boundaries, thus obtaining a set of valid point clouds. Step 6: Perform downsampling on the effective point cloud set and use a filtering algorithm to remove isolated points. Then, use a clustering algorithm to extract point cloud clusters of three different height feature regions from the effective point cloud set, namely, unharvested normal height point cloud clusters, unharvested middle and lower part point cloud clusters, and harvested area point cloud clusters. Keep the unharvested normal height point cloud clusters as the edge point set of the effective point cloud. Step 7: Use the latitude and longitude scanning method to extract boundary points from the edge point set, generating latitude scans parallel to the x-axis and longitude scans parallel to the y-axis. The latitude scan filters the corresponding y-coordinate point set and sorts them by x to obtain the first and last extreme points. The longitude scan filters the corresponding x-coordinate point set and sorts them by y to obtain the first and last extreme points. The two are then merged to obtain the initial boundary point set. Step 8: Fit the initial boundary point set with a third-order B-spline curve, then use a linear interpolation algorithm to generate a boundary curve using the fitted boundary points, and then superimpose the boundary curve onto the real-time RGB image. Step 9: Generate the corresponding navigation path based on the coordinates of the point cloud boundary points. Transform the navigation path from the pixel coordinate system into a sequence of coordinate points in the world coordinate system. Use a sliding window filtering algorithm to smooth the coordinate point sequence and remove inflection points in the path, thus completing the generation of the harvester navigation path.

2. The method for generating a harvester navigation path according to claim 1, characterized in that: The image distortion correction method described in step 1 is as follows: using a 7×5 checkerboard calibration board, at least 20 images of the calibration board are captured to ensure that all calibration board images cover the entire field of view of the camera. Then, the OpenCV's findChessboardCorners function is used to detect the corner points within the checkerboard, and the cornerSubPix function is used for sub-pixel refinement. The calibrateCamera function is used to calculate the camera intrinsic parameter matrix K and distortion coefficients D, expressed as: , where f x f y c is the focal length. x c y Principal point coordinates; , where k1, k2, and k3 are radial distortion coefficients, and p1 and p2 are tangential distortion coefficients; Then, the original camera coordinate system Distortion correction is performed to correct the distortion. The correction formula is: ,in .

3. The method for generating a harvester navigation path according to claim 1, characterized in that: In step 2, R is a 3×3 rotation matrix, expressed as: ; T is a 3×1 translation vector, expressed as: .

4. The method for generating a harvester navigation path according to claim 1, characterized in that: In step 2, the ATX LiDAR point cloud is projected onto the RGB image using a mapping formula. The reprojection error between the calibration board point cloud contour and the calibration board edge in the image is calculated. When the average error exceeds the threshold, the installation positions of the industrial RGB camera and ATX LiDAR are readjusted, or the number of calibration images is increased. Then step 2 is repeated until the reprojection error meets the threshold requirement.

5. The method for generating a harvester navigation path according to claim 1, characterized in that: In step 3, the Labelme annotation tool was used to annotate more than 1,000 distortion-corrected rice and wheat images, and the training set, validation set, and test set were divided in a ratio of 8:1:

1.

6. The method for generating a harvester navigation path according to claim 1, characterized in that: The binary foreground mask described in step 4, where a "1" value corresponds to the target region and a "0" value corresponds to the background redundancy region.

7. The method for generating a harvester navigation path according to claim 1, characterized in that: In step 6, a voxel lattice downsampling algorithm is used, with voxel size of 1cm×1cm×1cm, and the number of output point clouds is 1 / 5 of the number of original point clouds.

8. The method for generating a harvester navigation path according to claim 1, characterized in that: In step 6, the K-Means clustering algorithm is used, with K=3 clusters. The height characteristics of the three types of regions are as follows: unharvested normal height ≥ 0.5m, 0.15m < unharvested middle and lower part height ≤ 0.5m, and harvested area height ≤ 0.15m.

9. The method for generating a harvester navigation path according to claim 1, characterized in that: In step 8, any two adjacent boundary points and The equation of the straight line between them is: 。 10. The method for generating a harvester navigation path according to claim 1, characterized in that: In step 9, when generating the navigation path, the distance R between the sensor and the boundary of the harvester and the overlap amount r are set according to the harvester's working width W. The distance between the first path and the sensor is between (Rr) and R, and the distance between adjacent paths is (Wr).

Images