Method for extracting irregular field block border navigation path based on RGB-D multi-modal data

Abstract

The application discloses an irregular field block boundary navigation path extraction method based on RGB-D multi-modal data, which comprises the following steps: constructing an RGB-D ridge data set of a target field block; obtaining a semantic segmentation mask of a ridge area; extracting a boundary line based on density clustering; adaptively extracting a feature point based on a slope; converting the feature point based on multi-modal depth data; fitting the feature point by using a Bezier curve and adopting node smoothing processing to generate a navigation path. Compared with single-modal data, the method can realize higher-precision ridge area segmentation and more accurate navigation path extraction by using multi-modal RGB-D data. The method can be applied to automatic navigation operation of agricultural vehicles along the boundary of irregular field blocks, is suitable for edge sealing operation scenes in the planting operation process, and has important significance for realizing whole-process unmanned operation in small field areas. The method can adapt to different ridge shapes, different field block types and different agricultural vehicles.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of path planning of agricultural machinery field operation, and particularly relates to an irregular field block border navigation path extraction method based on RGB-D multi-modal data. BACKGROUND

[0002] With the rise of artificial intelligence technology, unmanned agricultural vehicles are becoming a research hotspot in the field of agricultural engineering. The core of unmanned technology is automatic navigation. At present, the research on straight-line navigation of agricultural vehicles in regular farmland is relatively mature, such as global path planning based on GNSS and crop row tracking based on vision. In the planting operation process of agricultural vehicles, in addition to field operation, edge sealing operation is often required to ensure the maximum utilization of land resources. In southern China, most of the farmland is built according to the terrain, so the shape of the farmland is irregular, and the edge sealing operation requires the agricultural vehicle to travel along the irregular field block edge. In this scenario, if the autonomous curve navigation of the agricultural vehicle can be realized, and supplemented by the relatively mature straight-line navigation in the field, both field operation and edge sealing operation can be unmanned, thereby realizing the whole process of unmanned operation in small field areas, which is of great significance. In addition, compared with straight-line navigation, curve navigation is more difficult but more adaptable.

[0003] Automatic navigation technology can be generally divided into GNSS navigation and visual navigation. GNSS navigation is easily limited by the operation environment because it relies on satellites, and cannot accurately judge the field information in real time. Visual navigation effectively makes up for the shortcomings of GNSS navigation, can perceive and make effective judgments on field information in real time, and has low cost. In addition, thanks to the real-time nature of visual navigation, it can generate a navigation curve according to the actual situation in the field in real time in a curve operation scenario, which has high accuracy and is very suitable for the automatic navigation scenario of the agricultural vehicle along the edge of the field block.

[0004] Visual navigation often needs to use a specific image post-processing algorithm to analyze image frames to detect the navigation reference line and generate a navigation line accordingly. Common navigation scenarios (such as field road navigation and crop row tracking) mostly have two navigation reference lines, while for navigation along the edge of the field block, there is only one navigation reference line, i.e. the inside boundary line of the dike. Due to the reduction of the number of reference lines, the difficulty of navigation line extraction is greater, so there are fewer related researches. At the same time, due to the lack of one reference line, the navigation line extraction in this scenario often needs a higher-precision image segmentation method, otherwise it will seriously affect the navigation line extraction accuracy.

[0005] Traditional image segmentation methods often employ machine vision techniques based on pixel color and brightness, which lack versatility, are highly susceptible to environmental influences, and cannot be widely applied. With the development of deep learning technology, neural networks can autonomously learn the features of each pixel in an image, achieving high-precision image segmentation with strong generalization capabilities. However, for certain field scenes, such as dry fields where the ridges and farmland areas have highly similar color and texture features, using a neural network that relies solely on RGB information as input can easily missegment some easily confused pixels.

[0006] Therefore, for the navigation scenario of irregular small field edges with only one navigation reference line, there is an urgent need to develop a high-precision navigation path extraction method for agricultural vehicles that is applicable to various types and shapes of fields to achieve curve navigation. Summary of the Invention

[0007] To address the challenge of autonomous navigation of agricultural vehicles along the boundaries of irregularly shaped small fields, this invention provides a method for extracting navigation paths along the boundaries of irregular fields based on RGB-D multimodal data. Taking into account the significant height difference between the field ridge area and the farmland area, an RGB-D multimodal input semantic segmentation model can achieve higher segmentation accuracy, making it suitable for edge navigation where higher segmentation precision is required. By employing slope-based adaptive feature point extraction and multimodal depth data-based feature point transformation, a smooth, continuous navigation path curve can be accurately extracted. This path curve is unaffected by small protrusions on the field ridge boundary, conforms to the shape of the field ridge, and is located in the center of the camera's field of view.

[0008] The objective of this invention is achieved through the following technical solution:

[0009] A method for extracting navigation paths along the boundaries of irregular fields based on RGB-D multimodal data includes the following steps:

[0010] S01. Construct an RGB image of the field ridge section of target field A using an agricultural vehicle equipped with an RGB-D camera. RGB and the corresponding depth image A DEPTH During image acquisition, agricultural vehicles simulate real field edge operation scenarios, and the A RGB and the A mentioned above DEPTH It includes complete and clearly defined field ridge areas and uncultivated farmland areas;

[0011] S02. Place the A RGB and the A mentioned above DEPTH The input is fed into the trained RGB-D semantic segmentation model, and the output contains only the A. RGB and the A mentioned above DEPTH Binary mask image G of the predicted values ​​in the Zhongtiangeng region mask ;

[0012] S03. For the binary mask image G mask Contour line extraction and image edge line filtering are performed to obtain a cluster of field ridge boundary lines L that does not contain image edge lines. ridge A density-based clustering method is used to cluster the L... ridge Perform clustering and calculate the centroid (x) of the boundary line for each class. i ,y i ) and the G mask Image center point (x) c ,y c Distance D i Select the one with the smallest D i The field ridge boundary line is used as the working boundary line L. w-ridge This completes the boundary line extraction;

[0013] S04. Regarding the L w-ridge The contour points are grouped into groups of n, and the slope between the first and last points of each group is denoted as . For each slope point, group them again, with each group consisting of m points, and calculate the variance. Let the variance of each group be V. i At this point, each variance point corresponds to the L mentioned above. w-ridge The previous contour point interval is [imn, (i+1)mn]; set the threshold T. d Threshold filtering is performed on the variance, and the variance value V is selected. i Below the threshold T d The midpoint of the contour point interval corresponding to the variance point is taken as the feature point P. feat ; Let P be an adjacent feature point P. feat The slope is Based on the slope difference, redundant intervals are identified, and all feature points in the non-redundant intervals and a subset of feature points within the redundant intervals are selected as the final feature point P. c-feat This completes the feature point extraction;

[0014] S05. Collect the A RGB and the A mentioned above DEPTH Extract the P data information. c-feat The pixel coordinates and depth values; combined with the calibration parameters and coordinate transformation formula of the RGB-D camera, the P is obtained. c-feat World coordinates; select the G mask The three points C, D, and E located at the center of the image determine the normal vector of the field plane. Let each feature point P be... c-feat Let A be the point, B be the βth point after A, and δ be the actual distance from the camera center to the edge of the field ridge. Then, according to the formula... The transformed feature points TP were calculated.c-feat world coordinates; according to a coordinate inverse transformation formula, the TP c-feat pixel coordinates are obtained, so as to complete the feature point conversion.

[0015] S06. Selecting every 3 TP c-feat s as a group for Bezier curve fitting, and connecting the start and end of each curve; dividing each fitted curve into l small segments, selecting the nodes of each Bezier curve and the end points of the adjacent two small segments for Bezier curve fitting again, replacing the newly fitted curve at the original curve, thereby generating a navigation path curve for the agricultural vehicle to travel along the boundary in the target field A.

[0016] Further, in the step S02, the A RGB and the A DEPTH are input into the trained RGB-D semantic segmentation model, and further comprising: obtaining RGB images and depth images of the farmland ridge area under different weather conditions and different plots, and labeling the ridge area in the RGB image as a training sample; training the RGB-D semantic segmentation model based on multiple training samples to obtain the trained RGB-D semantic segmentation model.

[0017] Further, in the step S04, if there are α feature points in the redundant interval, only feature points are finally selected, and it should be ensured that the feature points are uniformly selected in the redundant interval. represents the maximum integer not exceeding .

[0018] Further, in the step S05, the camera calibration parameters are the RGB-D camera intrinsic parameters c x , c y , f x , f y and extrinsic parameters r ij , t x , t y , t z obtained by a camera calibration method in advance; the coordinate transformation formula is and the coordinate inverse transformation formula is the reverse operation of the coordinate transformation formula.

[0019] Further, in the step S06, the finally obtained navigation path curve has the characteristics of smooth continuity, not being disturbed by small protruding parts on the ridge boundary line, conforming to the shape of the ridge, and being located in the central field of view of the camera.

[0020] Further, in the step S02, the RGB-D semantic segmentation model is ESANet.

[0021] Further, the target field plot A is a water field and a dry field without planting crops, and is a small irregularly shaped field plot.

[0022] The beneficial effects of the present application are that the method realizes the extraction of the boundary navigation path of the agricultural vehicle in the irregular field plot, can be applied to the automatic navigation of the agricultural vehicle along the boundary of the irregular field plot in the edge closing operation process, and has important significance for realizing the whole process unmanned operation in the small field area. Compared with the method based on GNSS navigation, the method can generate a suitable navigation curve in real time according to the actual shape of the field ridge. Compared with the traditional navigation method, the method can complete the navigation path planning in the scene with only one navigation reference line. Compared with the single modal data, the method using the multi-modal RGB-D data can realize higher precision of the field ridge area segmentation and more accurate navigation path extraction. The method is suitable for the edge navigation path planning of the agricultural vehicle in the water field and the dry field, is suitable for different field ridge shapes and different types of agricultural vehicles, and has high popularization and application value. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 It is a flowchart of an irregular field plot boundary navigation path extraction method based on RGB-D multi-modal data in an embodiment of the present application.

[0024] Figure 2 It is an RGB image of an example test field plot in an embodiment of the present application.

[0025] Figure 3 It is a depth image of an example test field plot in an embodiment of the present application.

[0026] Figure 4 It is a semantic segmentation mask of an example test field plot obtained using ESANet in an embodiment of the present application.

[0027] Figure 5 It is a schematic diagram after contour line extraction of the semantic segmentation mask in an embodiment of the present application.

[0028] Figure 6 It is a field ridge boundary line cluster L without image edge lines in an embodiment of the present application. ridge schematic diagram.

[0029] Figure 7 It is a field ridge boundary line cluster L after density-based clustering in an embodiment of the present application. ridge schematic diagram.

[0030] Figure 8 It is a working boundary line L finally selected in an embodiment of the present application. w-ridge schematic diagram.

[0031] Figure 9The feature point P extracted based on the slope in the example of the present application feat schematic diagram.

[0032] Figure 10 The final feature point P after the simplified processing in the example of the present application c-feat schematic diagram.

[0033] Figure 11 The schematic diagram of the feature point conversion related calculation process in the example of the present application.

[0034] Figure 12 The feature point TP after the feature point conversion step in the example of the present application c-feat schematic diagram.

[0035] Figure 13 The schematic diagram of the navigation path curve after the segmented Bezier curve fitting in the example of the present application.

[0036] Figure 14 The schematic diagram of the final navigation path curve after the node smoothing processing in the example of the present application.

[0037] Figure 15 The mask diagram of each step in the process of the navigation path curve planned for the example test field in the example of the present application. DETAILED DESCRIPTION

[0038] The specific embodiments of the application are further described below with reference to the accompanying drawings, and the following examples are only used to more clearly illustrate the technical solutions of the present application, and cannot be used to limit the protection scope of the present application, and the following examples select the image of the dry field ridge part of the un-planted crops in the natural field environment as an example for detailed description.

[0039] Figure 1 The flowchart of the irregular field along the boundary navigation path extraction method based on the RGB-D multi-modal data provided by the embodiment of the present application is given, and the steps included in the method are specifically as follows:

[0040] S01. Use the agricultural vehicle equipped with the RGB-D camera to construct the RGB image A of the ridge part of the target field A under different weather conditions and different field types RGB and the corresponding depth image A DEPTH In the image acquisition process, the agricultural vehicle simulates the real field edge operation scene, and the A RGB and the A DEPTH include complete and clear ridge area and non-crop farmland area, and the constructed RGB image and depth image of the target field are as shown in Figure 2 and Figure 3 ;

[0041] S02. Label the ridge region in the acquired RGB image of field plot A as a training sample in the Labelme annotation software using the region label; based on multiple training samples, train the RGB-D semantic segmentation model, where the semantic segmentation model is selected as ESANet. Input A RGB and A DEPTH into the trained RGB-D semantic segmentation model, and output a binary mask image containing only the predicted value of the ridge region in A RGB and A DEPTH , as shown in Figure 4 ;

[0042] S03. The binary mask image is processed by contour line extraction, image edge line filtering and density-based clustering, so as to extract the ridge boundary line closest to the image center in the binary mask image. The specific steps are as follows:

[0043] a01. Use the findContours function in the open source algorithm library OpenCV in Python to perform contour line extraction on the binary mask image, as shown in Figure 5 ;

[0044] a02. Remove the contour points with vertical coordinates h and horizontal coordinates w (h and w are the height value and width value of the binary mask image, respectively) from the extracted contour lines, to obtain the ridge boundary line cluster L ridge , as shown in Figure 6 ;

[0045] a03. Use the density-based clustering method DBSCAN to cluster the ridge boundary line cluster L ridge , as shown in Figure 7 , where each class of boundary line is represented by a different color;

[0046] a04. Calculate the distance D i between the centroid point (x i , y i ) of each class of boundary line and the center point (x c , y c ) of the binary mask image, and select the ridge boundary line with the smallest D i value as the working boundary line L w-ridge , so as to extract the ridge boundary line closest to the image center in the binary mask image, as shown in Figure 8 ;

[0047] S04. Feature point extraction is performed on the extracted working boundary line using a slope-based method, and feature point simplification is performed.

[0048] The specific steps are as follows:

[0049] a01. Grouping every n contour points on the working boundary line, and recording the slope between the first and last points of each group as Grouping each slope point again, and taking m points as a group to calculate the variance, and recording the variance of each group as V i At this time, each variance point corresponds to L w-ridge The last contour point interval [imn, (i+1)mn]; set threshold T d Threshold screening is performed on the variance points, and the variance value V i below the threshold T d The midpoint of the contour point interval corresponding to the variance point below the threshold T feat is taken as the feature point P Figure 9 .

[0050] a02. Record the slope between the adjacent two feature points P feat According to the slope difference, judge the redundant interval, and select all feature points in the non-redundant interval and part of the feature points in the redundant interval as the final feature point P c-feat , in particular, if there are α feature points in the redundant interval, then only feature points are selected, and it should be ensured that they are uniformly selected in the redundant interval represents the maximum integer not exceeding

[0051] The simplified final feature points are shown in Figure 10 .

[0052] S05. Collect data information in A RGB and A DEPTH , extract the pixel coordinates and depth values of P c-feat ; combine the RGB-D camera intrinsic parameters c x , c y , f x , f y and the extrinsic parameters r ij , t x , t y , t z obtained by the camera calibration method in advance and the coordinate transformation formula to obtain the world coordinates of P c-feat , and the calculation process is shown in Figure 11 (a) part; select three points C, D, E in the center of the binary mask image to determine the field plane normal vector Record each feature point P c-feat as point A, record the (β+1)th point after point A as point B, and record the real distance of the camera center from the edge of the ridge as δ, then the world coordinates of the converted feature point TP c-feat can be calculated according to the formula , and the calculation process is shown in Figure 11 ​The pixel coordinates of the TP are obtained according to the coordinate inverse transformation formula, so that the feature point conversion is completed, and the calculation process is as shown in part (c) of the figure c-feat Figure 11 The pixel coordinates of the TP are obtained according to the coordinate inverse transformation formula, so that the feature point conversion is completed, and the calculation process is as shown in part (c) of the figure c-feat Figure 12

[0053] S06. Select every 3 TP c-feat as a group to perform Bezier curve fitting, and each curve is connected at the head and tail to obtain a navigation path curve as shown in part (d) of the figure Figure 13 ; divide each fitting curve into l small segments, select the nodes of each Bezier curve, the end points of the adjacent two small segments of the nodes, and perform Bezier curve fitting again, replace the newly fitted curve at the original curve position, so as to complete the node smoothing processing, and the finally processed navigation path curve is as shown in part (e) of the figure Figure 14 , which can provide a reference navigation line for the agricultural vehicle to travel along the boundary curve in the target field A. Figure 15 The mask map is fused in each step of the navigation path curve extraction process, so that the characteristics of the navigation path curve extracted by the method of the present application, such as smoothness and continuity, non-interference of small protruding parts on the ridge boundary line, conformity to the ridge shape and location in the central part of the camera field of view, can be more intuitively displayed.

[0054] The above-described embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it cannot be understood as a limitation on the scope of the application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which all belong to the protection scope of the present application. The protection scope of the present application is given by the appended claims and any equivalent technical solutions thereof.​​​

Claims

1. A method for extracting navigation paths along the boundaries of irregular fields based on RGB-D multimodal data, characterized in that, The method includes the following steps: S01. Construct an RGB image of the field ridge section of target field A using an agricultural vehicle equipped with an RGB-D camera. RGB and the corresponding depth image A DEPTH During image acquisition, agricultural vehicles simulate real field edge operation scenarios, and the A RGB and the A mentioned above DEPTH It includes complete and clearly defined field ridge areas and uncultivated farmland areas; S02. Place the A RGB and the A mentioned above DEPTH The input is fed into the trained RGB-D semantic segmentation model, and the output contains only the A. RGB and the A mentioned above DEPTH Binary mask image G of the predicted values ​​in the Zhongtiangeng region mask ; S03. For the binary mask image G mask Contour line extraction and image edge line filtering are performed to obtain a cluster of field ridge boundary lines L that does not contain image edge lines. ridge A density-based clustering method is used to cluster the L... ridge Perform clustering and calculate the centroid (x) of the boundary line for each class. i ,y i ) and the G mask Image center point (x) c ,y c Distance D i Select the one with the smallest D i The field ridge boundary line is used as the working boundary line L. w-ridge This completes the boundary line extraction; S04. Regarding the L w-ridge Group the contour points into groups of n, and denote the slope between the first and last points of each group as . For each slope point, group them again, with each group consisting of m points, and calculate the variance. Let the variance of each group be V. i At this point, each variance point corresponds to the L mentioned above. w-ridge The previous contour point interval is [imn, (i+1)mn]; set the threshold T. d Threshold filtering is performed on the variance, and the variance value V is selected. i Below the threshold T d The midpoint of the contour point interval corresponding to the variance point is taken as the feature point P. feat ; Let P be an adjacent feature point P. feat The slope is Based on the slope difference, redundant intervals are identified, and all feature points in the non-redundant intervals and a subset of feature points within the redundant intervals are selected as the final feature point P. c-feat This completes the feature point extraction; S05. Collect the A RGB and the A mentioned above DEPTH Extract the P data information. c-feat The pixel coordinates and depth values; combined with the calibration parameters and coordinate transformation formula of the RGB-D camera, the P is obtained. c-feat World coordinates; select the G mask The three points C, D, and E located at the center of the image determine the normal vector of the field plane. Let each feature point P be... c-feat Let A be the point, B be the βth point after A, and δ be the actual distance from the camera center to the edge of the field ridge. Then, according to the formula... The transformed feature points TP were calculated. c-feat The world coordinates; according to the inverse coordinate transformation formula, the TP is obtained. c-feat The pixel coordinates are used to complete the feature point transformation; S06. Select every 3 TPs. c-feat A Bezier curve is fitted to a group of curves, with each curve segment connected end to end. Each fitted curve segment is divided into l segments. The nodes of each Bezier curve segment and the endpoints of the two adjacent segments are selected and Bezier curves are fitted again. The newly fitted curve is then replaced with the original curve, thereby generating a navigation path curve for agricultural vehicles to travel along the boundary within the target field A.

2. The method according to claim 1, characterized in that, In step S02, the A RGB and the A mentioned above DEPTH Before inputting into the trained RGB-D semantic segmentation model, the process further includes: acquiring RGB and depth images of farmland ridge areas under different weather conditions and on different plots, and labeling the ridge areas in the images as training samples; training the RGB-D semantic segmentation model based on multiple training samples to obtain the trained RGB-D semantic segmentation model.

3. The method according to claim 1, characterized in that, In step S04, if there are α feature points within the redundant interval, then only α feature points are ultimately selected. Each feature point should be selected uniformly within the redundant interval. Indicates no more than The largest integer.

4. The method according to claim 1, characterized in that, In step S05, the camera calibration parameters are the RGB-D camera intrinsic parameters c obtained in advance through the camera calibration method. x c y f x f y and external parameters r ij t x t y t z The coordinate transformation formula is: and The inverse coordinate transformation formula is the reverse application of the coordinate transformation formula.

5. The method according to claim 1, characterized in that, In step S06, the final obtained navigation path curve is characterized by being smooth and continuous, unaffected by small protrusions on the boundary line of the field ridge, conforming to the shape of the field ridge, and located in the center of the camera's field of view.

6. The method according to claim 1, characterized in that, In step S02, the RGB-D semantic segmentation model is ESANet.

7. The method according to claim 1, characterized in that, The target field A is an uncultivated paddy field and dry field, and is a small field with an irregular shape.

Images