Method for planning obstacle avoidance path along boundary of irregular field parcel based on RGB-D multi-modal data

Abstract

The invention discloses a method for planning an obstacle avoidance path along a boundary of an irregular field parcel based on RGB-D multi-modal data. The method comprises the following steps: constructing an RGB-D obstacle avoidance data set of a target field parcel; obtaining semantic segmentation masks of ridge and obstacle areas; establishing a point cloud based on the multi-modal RGB-D data; point cloud matching based on coordinate mapping; point cloud projection based on coordinate system transformation; path planning based on an improved artificial potential field method; and generating an obstacle avoidance path based on path inverse projection of coordinate inverse transformation. Compared with single-mode data, the method uses multi-mode RGB-D data to realize higher-precision ridge and obstacle area segmentation and more accurate obstacle avoidance path planning. The method can be applied to the automatic obstacle avoidance operation of the agricultural vehicle along the irregular field boundary, is suitable for the edge sealing operation scene of the planting operation, and is of great significance to the realization of the full-scene unmanned edge sealing operation of the small field area. The method can adapt to different ridge shapes and field types, different obstacle shapes and positions and different agricultural vehicles.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of path planning of agricultural machinery field operation, and particularly relates to an irregular field block border obstacle avoidance path planning method based on RGB-D multi-modal data. BACKGROUND

[0002] With the rise of artificial intelligence technology, unmanned agricultural vehicles are becoming a research hotspot in the field of agricultural engineering. The core of unmanned technology is automatic navigation. In the planting operation process of agricultural vehicles, in addition to field operation, edge sealing operation is often required to maximize the use of land resources. In southern China, most of the areas are hilly and mountainous, and farmland is built according to the terrain, so most of the farmland is irregular small field blocks. At this time, the completion of the edge sealing operation requires the agricultural vehicle to travel along the irregular field block edge. On the other hand, the farmland environment is complex and changeable, and most of them are unstructured scenes, especially there may be some obstacles at the edge of the field block. At this time, if the autonomous navigation and obstacle avoidance of the agricultural vehicle can be realized, the agricultural vehicle can automatically navigate and work around obstacles when encountering obstacles, and the full-scene unmanned edge sealing operation in small field areas can be realized, which is of great significance to agricultural production.

[0003] Unlike navigation path planning, obstacle avoidance path planning often needs to establish a field scene map first, and then plan the path on this map. For scene map construction, there are currently two schemes based on laser radar and vision. Although the commonly used laser radar-based obstacle avoidance technology has high accuracy and is relatively mature, the hardware cost is too high, which is seriously inconsistent with the actual agricultural production. The visual obstacle avoidance technology has lower cost and stronger real-time performance, and is more suitable for irregular field border obstacle avoidance operation scenes. However, the current visual obstacle avoidance technology is not mature, especially for border obstacle avoidance operation in farmland scenes, and targeted research is urgently needed.

[0004] For visual obstacle avoidance, scene segmentation and path planning are two key technologies that determine the accuracy of obstacle avoidance. Obstacle avoidance scene segmentation will directly affect the accuracy of the field scene map construction, and path planning will affect the final obstacle avoidance effect.

[0005] Traditional scene segmentation methods mostly use machine vision technology based on pixel color, brightness, etc., which has poor universality and is greatly affected by the environment, and cannot be widely applied. With the development of deep learning technology, neural networks can automatically learn the features of each pixel in the image, and high-precision scene segmentation with strong generalization ability can be achieved. However, for some field scenes, such as dry fields with highly similar color features and texture features in the ridge area and farmland area, the neural network using only RGB information as input is easy to misclassify some confusing pixels.

[0006] On the other hand, obstacle avoidance path planning can be generally divided into global obstacle avoidance path planning and local obstacle avoidance path planning. The scene map required for global obstacle avoidance path planning is the entire field scene, and the mapping of the entire scene needs to be completed in advance before path planning can be performed. Although this method can cover more scene information, it has poor real-time performance. The principle of local obstacle avoidance path planning is to construct a scene map in a frame in real time through a vision camera and simultaneously perform obstacle avoidance path planning. Although a single frame contains less scene information, this method has strong real-time performance, can adjust the obstacle avoidance path in real time according to the actual situation of the field scene, and is more suitable for obstacle avoidance tasks in complex field environments. However, the application of common local obstacle avoidance path planning algorithms such as the BUG method, the visible graph method, the artificial potential field method, the VFH method, and the dynamic window method in farmland environments has certain limitations, and related research is insufficient.

[0007] Therefore, for the more special field scene of obstacle avoidance operation along the irregular field boundary, only using existing algorithms is obviously not applicable, and it is necessary to propose a new local obstacle avoidance path planning method with high precision and strong adaptability according to the operation characteristics and the advantages of various classic algorithms to adapt to this special scene. SUMMARY

[0008] To solve the problem of autonomous obstacle avoidance of agricultural vehicles along the boundary in irregular small field areas, the present application provides an irregular field boundary obstacle avoidance path planning method based on RGB-D multi-modal data. In view of the obvious height difference between the ridge, obstacles and farmland area, the RGB-D multi-modal input semantic segmentation model can realize higher precision segmentation. The path planning algorithm based on the improved artificial potential field method and the path inverse projection algorithm can realize high-precision obstacle avoidance path planning and accurate mapping to the front view image, and is widely applicable to various scenes.

[0009] The purpose of the present application is achieved by the following technical solutions:

[0010] An irregular field boundary obstacle avoidance path planning method based on RGB-D multi-modal data, comprising the following steps:

[0011] S01. An agricultural vehicle equipped with an RGB-D camera is used to construct an RGB image A of the ridge and obstacle part of the target field A RGB and a corresponding depth image A DEPTH During the image acquisition process, the agricultural vehicle simulates a real field edge obstacle avoidance operation scene, and the A RGB and the A DEPTH contain complete and clear ridge area, obstacle area and farmland area without planted crops;

[0012] S02. The A RGB and the A DEPTHThe input is fed into the trained RGB-D semantic segmentation model, and the output contains only the A. RGB and the A mentioned above DEPTH Mask image G of the predicted values ​​of the Nakada Ridge and obstacle areas mask ;

[0013] S03. Collect the A RGB and the A mentioned above DEPTH From the data information, extract the pixel coordinates and depth values ​​of each pixel; combine the calibration parameters and coordinate transformation formula of the RGB-D camera to obtain the world coordinates (X, Y, Z) of each pixel. w Y w Z w ); using the aforementioned A RGB The R, G, and B values ​​of each pixel are matched with color information, and the spatial location information of each pixel is combined with the color information to complete the point cloud construction.

[0014] S04. Extract the G mask The pixel coordinate set of each category in the point cloud is mapped to the three-dimensional point coordinate set of each category in the point cloud based on the mapping relationship between pixel coordinates and point cloud, thereby determining the category to which each point in the point cloud belongs, thus completing point cloud matching;

[0015] S05. Select three points C, D, and E located at the center of the matched point cloud image and use the plane formed by them as the projection reference plane; determine the plane equation ax + by + cz + d = 0 and the plane normal vector based on the coordinates of the three points. Calculate the distance D from each 3D point to the plane. e And obtain the projected coordinates (X) of each point according to the projection formula. pro Y pro Z pro Select the A RGB The three-dimensional point O corresponding to the bottom center point is taken as the origin, and two points A and B are determined at the bottom. The direction is taken as the x-axis direction. Establish a Cartesian coordinate system with the y-axis as the direction, and obtain the Cartesian coordinates (X, Y, y) of each point using the coordinate transformation formula. rec Y rec The points are plotted in a planar coordinate system; the contours of the field ridges and obstacle category points are fitted using envelopes, and the contours represent the field ridges and obstacles. The origin of the coordinate system represents the camera position (i.e., the position of the equipment), resulting in a two-dimensional obstacle avoidance coordinate map G. 2D-obs This completes the point cloud projection;

[0016] S06. In the aforementioned G 2D-obs In this study, based on the improved artificial potential field method, the improved repulsive potential field function U is analyzed. req (x) and gravitational potential function Uatt (x) is obtained, the direction of the attractive force points to the target point, the direction of the repulsive force points to the opposite direction of the obstacle point, the vector sum of the attractive force and the repulsive force is the resultant force at the point, then the resultant force of the machine at each position is calculated, thereby completing the path planning;

[0017] S07. Collecting the planar rectangular coordinates (X rec,p , Y rec,p ) of each point on the planned obstacle avoidance path, combining the calibration parameters of the RGB-D camera and the coordinate inverse transformation formula, obtaining the pixel coordinates (u p , v p ) of each point; according to the pixel coordinates of each point in the A RGB , redraw to generate the final obstacle avoidance path, thereby completing the path inverse projection, and generating the obstacle avoidance path for the agricultural vehicle to travel along the boundary in the target field A.

[0018] Further, in the step S02, before the A RGB and the A DEPTH are input to the trained RGB-D semantic segmentation model, the method further comprises: obtaining RGB images and depth images of the field ridge and obstacle area of different weather conditions and different plots, and labeling the field ridge and obstacle area in the images as training samples; based on a plurality of training samples, training the RGB-D semantic segmentation model to obtain the trained RGB-D semantic segmentation model.

[0019] Further, in the step S05, the three-dimensional point to plane distance calculation formula is The projection formula is The coordinate system transformation formula is

[0020] Further, in the step S06, the improved repulsive potential field function U req (x) = U req,obs (x) + U req,hea (x), which is superimposed by the obstacle area repulsive potential field and the field ridge area repulsive potential field, wherein the obstacle area repulsive potential field The field ridge area repulsive potential field The attractive potential field function

[0021] Further, in the step S07, the coordinate inverse transformation formula is And

[0022]

[0023] Further, in the step S07, the finally obtained obstacle avoidance path has the characteristics of being located in the center of the image, entering the obstacle gently, exiting the obstacle to quickly restore the navigation state, and smooth path.

[0024] Further, in the step S02, the RGB-D semantic segmentation model is ESANet.

[0025] Further, the target field block A is a paddy field and a dry field without planting crops, and is an irregularly shaped small field block, and there are obstacles of different shapes and different positions at the edges of the field block.

[0026] The beneficial effects of the present application are that the method realizes the planning of the boundary obstacle avoidance path of the agricultural vehicle in the irregular field block, can be applied to the automatic obstacle avoidance of the agricultural vehicle along the boundary of the irregular field block in the edge sealing process, and has important significance for realizing the unmanned edge sealing operation of the whole scene in the small field area. Compared with the obstacle avoidance method based on laser radar, the method can realize real-time mapping and generation of a suitable obstacle avoidance path according to the field scene. Compared with the traditional obstacle avoidance path planning method, the method can realize the local obstacle avoidance path planning with the characteristics of being located in the center of the image, entering the obstacle gently, exiting the obstacle to quickly restore the navigation state, and smooth path. Compared with single-mode data, the method using multi-modal RGB-D data can realize higher-precision ridge and obstacle region segmentation and more accurate obstacle avoidance path planning. The method is suitable for the boundary obstacle avoidance path planning of agricultural vehicles in paddy fields and dry fields; is suitable for different ridge shapes, different field types, different obstacle shapes, different obstacle positions, and different types of agricultural vehicles; and has high popularization and application value. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 It is an RGB-D multi-modal data-based irregular field boundary obstacle avoidance path planning method flowchart in the present application example.

[0028] Figure 2 It is an RGB image of an example test field block in the present application example.

[0029] Figure 3 It is a depth image of an example test field block in the present application example.

[0030] Figure 4 It is a semantic segmentation mask of an example test field block obtained by using ESANet in the present application example.

[0031] Figure 5 It is a general flowchart of point cloud establishment in the present application example.

[0032] Figure 6 It is a schematic diagram of the final result of point cloud matching in the present application example.

[0033] Figure 7 A schematic diagram of the related calculation process in the point cloud projection in the example of the present application.

[0034] Figure 8 A two-dimensional obstacle avoidance coordinate graph G finally generated by the point cloud projection in the example of the present application 2D-obs A schematic diagram.

[0035] Figure 9 A schematic diagram of the overall flow of the path planning algorithm based on the improved artificial potential field method in the example of the present application.

[0036] Figure 10 A schematic diagram of the related calculation process in the path inverse projection in the example of the present application.

[0037] Figure 11 A schematic diagram of the obstacle avoidance path finally generated by the path inverse projection in the example of the present application. DETAILED DESCRIPTION

[0038] The specific embodiments of the application will be further described below in conjunction with the accompanying drawings, and the following examples are only used to more clearly illustrate the technical solutions of the present application, and cannot be used to limit the protection scope of the present application. The following examples select part of the images of the water field ridge and the obstacle in the natural field environment without planted crops as examples for detailed description.

[0039] Figure 1 A flowchart of an irregular field border obstacle avoidance path planning method based on RGB-D multi-modal data provided by the embodiment of the present application is given, and the steps included in the method are specifically as follows:

[0040] S01. An RGB image A of the ridge and obstacle part of a target field A under different weather conditions and different field types is constructed by using an agricultural vehicle carrying an RGB-D camera RGB and a corresponding depth image A DEPTH During the image acquisition process, the agricultural vehicle simulates a real field edge obstacle avoidance operation scene, and the A RGB and the A DEPTH contain complete and clear ridge areas, obstacle areas and non-crop farmland areas, and the constructed RGB image and depth image of the target field are as shown in Figure 2 and Figure 3 .

[0041] S02. The ridge and obstacle areas in the acquired RGB image of the field A are labeled as training samples in the Labelme labeling software using the area label; based on a plurality of training samples, the RGB-D semantic segmentation model is trained, and here the semantic segmentation model selects ESANet. A RGB and A DEPTHInput into the trained RGB-D semantic segmentation model, output only contains A RGB and A DEPTH Mask image G of the predicted value of the ridge and obstacle area mask As Figure 4 shown.

[0042] S03. Collect the data information of the A RGB and the A DEPTH , extract the pixel coordinates (u, v) and depth value d of each pixel point; combine the calibration parameters of the RGB-D camera and the coordinate transformation formula to obtain the world coordinates (X w , Y w , Z w ) corresponding to each pixel point, the calculation process is shown in part (b) of Figure 5 ; Use the R, G, B value matching color information of each pixel point in the A RGB , combine the spatial position information of each pixel point with the color information, so as to complete the point cloud establishment; the final point cloud graph is shown in part (c) of Figure 5 .

[0043] S04. Extract the pixel coordinate set of each category of pixel point in the G mask , according to the mapping relationship f between the pixel coordinates and the point cloud, map the three-dimensional point coordinate set corresponding to each category in the point cloud graph, determine the category of each point in the point cloud, so as to complete the point cloud matching; the final point cloud matching result is shown in Figure 6 .

[0044] S05. Perform point cloud projection based on coordinate system transformation on the point cloud graph after completing point cloud matching, so as to extract the two-dimensional obstacle avoidance coordinate graph, the specific steps are as follows:

[0045] a01. Select three points C, D, E in the center of the matched point cloud graph to ensure that the plane formed by them can represent the field plane (here the field is approximated as a plane), and take the plane formed by them as the projection reference surface;

[0046] a02. Determine the plane equation ax+by+cz+d=0 of the reference surface and its normal vector according to the coordinates of the three points e , and finally get the projected point coordinates (X pro , Y pro , Z pro ) according to the projection formula, the calculation process is shown in part (a) of Figure 7 ;

[0047] a03. Select the A RGBThe three-dimensional point O corresponding to the bottom center point is taken as the origin of the plane rectangular coordinate system, and two points A and B are determined at the bottom, so that the direction is taken as the x-axis direction, the direction is taken as the y-axis direction, and a plane rectangular coordinate system is established, and the plane rectangular coordinates (X rec , Y rec ) of each point are obtained according to the coordinate system conversion formula and plotted in the plane coordinate system, and the calculation process is shown in part (b) of Figure 7 ;

[0048] a04. The contour of the ridge and the obstacle category point set is fitted using the envelope line, the contour line is used to represent the ridge and the obstacle, the origin of the coordinate system represents the camera position (i.e. the position of the machine tool), and a two-dimensional obstacle avoidance coordinate graph G 2D-obs is obtained, so as to complete the point cloud projection,

[0049] The finally generated two-dimensional obstacle avoidance coordinate graph is shown in Figure 8 .

[0050] S06. In the G 2D-obs , based on the improved artificial potential field method, the magnitude of the attraction and repulsion of the machine tool at a point can be obtained by taking the negative gradient of the improved repulsive potential field function U req (x) and the attractive potential field function U att (x), the direction of the attraction points to the target point, and the direction of the repulsion points to the opposite direction of the obstacle point, and the vector sum of the attraction and repulsion is the resultant force at the point. Then the resultant force of the machine tool at each position is calculated, so as to complete the path planning; wherein the improved repulsive potential field function U req (x) = U req,obs (x) + U req,hea (x) is obtained by superimposing the obstacle region repulsive potential field and the ridge region repulsive potential field, and the calculation process is shown in part (d) of Figure 9 .

[0051] S07. The planned obstacle avoidance path is subjected to path inverse projection based on coordinate inverse transformation, and the path is remapped to the front view image, and the specific steps are as follows:

[0052] a01. Collect the plane rectangular coordinates (X rec,p , Y rec,p ) of each point on the planned obstacle avoidance path, and convert the plane rectangular coordinates of each point on the obstacle avoidance path into the projection coordinates (X pro,p , Y pro,p , Z pro,p ) in the world coordinate system using the inverse projection formula, and the calculation process is shown in part (a) of Figure 10 ;

[0053] a02. Since the field plane is chosen as the reference plane for the planar projection, the obstacle avoidance paths after conversion to projected coordinates all lie within the field plane, which conforms to the actual scene. Therefore, the projected coordinates are regarded as the true three-dimensional coordinates (X, Y, Z) before the planar projection. w,p Y w,p Z w,p )=(X pro,p Y pro,p Z pro,p );

[0054] a03. Combining the calibration parameters of the RGB-D camera and the inverse coordinate transformation formula, the world coordinates (X... w,p Y w,p Z w,p )

[0055] Convert to pixel coordinates (u p v p The calculation process is as follows: Figure 10 As shown in section (b);

[0056] a04. Based on the pixel coordinates of each point in the planned path, redraw the RGB image to generate the final obstacle avoidance path, thereby achieving path inverse projection. The final obstacle avoidance path after projection is as follows: Figure 11 As shown.

[0057] Figure 11 The central curve can provide a reference path for agricultural vehicles to avoid obstacles along the boundary curve within the target field A. It can intuitively demonstrate the characteristics of the obstacle avoidance path planned using the method of this invention: it is located in the center of the image, the entry into the obstacle is gentle, the navigation state can be restored as soon as possible after the obstacle is removed, and the path is smooth.

[0058] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, all of which fall within the protection scope of the present invention. The protection scope of the present invention is defined by the appended claims and any equivalent technical solutions.

Claims

1. An irregular field plot border-following obstacle avoidance path planning method based on RGB-D multi-modal data, characterized in that, The method comprises the following steps: S01. Constructing a target field A RGB image A of the field ridge and obstacle part of the target field A by using an agricultural vehicle equipped with an RGB-D camera RGB and a corresponding depth image A DEPTH , the agricultural vehicle simulates a real field edge obstacle avoidance operation scene during image acquisition, and the A RGB and the A DEPTH contain complete and clear field ridge area, obstacle area and non-crop farmland area; S02. The A RGB and the A DEPTH are input to the trained RGB-D semantic segmentation model, and an output mask image G RGB containing only the A DEPTH and the predicted value of the middle ridge and obstacle region is output mask ; S03. Collecting the A RGB and the data information in the A DEPTH , extracting the pixel coordinates and depth values of each pixel point; combining the calibration parameters of the RGB-D camera and the coordinate transformation formula to obtain the world coordinates (X w , Y w , Z w ) corresponding to each pixel point; using the R, G, B values of each pixel point in the A RGB to match the color information, combining the spatial position information of each pixel point with the color information, thereby completing the point cloud establishment; S04. Extracting the G mask corresponding to each category, and mapping a set of three-dimensional point coordinates corresponding to each category in the point cloud map according to a mapping relationship f between the pixel coordinates and the point cloud, to determine the category to which each point in the point cloud belongs, thereby completing point cloud matching. S05. Select three points C, D, and E located at the center of the matched point cloud image and use the plane formed by them as the projection reference plane; determine the plane equation ax + by + cz + d = 0 and the plane normal vector based on the coordinates of the three points. Calculate the distance D from each 3D point to the plane. e And obtain the projected coordinates (X) of each point according to the projection formula. pro Y pro Z pro Select the A RGB The three-dimensional point O corresponding to the bottom center point is taken as the origin, and two points A and B are determined at the bottom. The direction is taken as the x-axis direction. Establish a Cartesian coordinate system with the y-axis as the direction, and obtain the Cartesian coordinates (X, Y, y) of each point using the coordinate transformation formula. rec Y rec The points are plotted in a planar coordinate system; the contours of the field ridges and obstacle category points are fitted using envelopes, and the contours represent the field ridges and obstacles. The origin of the coordinate system represents the camera position (i.e., the position of the equipment), resulting in a two-dimensional obstacle avoidance coordinate map G. 2D-obs This completes the point cloud projection; S06. In the G 2D-obs artificial potential field method, the repulsive potential field function U req (x) and the attractive potential field function U att (x) are used to obtain the magnitude of the attractive force and the repulsive force on the machine at a certain point, the direction of the attractive force points to the target point, and the direction of the repulsive force points to the opposite direction of the obstacle point, the vector sum of the attractive force and the repulsive force is the resultant force on the machine at the point, then the resultant force on the machine at each position is calculated, thereby completing the path planning; the improved repulsive potential field function is obtained by superimposing the repulsive potential field of the obstacle region and the repulsive potential field of the ridge region. S07. Collect the Cartesian coordinates (X, Y) of each point on the planned obstacle avoidance path. rec,p Y rec,p By combining the calibration parameters of the RGB-D camera and the inverse coordinate transformation formula, the pixel coordinates (u) of each point are obtained. p v p According to the pixel coordinates of each point in A RGB The path is redrawn to generate the final obstacle avoidance path, thereby completing the path inverse projection and generating an obstacle avoidance path for agricultural vehicles to travel along the boundary within the target field A.

2. The method of claim 1, wherein, The step S02 comprises: RGB The step S02 comprises: DEPTH Before inputting into the trained RGB-D semantic segmentation model, the method further comprises: acquiring RGB images and depth images of farmland ridges and obstacle regions in different weather conditions and different farmland plots, and labeling the farmland ridges and obstacle regions in the images as training samples; training the RGB-D semantic segmentation model based on the plurality of training samples to obtain the trained RGB-D semantic segmentation model.

3. The method of claim 1, wherein, The three-dimensional point-to-plane distance calculation formula in the step S05 is The projection formula is The coordinate system transformation formula is 4. The method of claim 1, wherein, The improved repulsive potential field function U req (x) = U req,obs (x) + U req,hea (x), which is superimposed by the obstacle region repulsive potential field and the dike region repulsive potential field, wherein the obstacle region repulsive potential field The dike region repulsive potential field The attractive potential field function 5. The method of claim 1, wherein, In the step S07, the coordinate inverse transformation formula is and 6. The method of claim 1, wherein, In the step S07, the finally obtained obstacle avoidance path has the characteristics of being located in the center of the image, entering the obstacle gently, exiting the obstacle to quickly restore the navigation state, and smooth path.

7. The method of claim 1, wherein, In the step S02, the RGB-D semantic segmentation model is ESANet.

8. The method of claim 1, wherein, The target field block A is a paddy field and a dry field without planted crops, and is an irregularly shaped small field block, and there are obstacles of different shapes and different positions at the edges of the field block.

Images