Visual row control navigation method and system based on rice field adaptive color index

Abstract

The invention provides a visual alignment navigation method and system based on a rice field adaptive color index. The method comprises the following steps: screening optimal color index characteristics suitable for different rice field operation scenes; based on the rice seedling-soil-water surface coverage, the texture features, the brightness range and the Euclidean distance features corresponding to the waveform features, determining a rice field operation scene; obtaining a binarized image according to the optimal color index characteristics in the corresponding rice field operation scene, integrally communicating seedling crop row areas in the binarized image so as to fit seedling crop rows, and generating a navigation reference line in combination with a visual angle center seedling crop row line; and calculating a course angle and a transverse deviation required to be adjusted of the vehicle body based on the vehicle body driving direction reference line and the navigation reference line. According to the method, the problems of low visual alignment navigation precision and poor reliability caused by complex shadow and illumination conditions of the existing rice field scene are solved, and the method has better robustness.

Description

Technical Field

[0001] The present invention belongs to the technical field of automatic driving of unmanned agricultural machinery, and in particular relates to a machine vision row operation navigation method and system based on a paddy field adaptive color index. Background Art

[0002] Unmanned agricultural machinery (UAGV) is an important carrier for realizing unmanned farms, and automatic navigation technology is one of the key technologies to realize the full autonomous operation of unmanned agricultural machinery. At present, automatic navigation technology based on satellite positioning is relatively mature and has the ability to work around the clock. However, satellite signals are easily affected by trees and buildings around the field, resulting in signal loss and offset, so that it is difficult to independently achieve precise row operations when managing crops in the field. Considering the high real-time requirements of unmanned agricultural machinery operations, machine vision navigation technology can effectively compensate for the deviation of the satellite navigation system, improve navigation accuracy and avoid crushing seedlings.

[0003] Machine vision navigation technology has the advantages of low cost and easy operation, but there are still certain challenges in crop row recognition in complex rice field environments. Since the water surface and seedling coverage ratio of rice is constantly changing at different growth stages, this often brings different water surface shadows and reflection effects under different lighting conditions. If a single characteristic color index is used to identify crop rows, inconsistent path recognition results will be generated, and then uncontrollable errors will be generated, leading to seedling crushing. Therefore, studying the optimal characteristic color index for identifying rice crop rows under different growth periods, lighting conditions, water and soil coverage and other operating scenarios is of great significance for improving the row operation performance of unmanned agricultural machinery in complex rice field environments.

[0004] At present, most of the relevant literature focuses on the screening and identification of the optimal characteristic color index in specific crop growth period scenarios, while there are few studies on the entire crop field management stage under different lighting and water and soil cover conditions. For example, in 2020, HKSuh et al. from Wageningen University explored the combined segmentation performance of different color indices and threshold methods based on the characteristic elements of the operation scene such as beet plant size, shadow, and light intensity. In 2022, Sun Binfeng et al. from the Jiangxi Academy of Agricultural Sciences proposed a rice field image segmentation method based on color index and threshold method, and screened the color index most suitable for rice field image segmentation for rice field images obtained in clear and cloudless weather during the tillering and jointing stages. However, existing studies have not yet considered the impact of variable rice field operating environments such as lighting, water and soil cover conditions on the accuracy of machine vision recognition when facing a variety of color indices. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a machine vision row operation navigation method and system based on paddy field adaptive color index.

[0006] The present invention achieves the above technical objectives through the following technical means.

[0007] A visual row navigation method based on a rice field adaptive color index comprises the following steps:

[0008] Screen the optimal color index features suitable for different rice field operation scenarios; determine the rice field operation scene based on the Euclidean distance features corresponding to the seedling-soil-water surface coverage and texture features, brightness range and waveform features; obtain a binary image according to the optimal color index features in the corresponding rice field operation scene, connect the seedling crop row area in the binary image as a whole, and then fit the seedling crop row, and generate a navigation baseline based on the seedling crop row line at the center of the viewing angle; calculate the heading angle and lateral deviation that need to be adjusted for the vehicle body based on the vehicle body driving direction reference line and the navigation baseline.

[0009] Furthermore, there are 36 rice field operation scenes, including the direct light or backlight characteristics corresponding to different field types under the strong light operation conditions of the rice seedling stage, tillering stage and early jointing stage, and the direct light or backlight characteristics corresponding to different field types under the weak light operation conditions of the rice seedling stage, tillering stage and early jointing stage; the field types include paddy fields, dry fields, and mixed paddy and dry fields.

[0010] Furthermore, the screening of the optimal color index feature is specifically as follows: the color index is combined with the maximum inter-class variance method to perform binary segmentation processing on the seedling crop row video frames in different operation scenarios, and the inter-class segmentation accuracy is evaluated by combining the confusion matrix quantitative indicators to achieve the optimal color index feature screening.

[0011] Furthermore, the determination formula for the rice field operation scene is:

[0012]

[0013] Brightness Y = 0.299*R + 0.587*G + 0.114*B

[0014]

[0015] Among them, YM is the seedling coverage area, TR is the soil coverage area, SM is the water surface coverage area, the brightness Y range interval is (m, n), m is the lower limit of the brightness value, n is the upper limit of the brightness value, Y max is the maximum brightness, and R, G, and B are the red, green, and blue color index channels of the video frame, respectively.

[0016] Furthermore, the rice seedling crop row area is connected as a whole, the video frame is first partitioned, and then the centroid slice connection method is used to achieve it. The partition connection line length of the centroid slice connection method is defined as follows:

[0017]

[0018] Among them, k1, k2, k3 represent different lengths of connecting lines set according to the distant view, mid-view, and close view of the video frame, respectively, and u is the number of horizontal pixels of the video frame.

[0019] Furthermore, the fitting of the rice seedling crop row is specifically: using the maximum outer envelope method to obtain the overall periphery of the rice seedling crop row, and fitting the rice seedling crop row according to the least squares method.

[0020] Furthermore, the vehicle body driving direction reference line is the central axis of the video frame.

[0021] Furthermore, the heading angle calculation formula is as follows:

[0022]

[0023] Among them, points[0].x and points[0].y represent the horizontal and vertical coordinates of the starting pixel of the navigation baseline, points[1].x and points[1].y represent the horizontal and vertical coordinates of the ending pixel of the navigation baseline, points[2].x and points[2].y represent the horizontal and vertical coordinates of the starting pixel of the vehicle body travel direction reference line, and points[3].x and points[3].y represent the horizontal and vertical coordinates of the ending pixel of the vehicle body travel direction reference line.

[0024] Furthermore, the calculation of the lateral deviation is specifically as follows: based on the pixel coordinates of the navigation baseline and the pixel coordinates of the vehicle body driving direction reference line, the pixel coordinates are converted into base plane world coordinates using Zhang Zhengyou's camera calibration method, the navigation baseline coordinates and the vehicle body driving direction reference line coordinates in the world coordinate system are obtained, and then the lateral deviation is calculated.

[0025] To implement the above method, another technical solution adopted by the present invention is: providing a system for visual row navigation based on the adaptive color index of rice fields, implementing the visual row navigation method based on the adaptive color index of rice fields based on C++OpenCV, assisting the GNSS module fusion navigation, and realizing automatic driving row operation in the field.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] (1) Compared with the traditional method that only focuses on color index screening and discrimination under specific weather scenes, the present invention screens the optimal color index under different illumination and water surface coverage ratio scenes in the field management stage during the multi-growth period of rice, and has better applicability and robustness when applied to actual rice field row operation scenarios;

[0028] (2) Compared with the traditional method of using the least squares method to fit crop feature points to form a crop row navigation baseline, the present invention first uses the centroid slice connection method to connect the fragmented crop row target area before fitting the crop row line, and uses the maximum outer envelope line to obtain the overall contour of the crop row, which effectively overcomes the problem of inaccurate row line fitting due to lack of seedlings in the seedling stage;

[0029] (3) Compared with the traditional method, the present invention innovatively adds the content of complex rice field operation scene type recognition based on the rice field seedling-soil-water surface coverage, texture characteristics, video frame brightness and change fluctuation characteristics, making the present invention have stronger practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of the machine vision row operation navigation system based on the paddy field adaptive color index according to the present invention;

[0031] FIG. 2( a ) is a schematic diagram of a method for determining a front-lit operation scene under strong / weak light conditions according to the present invention;

[0032] FIG2( b ) is a schematic diagram of a method for determining a backlit operation scene under strong / weak light conditions according to the present invention;

[0033] Figure 3 It is a binary image obtained based on the optimal feature color index in the corresponding operation scene according to the present invention;

[0034] Figure 4 The image of the region of interest of a rice crop row obtained by the morphological preprocessing method such as opening operation filtering according to the present invention;

[0035] Figure 5 The image of the rice crop row area is connected based on the centroid slice connection method of the present invention;

[0036] Figure 6 The rice crop row contour image obtained based on the maximum outer envelope method of the present invention;

[0037] Figure 7 The rice crop row line based on least square method fitting according to the present invention;

[0038] Figure 8 It is the navigation reference line based on rice crop row line fitting as well as the heading angle and lateral deviation result image described in the present invention. DETAILED DESCRIPTION

[0039] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.

[0040] The purpose of the present invention is to solve the problems of low accuracy and poor reliability of visual row navigation caused by complex shadows and lighting conditions in existing rice field scenes, and to propose a visual row navigation method based on the rice field adaptive color index, so that it is more in line with the actual operation needs of unmanned agricultural machinery rice field row navigation and has better robustness.

[0041] like Figure 1 As shown, the present invention proposes a visual row navigation method based on the adaptive color index of rice fields, which screens the optimal color index features for different operation scenes in rice fields, and combines the rice field operation scene recognition and the inter-row navigation baseline extraction method to realize the adaptive optimal color index machine vision row navigation control of the unmanned agricultural machinery platform. Specifically, it includes the following steps:

[0042] Step 1: Use the depth camera on agricultural machinery to shoot the foreground video of rice field operations, and record the direct light / backlight features corresponding to paddy fields / dry fields / water-dry mixed scenes under strong light (sunny) / weak light (cloudy) operating conditions in the rice seedling stage, tillering stage, and early jointing stage. There are a total of 36 typical rice field operation scenes, and use Adobe Photoshop software to manually select the seedling coverage area in the video frames of typical rice field operation scenes as the true value verification data set for the optimal color index inter-class segmentation accuracy evaluation in the next step.

[0043] Step 2: Based on step 1, 40 commonly used color indices including R (red) G (green) B (blue) channel index, chroma and saturation index, and green vegetation index are used, and the maximum between-class variance method (OTSU) is combined to perform binary segmentation processing on the video frames of seedling crop rows in different operation scenarios. The inter-class segmentation accuracy is evaluated by combining the commonly used quantitative indicators of the confusion matrix. The higher the inter-class segmentation accuracy, the better the color index used, so as to screen the optimal color index corresponding to different rice field operation scenarios.

[0044] The calculation formulas for common image segmentation performance evaluation indicators such as accuracy, precision, recall, and specificity derived from the confusion matrix are as follows:

[0045]

[0046]

[0047] Among them, TP represents the true positive class, that is, the true class of the sample manually labeled is the positive class, and the recognition result is also the positive class; FN represents the false negative class, that is, the true class of the sample manually labeled is the positive class, but it is recognized as the negative class; FP represents the false positive class, that is, the true class of the sample manually labeled is the negative class, but it is recognized as the positive class; TN represents the true negative class, that is, the true class of the sample manually labeled is the negative class, and the recognition result is also the negative class.

[0048] Step 3: Based on step 1, determine the growth period, paddy field / dry field / mixed paddy field and other paddy field operation scene types based on the coverage and texture characteristics of seedlings (YM)-soil (TR)-water surface (SM); determine the paddy field operation scene in the front light under strong light / weak light conditions (Figure 2(a)) and the paddy field operation scene in the back light under strong light / weak light conditions (Figure 2(b)). The specific calculation method is as follows:

[0049]

[0050] Brightness Y = 0.299*R + 0.587*G + 0.114*B

[0051]

[0052] Among them, YM is the seedling coverage area, TR is the soil coverage area, SM is the water surface coverage area, the brightness Y range interval is (m, n), m is the lower limit of the brightness value, n is the upper limit of the brightness value, Y max is the maximum brightness, and R, G, and B are the red, green, and blue color index channels of the video frame, respectively.

[0053] Based on the above classification methods of growth period, field type and lighting conditions, the identification of complex rice field operation scene types can be effectively realized, that is, the growth period, field type and lighting conditions corresponding to a certain rice field operation scene type.

[0054] Step 4: Based on steps 2 and 3, according to the binary image obtained by the optimal color index feature in the corresponding operation scene, the morphological image preprocessing methods such as opening operation and removing small connected areas are used to obtain the region of interest of the seedling crop row, such as Figure 3 , 4 As shown in the figure, the rice seedlings in the seedling and tillering stages are relatively sparse, the present invention divides the video frame into distant, medium and near view zones according to the number of horizontal pixels u, and uses a connecting line with a length of k for different zones, and adopts a centroid slice connection method to achieve overall connectivity of the rice seedling crop row area, as shown in the figure. Figure 5 shown.

[0055] The partition connection line length of the centroid slice connection method is defined as follows:

[0056]

[0057] Among them, k1, k2, and k3 represent connecting lines of different lengths set according to the distant view, mid-view, and close-up view of the video frame, respectively.

[0058] Step 5: Based on step 4, use the maximum outer envelope method to obtain the overall contour of the seedling crop row ( Figure 6 ), and fitted the rice seedling crop rows according to the least squares method ( Figure 7 ), combined with the rice seedling crop row line at the center of the viewing angle, a navigation baseline is generated. This method can be used to fit the crop rows in the rice seedling stage, tillering stage, and early jointing stage, especially when it is difficult to accurately fit the rice seedling crop row line due to the lack of seedlings in the seedling stage.

[0059] Step 6: If Figure 8 As shown in the figure, the green line is the central axis of the video frame, which is the reference line for the vehicle's driving direction; the blue line is the navigation baseline between the rows of rice seedlings and crops generated according to step 5. The heading angle and lateral deviation that need to be adjusted for the vehicle are calculated based on the vehicle's driving direction reference line and the crop row navigation baseline.

[0060] The heading angle calculation formula is as follows:

[0061]

[0062] Among them, points[0].x and points[0].y represent the horizontal and vertical coordinates of the starting pixel of the navigation baseline, points[1].x and points[1].y represent the horizontal and vertical coordinates of the ending pixel of the navigation baseline, points[2].x and points[2].y represent the horizontal and vertical coordinates of the starting pixel of the vehicle body driving direction reference line, and points[3].x and points[3].y represent the horizontal and vertical coordinates of the ending pixel of the vehicle body driving direction reference line. That is, by comparing the starting and ending coordinates of the navigation baseline and the vehicle body driving direction reference line, the angle between the navigation baseline and the vehicle body driving direction reference line is determined, and then the heading angle to be adjusted is obtained.

[0063] Based on the pixel coordinates of the navigation baseline and the vehicle driving direction reference line, the pixel coordinates are converted into base plane world coordinates using Zhang Zhengyou's camera calibration method to obtain the navigation baseline coordinates and the vehicle driving direction reference line coordinates in the world coordinate system, and then the lateral deviation is calculated.

[0064] According to Zhang Zhengyou's camera calibration method, the image pixel coordinates are converted to base plane world coordinates, and the calculation formula is as follows:

[0065]

[0066] Among them, Z c Represents the pixel depth value of the video frame image, (u, v) represents a point in the pixel coordinate system, u0 represents the horizontal pixel difference between the pixel coordinates of the center of the image and the pixel coordinates of the origin, and v0 represents the vertical pixel difference between the pixel coordinates of the center of the image and the pixel coordinates of the origin; on the right side of the equal sign, the first matrix from left to right is the camera intrinsic parameter matrix, f x and f y Respectively represent the focal length of the camera in the horizontal and vertical directions. The second matrix is ​​composed of the rotation matrix R and the offset vector T, which is the camera external parameter matrix. The third matrix (X w ,Y w ,Z w ) is a point in the base plane world coordinate system.

[0067] Finally, based on C++OpenCV, the adaptive optimal color index machine vision row navigation control of the unmanned agricultural machinery platform in the complex rice field operation scene is realized, which can assist the GNSS module fusion navigation to realize the automatic driving row operation of the field. Therefore, this method has a high application value in actual field operations.

[0068] The embodiments are preferred implementations of the present invention, but the present invention is not limited to the above-mentioned implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essential content of the present invention belong to the protection scope of the present invention.

Claims

1. A visual navigation method based on rice field adaptive color index, characterized by: Screen the optimal color index features suitable for different rice field operation scenes; determine the rice field operation scene based on the Euclidean distance features corresponding to the seedling-soil-water surface coverage and texture features, brightness range and waveform features; obtain a binary image based on the optimal color index features in the corresponding rice field operation scene, connect the seedling crop row area in the binary image as a whole, and then fit the seedling crop row, and generate a navigation baseline combined with the seedling crop row line at the center of the viewing angle; The heading angle and lateral deviation that need to be adjusted for the vehicle body are calculated based on the vehicle body driving direction reference line and the navigation reference line.

2. The visual navigation method based on rice field adaptive color index according to claim 1 is characterized in that: There are 36 rice field operation scenes in total, including the direct light or backlight characteristics corresponding to different field types under strong light operation conditions in the rice seedling stage, tillering stage, and early jointing stage, as well as the direct light or backlight characteristics corresponding to different field types under weak light operation conditions in the rice seedling stage, tillering stage, and early jointing stage; the field types include paddy fields, dry fields, and mixed paddy and dry fields.

3. The visual navigation method based on rice field adaptive color index according to claim 1 is characterized in that: The screening of the optimal color index feature is specifically as follows: using the color index combined with the maximum inter-class variance method to perform binary segmentation processing on the seedling crop row video frames in different operation scenarios, combining the confusion matrix quantitative index to evaluate the inter-class segmentation accuracy, and realizing the screening of the optimal color index feature.

4. The visual navigation method based on rice field adaptive color index according to claim 1 is characterized in that: The judgment formula for the rice field operation scene is: Brightness Y = 0.299*R + 0.587*G + 0.114*B Among them, YM is the seedling coverage area, TR is the soil coverage area, SM is the water surface coverage area, the brightness Y range interval is (m, n), m is the lower limit of the brightness value, n is the upper limit of the brightness value, Y max is the maximum brightness, and R, G, and B are the red, green, and blue color index channels of the video frame, respectively.

5. The visual navigation method based on rice field adaptive color index according to claim 1 is characterized in that: The rice seedling crop row area is connected as a whole. The video frame is first partitioned and then the centroid slice connection method is used. The partition connection line length of the centroid slice connection method is defined as follows: Among them, k1, k2, k3 represent different lengths of connecting lines set according to the distant view, mid-view, and close view of the video frame, respectively, and u is the number of horizontal pixels of the video frame.

6. The visual navigation method based on rice field adaptive color index according to claim 1 is characterized in that: The described fitting of the rice seedling crop row specifically comprises: using the maximum outer envelope method to obtain the overall periphery contour of the rice seedling crop row, and fitting the rice seedling crop row according to the least square method.

7. The visual navigation method based on rice field adaptive color index according to claim 1 is characterized in that: The vehicle body driving direction reference line is the central axis of the video frame.

8. The visual navigation method based on rice field adaptive color index according to claim 1 is characterized in that: The heading angle calculation formula is as follows: Among them, points[0].x and points[0].y represent the horizontal and vertical coordinates of the starting pixel of the navigation baseline, points[1].x and points[1].y represent the horizontal and vertical coordinates of the ending pixel of the navigation baseline, points[2].x and points[2].y represent the horizontal and vertical coordinates of the starting pixel of the vehicle body travel direction reference line, and points[3].x and points[3].y represent the horizontal and vertical coordinates of the ending pixel of the vehicle body travel direction reference line.

9. The visual navigation method based on rice field adaptive color index according to claim 1, characterized in that: The calculation of the lateral deviation is specifically as follows: based on the pixel coordinates of the navigation baseline and the pixel coordinates of the vehicle body driving direction reference line, the pixel coordinates are converted into base plane world coordinates using Zhang Zhengyou's camera calibration method, the navigation baseline coordinates and the vehicle body driving direction reference line coordinates in the world coordinate system are obtained, and then the lateral deviation is calculated.

10. A system based on the visual navigation method based on the rice field adaptive color index according to any one of claims 1 to 9, characterized in that: The visual row navigation method based on the adaptive color index of rice fields is implemented based on C++OpenCV, and the GNSS module fusion navigation is assisted to realize the automatic driving row operation of the field.

Images