Rice row line detection and navigation information extraction method based on camera pose compensation

Abstract

Aiming at the complex environment of paddy field and the dramatic change of the posture of the weeding machine, a method of paddy row line detection and navigation information extraction based on camera pose compensation is proposed. Firstly, the real-time pose of the camera is measured by using the tilt sensor, and the mapping and compensation relationship between the corresponding coordinates of the camera pixel coordinate system and the world coordinate system is established. The perspective transformation is performed on the paddy image collected by the camera to obtain the image containing the actual position information. The interested region is intercepted from the perspective transformed image and preprocessed, and the paddy part in the region is identified to obtain the binary image. Further, the sliding window method is used for row line detection, the row line detection result is inversely perspective transformed to the original image, the row line fitting result is obtained, and the pose compensation is used for correction. The row line recognition result is projected to the ground coordinate system. Finally, aiming at the special situation of the bending, broken row and connected row of the paddy row line, the targeted correction processing is performed to improve the robustness of the algorithm.

Description

Technical Field

[0001] This invention belongs to the field of intelligent agricultural machinery, specifically relating to a method for detecting and extracting rice row line information for automated agricultural machinery based on camera pose compensation. Background Technology

[0002] Automatic navigation technology for agricultural machinery is widely used in all stages of agricultural production, including tilling, planting, management, and harvesting. The position and heading of the crop rows relative to the machinery are crucial for automatic navigation. While GNSS-based sensing and positioning methods are widely used in agricultural machinery navigation, for rice paddy weeding robots, it's essential to avoid the weeder wheels running over existing rice seedlings during weeding operations to prevent excessive damage. Navigation algorithms based solely on satellite positioning cannot identify rice rows or adapt to their curvature. Therefore, other sensing technologies are needed for rice row identification, combined with satellite positioning navigation, to minimize damage to rice seedlings during automated weeding.

[0003] Among crop row recognition methods, visual sensor-based crop row recognition technology has the most extensive research and application, with monocular camera machine vision-based methods receiving the most attention. Existing methods typically construct coordinate transformation relationships by fixing the camera height and pose and the pose relationship between the crop row and seedlings, assuming the ground is horizontal, to achieve the conversion from crop boundary lines to navigation paths. However, the paddy field environment is complex. During operation, the posture of the weeding machine changes drastically, and the relative pose relationship between the robot, camera, and crops exhibits dynamic changes. Existing methods, which build simplified models, struggle to extract accurate navigation information. Furthermore, the complex environment of paddy fields significantly impacts the sowing and transplanting efficiency of seeding machinery. Actual rice row lines may exhibit typical scenarios such as row bends, row breaks, and row connectivity issues. Row detection must also consider how to identify these scenarios and address them accordingly. Summary of the Invention

[0004] This invention addresses the challenges posed by the complex environment of paddy fields and the drastic changes in the vehicle posture of weeders during operation. It provides a method for rice row detection and navigation information extraction based on camera pose compensation. Under a defined coordinate system, a mapping and compensation relationship between the pixel coordinate system and the corresponding coordinates in the world coordinate system is established using the camera pose. Based on this coordinate mapping and compensation relationship, the rice row recognition results are accurately converted into autonomous vehicle navigation information in the world coordinate system. Furthermore, the method can correctly identify rice rows and extract navigation information even in most typical cases, such as curved, broken, or connected rows, demonstrating high robustness.

[0005] This invention comprises two parts: hardware components and software algorithms. The hardware components are as follows:Figure 1 As shown, the system includes a camera mounting bracket 2, a monocular camera 3, and an attitude sensor 4, all mounted on the body 1 of the weeding robot. The camera mounting bracket 2 is fixed to the upper front of the robot body 1. The monocular camera 3 is mounted on the camera mounting bracket 2, with one degree of pitch freedom between them. The monocular camera 3 can be tilted relative to the camera mounting bracket 2, and after adjustment, it is tightened with screws. The attitude sensor 3 is attached to the upper surface of the monocular camera 3, ensuring that both maintain the same attitude. To facilitate the detection of line and navigation lines, the following definitions are provided. Figure 1 and Figure 2 Coordinate system shown:

[0006] World coordinate system O W X W Y W Z W —The system's absolute coordinate system;

[0007] Camera coordinate system O C X C Y C Z C —The origin is at camera O C It is the optical center of the camera, and the three coordinate axes are respectively along the three axes of the camera;

[0008] Geodetic coordinate system O G X G Y G Z G —The coordinate system plane coincides with the horizontal plane, O G Y G With O C Y C The projections of the two coordinate systems coincide, and the origin of the coordinate system is obtained by projecting the origin of the camera coordinate system onto the ground. The distance between the two is HC.

[0009] Image coordinate system O I xy — This is a two-dimensional coordinate system within the camera's imaging plane, with the origin at the center of the imaging field of view, and the coordinate axes as shown in the diagram. Figure 2 As shown;

[0010] Pixel coordinate system O P uv—a two-dimensional coordinate system within the camera's imaging plane, with the origin at the upper right vertex of the imaging field of view, and the coordinate axes as shown. Figure 2 As shown;

[0011] The software algorithm flow is as follows: Figure 3 As shown, the specific implementation steps are as follows:

[0012] 1) Based on the camera pose and the camera's pitch angle relative to the vehicle, establish a mapping relationship between pixel coordinates and world coordinates with terrain interference pose compensation.

[0013] 2) Acquire the original image of the rice row line and perform perspective transformation on the image to obtain an image with actual position information. Then, extract the region of interest (ROI) containing the wheel and the left and right adjacent rice seedling rows from the perspective transformed image.

[0014] 3) Preprocess the extracted region of interest to identify the rice portion within the region and obtain a binarized image;

[0015] 4) The sliding window method is used to detect line lines, and the line line detection results are transformed back into the original image to obtain the line line fitting results;

[0016] 5) Based on the coordinate system projection relationship and perspective transformation relationship, project the line recognition results onto the geodetic coordinate system and extract the navigation information.

[0017] 6) Targeted measures are taken to address potential issues such as bent lines, broken lines, and disconnected lines during the line recognition process, thereby improving the robustness of the algorithm.

[0018] Compared with the prior art, the advantages of the present invention are as follows:

[0019] 1) Based on the pose of a monocular camera, a mapping relationship between the pixel coordinate system and the world coordinate system was established, which theoretically eliminated the influence of the vehicle posture change on the extraction of navigation information when the paddy field weeder is working in the paddy field.

[0020] 2) It can correctly identify the line and extract navigation information for most typical cases such as line bending, line breakage, and line connection, and has high robustness. Attached Figure Description

[0021] Figure 1 Relative relationships of coordinate systems

[0022] Figure 2 Relative relationship between image coordinate system and pixel coordinate system

[0023] Figure 3 Software algorithm flowchart

[0024] Figure 4 Comparison images before and after perspective transformation, where a is the original image and b is the perspective-transformed image.

[0025] Figure 5 Schematic diagram after image preprocessing

[0026] Figure 6 Line recognition results

[0027] Figure 7 Navigation information diagram

[0028] Figure 8Typical scenario processing results are shown in the figure, where a is the simulation diagram of the line bending, b is the line breakage, and c is the line connection. Detailed Implementation

[0029] The present invention will be described in detail below with reference to the accompanying drawings and embodiments, but this should not be construed as limiting the scope of protection of the present invention.

[0030] The specific steps for detecting rice row lines and extracting navigation information in this invention are as follows:

[0031] Step (1): Establish the mapping relationship between pixel coordinates and world coordinates. The relative relationships of the coordinate systems are as follows: Figure 1 As shown, it involves the pixel coordinate system O P uv, image coordinate system O I xy, camera coordinate system O C X C Y C Z C Geodetic coordinate system O G X G Y G Z G and world coordinate system O W X W Y W Z W In the geodetic coordinate system, O G X G Y G The plane coincides with the horizontal plane, O G Y G With O C Y C The projections coincide, and the origin of the coordinate system is obtained by projecting the origin of the camera coordinate system onto the ground. The distance between the two is H. C O of the camera coordinate system C Z C Shaft, O C X C The Euler angles of the axes are α and β, respectively.

[0032] For any point P in the camera's field of view in the geodetic coordinate system G (X G Y G Z G ) T The coordinates of P in the camera coordinate system are C (X C Y C Z C ) T It can be converted into a unique point P in the pixel coordinate system. P (uv) T The conversion relationship is as follows:

[0033]

[0034] Here, M1 is the camera intrinsic parameter matrix, and R is the P... G To P C The rotation transformation matrix, T is P G To P C The translation transformation matrix is ​​given. M1 can be obtained through camera calibration, and R and T can be determined based on the camera pose α, β, and H. C get:

[0035]

[0036]

[0037] Where R X It is around X G The rotation matrix of the axis, R Y It is around Y G The rotation matrix of the axis, R Z It is around Z G The rotation matrix of the axis.

[0038]

[0039] Treating rice seedlings as projections onto the surface of the paddy field, the desired row lines and the information needed to obtain them can both be viewed as projections of points on the paddy field surface. These points satisfy Z... G =0. Formula (1) can be simplified to:

[0040]

[0041] Formula (4) contains the unknown parameter Z. C X G Y G All other parameters are known, and the unknown parameters can be calculated using the formula.

[0042]

[0043] Where *[i] represents the i-th row of matrix *.

[0044] Step (2): To facilitate line detection and the conversion of line information to navigation lines, perspective transformation and region of interest (ROI) extraction are required for the acquired image. Perspective transformation requires calculating the perspective transformation matrix. The coordinates of the four scattered pixels in the pixel coordinate system mapped to the geodetic coordinate system are obtained using formula (5) and then translated and scaled. The perspective transformation matrix can be obtained based on the correspondence between the four points before and after projection. Figure 4 After performing perspective transformation and cropping, we can obtain an image with actual positional information. Figure 4 b.

[0045] Figure 4 b contains useless information, which interferes with the key information. Therefore, the region of interest is selected for subsequent image processing. Four vertices of the original image are selected, and their coordinates in the geodetic coordinate system are calculated using formula (5). G1 =(X G1 ,Y G1 ) T P G2 =(X G2 ,Y G2 ) T P G3 =(X G3 ,Y G3 ) T P G4 =(X G4 ,Y G4 ) T (O G X G Y G (In-plane). Using four points as reference, crop inwards, retaining an image length with a height of 4000 pixels as the ROI, with a range of P. ROI1 =(u ROI1 ,v ROI1 ) T P ROI2 =(u ROI2 ,v ROI2 ) T P ROI3 =(u ROI3 ,v ROI3 ) T P ROI4 =(u ROI4 ,v ROI4 ) T The region of interest corresponds to Figure 4 b. The larger rectangular area (including the smaller rectangle).

[0046] To identify the curvature of the lines, the region of interest (ROI) is further divided into blocks of image height, and processed block by block. In this algorithm, height = 1000, and the first ROI1 is selected, corresponding to... Figure 4 b is the area of ​​the small rectangle at the bottom.

[0047] Step (3): Before performing line recognition on ROIi, some preprocessing is required to reduce noise and interference in the image, facilitating line recognition. ROI1 is grayscaled using the super-green method, and then Gaussian filtering with a 15×15 Gaussian kernel is applied to the grayscale image to eliminate noise. Afterwards, the image is binarized. Considering the significant difference between the foreground and background, the OTU algorithm is used for binarization. To reduce the interference of white pixels between rows on the line recognition results, erosion and dilation are applied to the binarized image. The processed image is shown below. Figure 5 As shown.

[0048] Step (4): Line detection is performed using the sliding window method. The base point P of the first set of left and right windows is selected at the bottom of the ROI1 image. L1 =(u L1 ,v L1 ), P R1 =(u R1 ,v R1 The first set of base points satisfies formula (6), and the scanning direction is the negative direction of the v-axis. Based on height = 1000 and the row spacing of rice is 250mm, the window size is selected as 100×250, and the bottom midpoint of the window is the base point of the window.

[0049]

[0050] in The height (in pixels) of ROI1, u L1crest u R1crest The coordinates (in pixels) of the first peak point on either side of the bottom midpoint.

[0051] After determining the base points, the center points of the left and right windows are calculated based on the coordinate distribution of the white pixels within the window. The center points are obtained by formula (7), and the next sliding base point is generated by projecting the center point in the sliding direction. Except for the first set of sliding window base points of ROI1, which are obtained by formula (6), the first set of sliding window base points of the other ROIi images are all obtained by projecting the last set of center points of ROIi-1.

[0052]

[0053] Where n is the number of white pixels, (u i ,v i Let (u) be the coordinates of the i-th white pixel. c ,v c () represents the coordinates of the center point.

[0054] The midpoint of the center points of the left and right windows is calculated and used as the line fitting point. The line is then fitted using the least squares method. To ensure the continuity of the line, all ROIi except ROI1 are fitted using the least squares method with a base point, which is the intersection of the fitted line of ROIi-1 and the lower boundary of ROIi. The sliding window scanning process and line fitting results for ROI1 are shown below. Figure 6 As shown.

[0055] Step (5): After completing the line recognition, based on the coordinate system projection relationship and perspective transformation relationship, the line recognition results can be projected onto the geodetic coordinate system to extract the navigation information. The relationship between the navigation information and the vehicle body is as follows: Figure 5 As shown, vehicle positioning information and navigation information in the geodetic coordinate system can be transformed to the world coordinate system using the Gauss-Kruger projection relationship. Based on the transformation result, navigation control of the operating vehicle can be achieved.

[0056] Step (6): Targeted measures are taken to address potential issues such as line bending, line breaks, and line connectivity during line recognition, thereby improving the robustness of the algorithm. For line bending, the height can be dynamically adjusted based on the line fitting accuracy and control requirements. When the line bending is severe and the fitting error is large, the height is reduced, and the fitting is repeated. This method selects height = 1000, corresponding to an actual length of 1000mm, which achieves good experimental results. Since extreme line bending is relatively rare, a simulation environment is used for simulation. The recognition results of the simulated image are as follows: Figure 8 As shown in a. When there are cases of broken lines and connected lines, in order to reduce the impact of the corresponding area on line recognition, the information of the corresponding area can be discarded. Before line recognition, the upper and lower thresholds of the sliding window are first calculated according to the number of white pixels in the region of interest using formula (8).

[0057]

[0058] in The number of white pixels in ROIi. For ROIi size, s win n is the sliding window size. win k is the average number of white pixels in the window. max k min n is the threshold coefficient for white pixels. limmax n limmin This represents the threshold number of white pixels.

[0059] Before extracting the center point of the window, calculate the number of real white pixels n within the window. winreal If not satisfied

[0060] n lim min <nwinreal <n lim max (9)

[0061] This indicates the presence of either broken rows or connected row lines within the sliding window. When a broken row or connected row line fills the entire ROIi, the row lines for this region can be generated using the row line fitting results from ROIi+1 and ROIi-1. For the cases of broken rows and connected row lines, the processing result is as follows: Figure 8 As shown in b and 8c.

Claims

1. A method for detecting and extracting navigation information of rice rows based on camera pose compensation, characterized in that, include: Step 1: Fix the attitude sensor to the surface of the monocular camera and install it at a certain angle on the body of the weeding robot; Step 2: Based on the monocular camera pose, establish the mapping relationship between pixel coordinates and world coordinates; Step 3: Acquire the original image and perform perspective transformation on the original image to obtain an image with actual position information, that is, convert the original image, which was originally viewed from a downward angle, into a top-down view, and extract the region of interest from the perspective-transformed image; Step 4: Preprocess the region of interest, identify the rice part in the region, and obtain a binarized image. The white part in the binarized image is the rice part. Step 5: Use the sliding window method to perform row line detection on the binarized image described in Step 4. Set left and right base points at the bottom of the image, and construct a sliding window with the base points as the bottom midpoint. The center point of the left and right windows can be calculated based on the coordinate distribution of the white pixels in the window. The center point is used as the feature point for rice row line fitting. Projecting the center point into the sliding direction can generate the next sliding window base point. Transform the row line detection result into the original image through reverse perspective to obtain the row line fitting result. Step 6: Based on the coordinate system projection relationship and perspective transformation relationship, project the line fitting result onto the geodetic coordinate system and extract the navigation information; Step 2 establishes a mapping relationship between pixel coordinates and world coordinates based on the monocular camera pose, specifically including: setting up a pixel coordinate system. Image coordinate system Camera coordinate system Geodetic coordinate system and world coordinate system The geodetic coordinate system The plane coincides with the horizontal plane. and The projections coincide, and the origin of the coordinate system is obtained by projecting the origin of the camera coordinate system onto the ground. The distance between the two is... Camera coordinate system axis, The Euler angles of the axes are respectively and ; If we consider rice seedlings as projections onto the surface of the paddy field, and the desired row lines and the information needed to obtain them are considered as projections of points on the paddy field surface, then... In the formula, It is the camera intrinsic parameter matrix. yes arrive The rotation transformation matrix, yes arrive The translation transformation matrix, For camera coordinate system coordinates, These are pixel coordinates. These are coordinates in the geodetic coordinate system. The perspective transformation mentioned in step 3 is calculated as follows: In the formula, It is the camera intrinsic parameter matrix. yes arrive The rotation transformation matrix, yes arrive The translation transformation matrix, For camera coordinate system coordinates, These are pixel coordinates. These are coordinates in the geodetic coordinate system. Representation matrix The OK.

2. The method for rice row detection and navigation information extraction based on camera pose compensation according to claim 1, characterized in that, The preprocessing in step 4 includes grayscale conversion using the super green method, followed by Gaussian filtering of the grayscale image using a 15×15 Gaussian kernel to eliminate noise, and then binarization of the image using the OTU algorithm.

3. The method for rice row detection and navigation information extraction based on camera pose compensation according to claim 1, characterized in that, Step 5 uses the sliding window method to perform line detection on the binarized image, and then performs a reverse perspective transformation on the line detection result to obtain the line fitting result; specifically, it includes: Step 5.1: Select the base point of the first group of left and right windows at the bottom of the binarized image. Take a portion with a row spacing on each side of the bottom center line of the binarized image, with a size of 100×250 pixels. Calculate the number of white pixels in each column. Find a maximum value on each side of the center line. The bottom point of the corresponding column is the base point of the sliding window scan. Construct a sliding window with the base point as the bottom center point. Step 5.2: After determining the first set of base points and windows, calculate the center points of the left and right windows based on the coordinate distribution of the white pixels within the windows, using the following formula: in The number of white pixels. For the first The coordinates of the white pixel The coordinates of the center point; The center point was used as the feature point for fitting the rice row lines. The projection of the center point into the sliding direction generates the next sliding base point; Step 5.3: Calculate the midpoint of the center points of the left and right windows, use it as the line fitting point, and use the least squares method to fit the line.

Images