A real-time path correction method for agricultural machinery based on visual lane detection

Through visual lane detection technology, the lateral offset distance and yaw angle of the agricultural machinery are calculated using a binocular camera, which solves the problem of low detection accuracy of lane line on farmland roads, and realizes the real-time lane maintenance function of the agricultural machinery in complex environments.

CN114399748BActive Publication Date: 2024-11-22SOUTHEAST UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202210062245.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-19
Publication Date
2024-11-22
Estimated Expiration
2042-01-19

AI Technical Summary

Technical Problem

On farmland roads, it is difficult to extract and maintain lane lines of autonomous driving agricultural locomotives. The prior art relies on expensive sensor equipment and has low recognition accuracy in complex farmland environments, so it cannot be directly applied to farmland roads.

Method used

Using vision-based lane detection method, images are acquired in real time through vehicle-mounted cameras, lateral offset distance and yaw angle of the vehicle are calculated using binocular cameras, short-distance transverse strip window sliding method and inverse perspective transformation technology, combined with Hough transformation Fitting with least squares, realizes lane line detection and deviation calculation, and outputs data to the control system in real time.

Benefits of technology

It improves the accuracy and robustness of lane line detection of agricultural machinery on farmland roads, realizes real-time lane keeping function, and enhances the stability of autonomous driving in complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114399748B_ABST
    Figure CN114399748B_ABST
Patent Text Reader

Abstract

The present invention discloses a real-time path correction method for agricultural machinery based on visual lane detection, which relates to the field of vehicle automatic driving. Based on the existing feature-based lane line detection method, the present invention improves the detection process and proposes a method for detecting lane lines by sliding a transverse strip window. The detection of the selected small window is not only suitable for straight line detection, but also has good robustness and accuracy for detection of curves, etc. At the same time, for the detected lane lines, a method for calculating the lateral offset distance and yaw angle of the agricultural machinery during driving using binocular vision is proposed. The method outputs data to the controller in real time, and the controller makes decisions to control the agricultural machinery, thereby realizing the lane keeping function.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image processing and automatic driving, and in particular relates to a real-time path correction method for agricultural machinery based on visual lane detection. Background Art

[0002] In recent years, with the update of software algorithms, hardware upgrades and the support of national policies, the technology in the field of autonomous driving has developed rapidly. The premise of autonomous driving is driving within the specified area, and intelligent lane recognition and vehicle route correction based on the recognized lane lines are important components.

[0003] At present, the research on unmanned driving of road vehicles is quite extensive, but in the agricultural field, especially on farmland roads, there are few studies on automatic driving. Due to the complexity of farmland roads, the automatic driving related technologies on roads cannot be directly transplanted to farmland roads. As for lane line extraction, the characteristics of farmland roads are mostly one-way streets, irregular road edges, and uneven road surfaces, which increase the difficulty of lane line extraction during the driving of agricultural machinery. At the same time, farmland roads generally have fewer people and vehicles, agricultural machinery travels slowly, and agricultural machinery generally operates in good weather. The conditions make the images obtained by the camera less noisy and easy to process. In addition, the unmanned driving technology of automobiles uses expensive sensor equipment such as laser radar, which is not convenient to promote and use in the agricultural field. However, the ranging range of binocular cameras is tens of meters, the price is low, and it meets the application scenarios. Therefore, using cameras to obtain images and measure distances has great advantages in realizing lane departure warning and lane keeping of agricultural machinery.

[0004] In the planned road, the lane is planned by two parallel boundary lines on the left and right and extends to the distance. Lane line recognition and tracking technology can be divided into two categories. One is a feature-based method, which uses the edge, texture and other features of the lane line to process and enhance the features, and then fits the lane line by designing a lane line model. Common models include straight lines, parabolas, spline curves, etc. However, direct use of this method has inaccurate feature extraction and inaccurate models, resulting in low fitting accuracy. Another method is a deep learning method that annotates large sample training data and designs neural networks for training to achieve the purpose of identifying lane lines. This method requires a large amount of sufficient data for training. Farmland roads have diverse features and there is a situation where features are lost.

[0005] The boundary of the road constrains the lateral position of the agricultural machinery, and the extension direction of the lane is also the direction of the agricultural machinery. The center line of the road is also the predetermined route for the vehicle. After determining the lane line, on the one hand, the lateral position of the vehicle can be determined based on the lane line information, and it can be judged whether it deviates too far from the center line to achieve lane departure warning. On the other hand, according to the position of the lane line, combined with the depth information obtained by the stereo camera, the vehicle's yaw angle can be calculated. The data is sent back to the control system in real time, and the control system can make decisions based on the obtained lateral deviation distance and yaw angle to achieve lane keeping. The methods adopted by the existing schemes are mainly used on highway pavement, and rely on the slope of the straight line detected by the lane line to qualitatively analyze whether it is deviated. This detection method does not quantitatively give the lateral offset distance, has poor accuracy, and low robustness. On farmland roads, the reliability of information is low. Summary of the invention

[0006] To solve the above problems, the present invention discloses a real-time path correction method for agricultural machinery based on visual lane detection. According to binocular visual ranging, the lateral offset distance and yaw angle of the agricultural machinery during driving are given to the control system in real time, thereby achieving the purpose of real-time correction of the vehicle's driving direction.

[0007] To achieve the above object, the technical solution of the present invention is as follows:

[0008] A real-time path correction method for agricultural machinery based on visual lane detection includes two steps:

[0009] 1. Lane detection: The vehicle-mounted camera is used to obtain the image of the road in real time, and the fitting lane line is obtained through the short-distance horizontal strip window sliding method. Based on the fitting lane line, the center line of the road is obtained;

[0010] 2. Lane trajectory deviation calculation and control: The binocular camera synchronously collects images from the left and right cameras to calculate the parallax between the images, thereby estimating the depth of each pixel. The actual distance between the lane line pixel on the image and the camera center can be calculated. The lateral yaw distance and yaw angle of the camera center can be obtained through calculation, and then output to the control system in real time to control the vehicle turning angle.

[0011] Furthermore, for lane line detection, the details are as follows:

[0012] Using the horizontal strip window sliding method of a single image, the lane line is considered to be a straight line in a small area, and edge detection, Gaussian blur, and Hough transform can be used for straight line fitting. At the same time, the center point of the lane line is calculated based on the fitting points to obtain the lane centerline, which is the vehicle's scheduled driving route.

[0013] Step 1: Camera calibration and coordinate system transformation. The lane lines in the image obtained by the camera are two inclined straight lines, which are inconsistent with the actual lane lines. Here, inverse perspective transformation is used to eliminate the camera's perspective effect of large near and small far.

[0014] There are mainly three coordinate systems involved, the pixel coordinate system, the camera coordinate system and the world coordinate system. The coordinates in the world coordinate system are converted to the camera coordinate system. After obtaining the coordinates in the camera coordinate system, they are converted to the coordinate system of the two-dimensional image. The conversion formula is as follows:

[0015]

[0016] Among them, Z c is a constant, f is the focal length of the camera, (u,v) is the coordinate in the pixel coordinate system, the origin of the coordinate system (u0,v0) is the upper left corner of the image, (X,Y,Z) is the coordinate in the world coordinate system, dx, dy are the unit sizes of the image along the u-axis and v-axis, R is the rotation matrix, and t is the translation amount.

[0017] When obtaining parameters, it can be considered that

[0018]

[0019] K is a 3×4 matrix with 12 sets of parameters. So we take 6 sets of corresponding points, list the equations by the method of undetermined coefficients, and find the least squares solution K. After the transformation, in the v direction of the image, the spacing between the points in the world coordinate system corresponding to the equidistant pixels is also the same;

[0020] Step 2: Image preprocessing: Process the inverse transformed image. To reduce the amount of calculation, grayscale the image to obtain a grayscale image. Considering that the image obtained by the camera has many noise points, Gaussian filtering is required to reduce the impact of noise. Then, edge Canny edge detection is used to extract typical edge features of the image.

[0021] Step 3: Perform pixel distance segmentation on the entire image along the lane line direction. Since the actual distance between pixels in the lane line direction corresponds one-to-one with the image pixels along the v direction on the image after inverse perspective processing, the segmentation is performed at equal distances. The segmentation points correspond to R0, R1, R2…R i ...The segmentation distance cannot be too small, so i cannot be too large;

[0022] Step 4: Perform Hough transform on the unit distance image fragment from the bottom to the top, and detect the two coordinates R of all the lines in the area of ​​interest in the image fragment. i R i+1 (u i ,v i ), R i Ri+1 (u i+1 ,v i+1 ), the slope of the detection line can be determined from these two points;

[0023] Step 5: Detect points in the ROI area of ​​the small window. Considering that there will be points perpendicular to the u axis after the inverse transformation, the reciprocals of all slopes and their mean k' are calculated here, and the reciprocals of all straight line slopes k are calculated. ij Compared with the mean, the deviation threshold δ is set. If the deviation is large, the coordinate point corresponding to the slope is discarded, otherwise it is retained;

[0024]

[0025] |k ij -k'|<δ (4)

[0026] Step 6: Perform least squares fitting on the remaining points to get the straight line corresponding to the lane line. The size of the lane line coordinate u can be used to determine whether it is the left lane line or the right lane line;

[0027] Step 7: Based on the obtained lane lines, in the small window ROI area, take two points on the left and right lane lines respectively, calculate the midpoint, and obtain the center line of the lane through least squares fitting. The center line of the lane is also the scheduled driving route of the vehicle.

[0028] Furthermore, the lane trajectory deviation calculation and control are as follows:

[0029] 1. Calculate the vehicle deflection distance

[0030] The vehicle's driving trajectory defaults to the center line of the lane line. Considering that the vehicle may deviate from the center line during driving, the present invention provides a method for calculating the distance of the vehicle's deviation from the center line. After the camera acquires the image and undergoes inverse perspective transformation, the detected lane lines are parallel to each other, and the lane lines detected in the ROI area of ​​the small window on the image can be considered as straight lines.

[0031] Step 1: Measure the distance between the two lane lines as d, then the distance between the center line and one side of the lane line is

[0032] Step 2: In the pixel coordinate system, take points P1(u1,v1), P2(u2,v2)… on the lane line in the ROI area within the window at equal distances. Since the world coordinate system is established on the ground and the lane line is also on the ground, the value of the coordinate z is considered to be 0. Combined with the binocular camera, the coordinates P1'(x1,y1,z1), P2'(x2,y2,z2)… in the three-dimensional world coordinate system can be restored. The distance between the two points is:

[0033]

[0034] Step 3: Based on the coordinates of the pixel points P1, P2..., the binocular camera calculates the disparity between the images by synchronously collecting the images of the left and right cameras, thereby estimating the depth of each pixel. In this way, the actual distance between the camera center and the pixel corresponding to the three-dimensional coordinates of the world coordinates can be obtained, that is, |P1'O'|, |P2'O'|, |P3'O'|... Then in the triangle |P1'P2'O'|, the cosine theorem shows that the angle α between one side of the lane line and P1'O' satisfies:

[0035]

[0036] Then the distance △d from the center line is:

[0037]

[0038] The positive or negative sign of △d represents whether it is left-biased or right-biased.

[0039] 2. Calculate the vehicle yaw angle

[0040] The driving vehicle has deviated from the route. Based on the calculation of the vehicle's deviation distance, the yaw angle can be further obtained. First, the time interval is discretized into △t. According to the displacement of the vehicle in the time △t on the deviated route is △s, and then the distance between the vehicle and the lane line at the previous moment and the current moment is calculated, the yaw angle of the vehicle during driving can be obtained.

[0041] Step 1: Discretize time. At each moment, the distance between the vehicle and one side of the lane line needs to be calculated. Here, |P' 0(k-1) O' (k-1) |,|P' 0k O' k | is the distance between the vehicle and one side of the lane line at the previous moment and the current moment;

[0042] Step 2: The vehicle's own odometer can output the travel distance information in real time, that is, the distance traveled by the vehicle within △t is △s;

[0043] Step 3: Vehicle from O' (k-1) Move the point to O' k Point, displacement is △s, through O' k Draw a perpendicular line from point P' 0(k-1) O' (k-1) At point Q, in triangle O' (k-1) O' k Q, we know that the yaw angle β is:

[0044]

[0045] Thus, the lateral offset distance △d and the yaw angle β of the vehicle are obtained. The obtained data can be output to the control module in real time, and the vehicle steering can be controlled in real time.

[0046] The beneficial effects of the present invention are:

[0047] The method for real-time path correction of agricultural machinery based on visual farmland lane detection described in the present invention improves the detection process on the basis of the existing feature-based lane line detection method, and proposes a method for detecting lane lines by sliding a transverse strip window. The detection of the selected small window is not only suitable for straight line detection, but also has good robustness and accuracy for detection of curves, etc. At the same time, for the detected lane lines, a method for calculating the lateral offset distance and yaw angle of the vehicle during driving using binocular vision is proposed. The method outputs data to the controller in real time, and the controller makes decisions to control the vehicle, thereby realizing the lane keeping function. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 System structure schematic diagram.

[0049] Figure 2 Schematic diagram of perspective coordinate transformation.

[0050] Figure 3 Actual picture of the camera installation location on agricultural machinery.

[0051] Figure 4 Schematic diagram of lateral offset distance.

[0052] Figure 5 Schematic diagram of calculating vehicle yaw angle. DETAILED DESCRIPTION

[0053] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0054] The present invention improves the existing lane line detection process and detects lane lines from the ROI area of ​​the sliding window at the bottom of the image to achieve better detection efficiency and enhance robustness. According to the characteristics of the detected lane lines using a binocular camera to estimate the image depth, a method for obtaining the vehicle's lateral offset distance and yaw angle based on the image depth information is proposed.

[0055] For lane line detection, the following steps are specifically included:

[0056] Step 1: Figure 2 and Figure 3 As shown, fix the position of the binocular camera, measure the position information of the camera installation, and calibrate the binocular camera. Software calibration can be used;

[0057] Step 2: Image preprocessing: Select one of the binocular cameras and perform inverse perspective transformation on the image. To reduce the amount of calculation, grayscale the image to obtain a grayscale image. Considering that the image obtained by the camera has many noise points, Gaussian filtering is required to reduce the impact of noise. Then, edge Canny edge detection is used to extract typical edge features of the image.

[0058] Step 3: Segment the entire image by pixel distance along the lane line direction. Since the actual distance between pixels in the lane line direction corresponds one-to-one with the image pixels along the v direction on the image after inverse perspective processing, it can be segmented at equal distances. The segmentation points correspond to R0, R1, R2…R i ...The segmentation distance cannot be too small, so i cannot be too large;

[0059] Step 4: Perform Hough transform on the unit distance image fragment from the bottom to the top, and detect the two coordinates R of all the lines in the area of ​​interest in the image fragment. i R i+1 (u i ,v i ), R i R i+1 (u i+1 ,v i+1 ), the slope of the detection line can be determined from these two points;

[0060] Step 5: Detect points in the ROI area of ​​the small window. Considering that there will be points perpendicular to the u axis after the inverse transformation, the reciprocals of all slopes and their mean k' are calculated here, and the reciprocals of all straight line slopes k are calculated. ij Compared with the mean, the deviation threshold δ is set. If the deviation is large, the coordinate point corresponding to the slope is discarded, otherwise it is retained;

[0061]

[0062] |k ij -k'|<δ (2)

[0063] Step 6: Perform least squares fitting on the remaining points to get the straight line corresponding to the lane line. The size of the lane line coordinate u can be used to determine whether it is the left lane line or the right lane line;

[0064] Step 7: Based on the obtained lane lines, in the small window ROI area, take two points on the left and right lane lines respectively, calculate the midpoint, and obtain the center line of the lane through least squares fitting. The center line of the lane is also the scheduled driving route of the vehicle.

[0065] For lane trajectory deviation calculation module.

[0066] 1. Calculate the vehicle deflection distance

[0067] The driving trajectory of the vehicle should follow the center line of the lane line. Considering that the vehicle may deviate from the center line during driving, the present invention provides a method for calculating the distance of the vehicle offset from the center line. After the image acquired by the camera is subjected to inverse perspective transformation, the detected lane lines are parallel to each other, and the lane lines detected in the ROI area of ​​the small window on the image can be considered as straight lines.

[0068] Step 1: Figure 4 As shown in the figure, the distance between the two lane lines is measured as d, then the distance between the center line and one side of the lane line is

[0069] Step 2: In the pixel coordinate system, take points P1(u1,v1), P2(u2,v2)… on the lane line in the ROI area within the window at equal distances. Since the world coordinate system is established on the ground and the lane line is also on the ground, the value of the coordinate z is considered to be 0. The pixel coordinates can be reversed to obtain the coordinates P1'(x1,y1,z1), P2'(x2,y2,z2)… in the world coordinate system. The distance between the two points is:

[0070]

[0071] Step 3: Based on the coordinates of the pixel points P1, P2..., the binocular camera calculates the disparity between the images by synchronously collecting the images of the left and right cameras, thereby estimating the depth of each pixel. In this way, the actual distance between the camera center and the pixel corresponding to the three-dimensional coordinates of the world coordinates can be obtained, that is, |P1'O'|, |P2'O'|, |P3'O'|... Then in the triangle |P1'P2'O'|, the cosine theorem shows that the angle α between one side of the lane line and P1'O' satisfies:

[0072]

[0073] Then the distance △d from the center line is:

[0074]

[0075] exist Figure 4 In the equation, the positive or negative sign of △d represents whether it is left-biased or right-biased.

[0076] 2. Calculate the vehicle yaw angle

[0077] The driving vehicle has deviated from the route. Based on the calculation of the vehicle's deviation distance, the yaw angle can be further obtained. First, the time interval is discretized into △t. According to the displacement of the vehicle in the time △t on the deviated route is △s, and then the distance between the vehicle and the lane line at the previous moment and the current moment is calculated, the yaw angle of the vehicle during driving can be obtained.

[0078] Step 1: Discretize time. At each moment, the distance between the vehicle and one side of the lane line needs to be calculated. Here, |P' 0(k-1) O' (k-1) |,|P' 0k O' k | is the distance between the vehicle and one side of the lane line at the previous moment and the current moment;

[0079] Step 2: Figure 5 As shown, the vehicle's own odometer can output the travel distance information in real time, that is, the distance traveled by the vehicle within △t is △s;

[0080] Step 3: Vehicle from O' (k-1) Move the point to O' k Point, displacement is △s, through O' k Draw a perpendicular line from point P' 0(k-1) O' (k-1) At point Q, in triangle O' (k-1) O' k Q, we know that the yaw angle β is:

[0081]

[0082] Thus, the lateral offset distance △d and the yaw angle β of the vehicle are obtained. The obtained data can be output to the control module in real time, and the vehicle steering can be controlled in real time.

[0083] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A real-time path correction method for agricultural machinery based on visual lane detection, characterized in that: There are two steps:

1. Lane detection: Calibrate the camera by determining its installation position, and then obtain a bird's-eye view of the road in real time through inverse perspective transformation. Perform image grayscale processing, Gaussian filtering, and edge detection processing. Obtain the fitted lane line through a short-distance horizontal strip window sliding method. Obtain the center line of the road based on the fitted lane line, which is the vehicle's scheduled driving route.

2. Lane trajectory deviation calculation and control: The binocular camera synchronously collects images from the left and right cameras to calculate the parallax between the images, thereby estimating the depth of each pixel and calculating the actual distance between the lane line pixel point on the image and the camera center. The lateral yaw distance and yaw angle of the camera center are calculated and then output to the control system in real time to control the vehicle turning angle; The lane trajectory deviation calculation and control are as follows: (1) Calculate the lateral offset distance of the vehicle Step 1: Measure the distance between the two lane lines as d, then the distance between the center line and one side of the lane line is Step 2: In the pixel coordinate system, take points P1(u1,v1), P2(u2,v2)… on the lane line in the ROI area within the window at equal distances. Since the world coordinate system is established on the ground and the lane line is also on the ground, the value of the coordinate z is considered to be 0. Combined with the binocular camera, the coordinates P1'(x1,y1,z1), P2'(x2,y2,z2)… in the 3D world coordinate system are restored. The distance between the two points is: Step 3: According to the coordinates of the pixel points P1, P2..., the binocular camera calculates the disparity between the images by synchronously collecting the images of the left and right cameras, thereby estimating the depth of each pixel; in this way, the actual distance between the camera center and the pixel point corresponding to the three-dimensional coordinates of the world coordinates is obtained, that is, |P1'O'|, |P2'O'|, |P3'O'|... Then, in the triangle |P1'P2'O'|, the cosine theorem shows that the angle α between one side of the lane line and P1'O' satisfies: Then the distance Δd from the center line is: The sign of Δd represents whether it is left-biased or right-biased. (2) Calculate the vehicle yaw angle The driving vehicle has deviated from the route. On the basis of calculating the vehicle's deviation distance, the yaw angle is further obtained. First, the time interval is discretized into Δt. According to the displacement of the vehicle in the time of Δt on the deviated route is Δs, the distance between the vehicle and the lane line at the previous moment and the current moment is calculated to obtain the vehicle's yaw angle during driving. Step 1: Discretize time. At each moment, the distance between the vehicle and one side of the lane line needs to be calculated; here |P' 0(k-1) O' (k-1) |,|P' 0k O' k | is the distance between the vehicle and one side of the lane line at the previous moment and the current moment; Step 2: The vehicle's own odometer outputs the travel distance information in real time, that is, the distance traveled by the vehicle within Δt is Δs; Step 3: Vehicle from O' (k-1) Move the point to O' k point, the displacement is Δs, through O' k Draw a perpendicular line from point P' 0(k-1) O' (k-1) At point Q, in triangle O' (k-1) O' k In Q, the yaw angle β is known to be: Thus, the lateral offset distance Δd and the yaw angle β of the vehicle are obtained; the obtained data are output to the control module in real time to control the steering of the vehicle in real time.

2. The method for real-time path correction of agricultural machinery based on visual lane detection according to claim 1, characterized in that: The lane line detection described in step 1 is as follows: Step 1: Preprocess the image obtained by the camera and extract edge features. Perform pixel distance segmentation on the entire image along the lane line direction. Since the actual distance between pixels in the lane line direction corresponds one-to-one with the image pixels along the v direction on the image after inverse perspective processing, equidistant segmentation is performed. The segmentation points correspond to R0, R1, R2…R i ...The segmentation distance cannot be too small, so i cannot be too large; Step 2: Perform Hough transform on the unit distance image fragment from the bottom to the top, and detect the two coordinates R of all the lines in the area of ​​interest in the image fragment. i R i+1 (u i ,v i ), R i R i+1 (u i+1 ,v i+1 ), then the slope of the straight line is determined; Step 3: Detect points in the ROI area of ​​the small window. Considering that there will be points perpendicular to the u axis after the inverse transformation, the reciprocals of all slopes and their mean k' are calculated here, and the reciprocals of all straight line slopes k are calculated. ij Compared with the mean, the deviation threshold δ is set. If the deviation is large, the coordinate point corresponding to the slope is discarded, otherwise it is retained; |k ij -k'|<δ (2) Step 4: Perform least squares fitting on the retained points to obtain the straight line corresponding to the lane line; determine whether it is the left lane line or the right lane line based on the size of the lane line coordinate u; based on the obtained lane line, take two points on the left and right lane lines respectively in the small window ROI area, calculate the midpoint, and obtain the center line of the lane through least squares fitting; the center line of the lane is also the vehicle's scheduled driving route.

Citation Information

Patent Citations

  • Road-vehicle fusion perception based early warning method against lane departure

    CN110517521A

  • Multi-strategy adaptive lane line detection method based on stereoscopic vision

    CN110569704A

  • Lane departure automatic measurement method based on inverse perspective principle

    CN110647850A