Lane departure alarm system and method based on machine vision

Abstract

The invention discloses a lane departure alarm system and method based on machine vision, and the method comprises the steps: obtaining real-time image data in the running process of a vehicle based on an image sensor, extracting lane image feature points in the real-time image data, and obtaining lane feature image data; performing lane line fitting on the lane feature image data by using a CNN neural network to obtain real-time lane line data; lane line deviation is detected, and whether the vehicle driving state deviates from the real-time lane line data or not is judged; and if the vehicle driving state deviates from the real-time lane line data, carrying out vehicle driving early warning on a driver. The driving state of the vehicle can be monitored in real time, the situation that the vehicle possibly deviates from the lane is predicted, and an early warning or intervention instruction is sent to a driver when necessary.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving, and in particular to a lane departure warning system and method based on machine vision. Background Art

[0002] Typically, one or more image sensors provide multiple frames of images of the road, which are connected to multiple video ports of the processor. After the data enters the system, it is converted into a processable format in real time. Inside the processor, preprocessing is first performed to filter out the noise mixed in during image capture; then the position of the vehicle relative to the lane markings is detected, and the input information stream of the road image is converted into a series of lines that outline the road surface. The lane markings can be found by looking for edges in the data field. These edges actually form the boundaries that the vehicle should maintain when driving forward. The processor must always track these markings to determine whether the driving route is normal. Once it is found that the vehicle has accidentally deviated from the roadway, the processor makes a judgment and outputs a signal to drive the alarm circuit, allowing the driver to immediately correct the driving route. The alarm can be in the form of a buzzer or speaker, or a verbal prompt, or a vibration seat or steering wheel to remind the driver. The LDW system also takes into account the normal use of the car's braking and steering devices, which will affect the operation of the LDW and complicate the system. Therefore, the LDW system does not work when driving slowly or braking and turning normally.

[0003] However, in some complex road sections, due to environmental reasons, the lane lines are covered with mud, leaves, garbage, etc., resulting in recognition errors or false alarms.

[0004] The pipeline used by most traditional detection algorithms includes image preprocessing, feature extraction, lane model fitting, and line tracking. The purpose of image preprocessing is to remove some noise from the image. Feature extraction uses the features of lane lines to extract lane line-like regions. Lane models are then fitted and tracked using various methods. Feature detection is an important lane detection algorithm that affects the performance. Therefore, in many traditional methods used to determine the quality of features for lane detection tasks, the image preprocessing stage is required. The construction of the region of interest (ROI), image enhancement for extracting lane line information, and removal of non-lane line details are all part of image processing. ROI extraction methods effectively reduce redundant information in the image preprocessing part by selecting the lower part of the image. Some studies have created ROIs using vanishing point detection techniques. In addition, the creation of ROIs minimizes image noise, although it is not robust against shadows or cars. In the feature extraction process, specific features are extracted to detect lane lines, such as color, edges, geometry, etc. Several techniques such as inverse perspective map (IPM), perspective transformation, filtering techniques, edge detection-based techniques, image region extraction, morphological operators, feature points based on neighborhood search, grayscale, thresholding, and clustering are used.

[0005] In addition, heterogeneous operators and sliding windows have been used in the past to reduce the impact of noise and conveniently extract lane lines. The lane line model is then fitted using line segment detectors (LSD) and fitting-based methods, including B-spline, quadratic, polynomial, parabola, hyperbola, and least squares. Bresenham line voting space (BLVS), vanishing points, waveforms, geometric modeling, harmony search (HS) algorithm, contrast-limited adaptive histogram equalization (CLAHE), random sample consistency (RANSAC), graph-based seed filling algorithm, histogram analysis, model predictive control (MPC), a region-based iterative seed method, ant colony optimization, using scene understanding physical enhancement real-time (SUPER) method, nested fusion, and linear regression. Lucas Kanade method, Kanade Lucas Tomasi (KLT), and Lucas Kanad optical flow are matched with lane line models. Meanwhile, the most widely used algorithms for tracking road lane line detection are Kalman filter, lane line classification, and parabolic equation. Tracking is often used as a post-processing step to compensate for lighting fluctuations. Therefore, tracking contributes to the false detection of occlusions due to lane line labeling errors. Summary of the invention

[0006] The purpose of the present invention is to solve the above-mentioned problem and to design a lane departure warning system and method based on machine vision.

[0007] To achieve the above object, the technical solution of the present invention is that, further, in the above lane departure warning system based on machine vision, the lane departure warning system comprises the following steps:

[0008] An environment perception module, used to obtain real-time image data of a vehicle in motion based on an image sensor, extract lane image feature points from the real-time image data, and obtain lane feature image data;

[0009] A lane line detection module is used to perform lane line fitting on the lane feature image data using a CNN neural network to obtain real-time lane line data;

[0010] A deviation decision module is used to detect lane line deviation and determine whether the vehicle driving state deviates from the real-time lane line data;

[0011] The warning issuing module is used to issue a vehicle driving warning to the driver if the vehicle driving state deviates from the real-time lane line data.

[0012] Furthermore, in a lane departure warning method based on machine vision, the lane departure warning method comprises the following steps:

[0013] Acquire real-time image data of the vehicle while it is traveling based on an image sensor, extract lane image feature points from the real-time image data, and obtain lane feature image data;

[0014] Using a CNN neural network to perform lane line fitting on the lane feature image data to obtain real-time lane line data;

[0015] Detecting lane line deviation to determine whether the vehicle's driving state deviates from the real-time lane line data;

[0016] If the vehicle's driving state deviates from the real-time lane line data, a vehicle driving warning is issued to the driver.

[0017] Furthermore, in the above-mentioned lane departure warning method based on machine vision, the step of acquiring real-time image data of a vehicle in motion based on an image sensor, extracting lane image feature points in the real-time image data, and obtaining lane feature image data includes the following steps:

[0018] Using the intrinsic and extrinsic parameters of the image sensor for data conversion, the camera is installed at a distance h from the road plane, tilted downward;

[0019] The angle between the optical axis and the road plane axis is θ, and the world coordinate system {F w}={X w ,Y w ,Z w,1}, which is the intersection of the optical center of the camera and the road plane; define the camera coordinate system {F c}={X c ,Y c ,Z c ,1} and the image coordinate system {F i}={u i ,v i ,1};

[0020] Convert the road plane coordinates to the camera coordinates with a θ around X w Axis rotation, as -H / sinθ along Z w The camera calibration matrix contains the camera's intrinsic parameters, which are converted from camera coordinates to coordinates on the image plane. The 3D world coordinate point (X w ,Y w ,Z w ) and image projection (u i ,v i ) can be expressed as follows:

[0021] (u i ,v i ,1)=KTR(X w ,Y w ,Z w ,1) (1);

[0022] R represents the rotation matrix:

[0023]

[0024] T represents the transformation matrix:

[0025]

[0026] K represents the camera parameter matrix:

[0027]

[0028] f represents the focal length of the camera, s is the slope of the pixel, and (ku*kv) is the size ratio of the pixel. Formula (1) can also be expressed as:

[0029]

[0030] The real world coordinate system is the plane coordinate Y w =0, which can be simplified to:

[0031]

[0032] Use equation (6) to project a window of interest on a real-world road onto the image plane.

[0033] Furthermore, in the above-mentioned lane departure warning method based on machine vision, the step of acquiring real-time image data of a vehicle in motion based on an image sensor, extracting lane image feature points in the real-time image data, and obtaining lane feature image data includes the following steps:

[0034] Acquire real-time image data of the vehicle while it is traveling based on an image sensor, and perform grayscale processing on the real-time image data to obtain grayscale real-time image data;

[0035] Performing filtering and enhancement processing on the grayscale real-time image data to obtain enhanced real-time image data;

[0036] Extracting lane image feature points from the enhanced real-time image data to obtain lane feature image data;

[0037] The lane feature extraction formula is as follows:

[0038]

[0039] ∠ABC=sin -1 (1 / r) and tan∠ABC=ac / r. Assume that triangle ABC is a right triangle. Use the Pythagorean theorem to prove that (e+r) 2 =r 2 +ac 2 , substitute into (1) and simplify, according to the curvature r and edge error e, the detection range l is obtained;

[0040]

[0041] The method of establishing a search window in the region of interest ROI is to determine the width of the ROI based on the width of the lane line. The expression is as follows:

[0042] w i =αv i (9);

[0043] Where w i is the width of ROI at the i-th scan line; v i is the width of the lane marking line on the i-th scan line in the image coordinate system; α is a set constant, which represents the relationship between the ROI width and the marking line width, v i The lane marking line width in the world coordinate system is obtained by perspective transformation. Assuming that the lane marking line block structure width in the vehicle body coordinate system is v i ', then the transformation relationship between the two can be expressed as:

[0044] v i =F(v i ',i) (10);

[0045] Where F represents the perspective transformation relationship of the width of the sign line from the vehicle body coordinate system to the image coordinate system. In the image coordinate system, the basic rule of ROI width change is wide at the bottom and narrow at the top. For the lane line edge points searched by the search window, a constant threshold is set, and pixels below the threshold are set to the constant.

[0046] Furthermore, in the lane departure warning method based on machine vision, the use of a CNN neural network to perform lane line fitting on the lane feature image data to obtain real-time lane line data includes the following steps:

[0047] Acquire lane feature image data, and perform Hough transform on the ROI image in the lane feature image data;

[0048] Calculate the normalized intensity of the red, green and blue channels of the lane feature image data using the CNN neural network algorithm;

[0049] Obtain all vertical lines in the ROI image, and draw the vertical lines as red lines;

[0050] Kalman filtering is used to calculate the correlation between lane line positions between adjacent frames, and the information obtained from the previous frame image is used to guide the detection of lane lines in the next frame, thereby performing real-time tracking of lane lines.

[0051] Furthermore, in the above-mentioned lane departure warning method based on machine vision, the detecting of lane line deviation and judging whether the vehicle driving state deviates from the real-time lane line data comprises the following steps:

[0052] Lane departure is determined by the system by monitoring the position of the vehicle relative to the detected lane boundary;

[0053] The system issues a warning when the vehicle approaches the adjacent lane. By inverting the transformation matrix (6), the lane boundary coordinate mapping from the image plane to the road plane is calculated as follows:

[0054]

[0055] The obtained road plane coordinates are used to create the line equation of each lane boundary AX+BY+C=0. Using (12), the vertical distance d from the detected lane boundary to the vehicle origin (m, n) is calculated;

[0056]

[0057] Furthermore, in the lane departure warning method based on machine vision, if the vehicle driving state deviates from the real-time lane line data, the vehicle driving warning to the driver includes the following steps:

[0058] The origin of the vehicle is taken as a point on the road plane directly below the camera, where (M, N) = (0, 0). When the vehicle moves from the right lane to the left lane, in each forward image, the short lines represent the vehicle width and the long lines represent the detected lane boundaries. The digital display in the upper left corner indicates the (d-(W) of each detected lane boundary. v / 2)) value, W v represents the vehicle wheelbase. When the vehicle is near the lane marking, its value is 0. The system issues a lane departure warning, that is, when (12) is satisfied;

[0059]

[0060] Furthermore, in the lane departure warning method based on machine vision, if the vehicle driving state deviates from the real-time lane line data, the vehicle driving warning to the driver also includes the following steps:

[0061] The warning mechanism only focuses on the distance to the lane boundary, which is obtained directly from the image without complex calculations. The warning signal will only be triggered when the lane boundary approaches the center of the image.

[0062] Furthermore, in the lane departure warning method based on machine vision, if the vehicle driving state deviates from the real-time lane line data, the vehicle driving warning to the driver also includes the following steps:

[0063] The spatial mechanism is used to identify whether the vehicle is close to a dangerous position on the road. The implementation principle is as follows:

[0064] Define the warning box as a rectangle with the same width as the image and half the height of the image. The midpoint of the upper edge of the warning box is at the vanishing point obtained in the front. The definition of the danger zone is the area centered in the warning box and half the width of the image.

[0065] Assumption d M and d S They are the distances to the interception points M and S at the bottom of the warning box, and the red cross point p wi It is the intersection of the main lane extension line and the warning box line. wc is the turning point at the bottom of the warning box, d M Defined as p wi and p wc The distance between S Defined as the distance between lanes, when the red intersection p wi When outside the warning box, no matter M Or d S are considered to be 0, so d M and d SIt can be expressed by the following formula:

[0066]

[0067]

[0068] Space early warning mechanism in d M or S Greater than 1 / 4I w Start warning at 1:00 w and I h are the width and height of the image respectively.

[0069] Furthermore, in the lane departure warning method based on machine vision, if the vehicle driving state deviates from the real-time lane line data, the vehicle driving warning to the driver also includes the following steps:

[0070] The principle of the time mechanism is to detect d M and d S The value change of , L(t) represents the change of d from time t to time tn M and d S The mean of the sum, L(t) is expressed as follows:

[0071]

[0072] n is the number of frames used to calculate L(t), and the variable ζ is defined as:

[0073] ζ=l(t 1 )-l(t 2 ) (15).

[0074] Its beneficial effects are as follows: (1) By integrating the data of the lane departure warning system with the data of other vehicle sensors (such as cameras, radars, GPS, etc.), a more comprehensive understanding of the vehicle's surrounding environment can be achieved, thereby improving the vehicle's safety performance. By integrating the data of multiple sensors, the detection accuracy and stability of the system can be improved, while expanding the application scope of the system. For example, by integrating the data of the lane departure warning system with the data of the vehicle's blind spot monitoring system, the safety performance of the vehicle can be further improved. Data sharing and interconnection: The lane departure warning system can communicate intelligently with other vehicles and road infrastructure to achieve data sharing and interconnection. For example, when a vehicle deviates from its lane, the lane departure warning system of the vehicle can send warning signals to other vehicles and road infrastructure to remind other vehicles to pay attention to traffic safety. (2) Big data analysis: By mining and analyzing a large amount of lane departure warning system data, a lot of useful information can be obtained. For example, by conducting in-depth analysis of the deviation data of a certain section of road, important information such as traffic flow, vehicle driving habits, and driver behavior of the section of road can be obtained, providing strong support for road management and traffic planning. (3) Real-time warning and intervention: Using the data from the lane departure warning system, the vehicle's driving status can be monitored in real time, the vehicle may be predicted to deviate from the lane, and a warning or intervention instruction can be issued to the driver when necessary. For example, when the system detects that the vehicle is about to deviate from the lane, it can automatically or semi-automatically adjust the vehicle's driving trajectory to ensure the vehicle's driving safety. (4) Autonomous driving technology: The data from the lane departure warning system can be combined with autonomous driving technology to achieve a higher level of autonomous driving functions. For example, autonomous driving vehicles can share and connect data with the lane departure warning system to perceive the surrounding vehicles and road conditions in real time, automatically adjust the driving trajectory and speed, and ensure safe and smooth driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.

[0076] Figure 1 Schematic diagram of a first embodiment of a lane departure warning system based on machine vision in an embodiment of the present invention;

[0077] Figure 2 Schematic diagram of a first embodiment of a lane departure warning method based on machine vision in an embodiment of the present invention;

[0078] Figure 3 Schematic diagram of the system hardware framework of a lane departure warning method based on machine vision in an embodiment of the present invention;

[0079] Figure 4 A schematic diagram of camera shooting angles of a lane departure warning method based on machine vision in an embodiment of the present invention;

[0080] Figure 5 Schematic diagram of a first lane line detection in a lane departure warning method based on machine vision in an embodiment of the present invention;

[0081] Figure 6 Schematic diagram of a second lane line detection of a lane departure warning method based on machine vision in an embodiment of the present invention;

[0082] Figure 7 Schematic diagram of the third lane line detection of a lane departure warning method based on machine vision in an embodiment of the present invention;

[0083] Figure 8 Schematic diagram of lane line detection of a lane departure warning method based on machine vision in an embodiment of the present invention;

[0084] Fig. 9 1 is a first lane departure detection schematic diagram of a lane departure warning method based on machine vision in an embodiment of the present invention;

[0085] Fig.10 Schematic diagram of a second lane departure detection of a lane departure warning method based on machine vision in an embodiment of the present invention;

[0086] Fig.11 Schematic diagram of a third lane departure detection of a lane departure warning method based on machine vision in an embodiment of the present invention;

[0087] Fig.12 Schematic diagram of a first actual detection effect of a lane departure warning method based on machine vision in an embodiment of the present invention;

[0088] Fig.13 Schematic diagram of a second actual detection effect of a lane departure warning method based on machine vision in an embodiment of the present invention. DETAILED DESCRIPTION

[0089] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0090] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0091] The present invention will be described in detail below in conjunction with the accompanying drawings. Figure 1 As shown, a lane departure warning system based on machine vision includes the following modules:

[0092] An environment perception module is used to obtain real-time image data of a vehicle in motion based on an image sensor, extract lane image feature points from the real-time image data, and obtain lane feature image data;

[0093] The lane line detection module is used to use the CNN neural network to perform lane line fitting on the lane feature image data to obtain real-time lane line data;

[0094] The deviation decision module is used to detect lane line deviation and determine whether the vehicle's driving status deviates from the real-time lane line data;

[0095] The warning issuing module is used to issue a vehicle driving warning to the driver if the vehicle driving status deviates from the real-time lane line data.

[0096] Its beneficial effects are as follows: (1) By integrating the data of the lane departure warning system with the data of other vehicle sensors (such as cameras, radars, GPS, etc.), a more comprehensive understanding of the vehicle's surrounding environment can be achieved, thereby improving the vehicle's safety performance. By integrating the data of multiple sensors, the detection accuracy and stability of the system can be improved, while expanding the application scope of the system. For example, by integrating the data of the lane departure warning system with the data of the vehicle's blind spot monitoring system, the safety performance of the vehicle can be further improved. Data sharing and interconnection: The lane departure warning system can communicate intelligently with other vehicles and road infrastructure to achieve data sharing and interconnection. For example, when a vehicle deviates from its lane, the lane departure warning system of the vehicle can send warning signals to other vehicles and road infrastructure to remind other vehicles to pay attention to traffic safety. (2) Big data analysis: By mining and analyzing a large amount of lane departure warning system data, a lot of useful information can be obtained. For example, by conducting in-depth analysis of the deviation data of a certain section of road, important information such as traffic flow, vehicle driving habits, and driver behavior of the section of road can be obtained, providing strong support for road management and traffic planning. (3) Real-time warning and intervention: Using the data from the lane departure warning system, the vehicle's driving status can be monitored in real time, the vehicle may be predicted to deviate from the lane, and a warning or intervention instruction can be issued to the driver when necessary. For example, when the system detects that the vehicle is about to deviate from the lane, it can automatically or semi-automatically adjust the vehicle's driving trajectory to ensure the vehicle's driving safety. (4) Autonomous driving technology: The data from the lane departure warning system can be combined with autonomous driving technology to achieve a higher level of autonomous driving functions. For example, autonomous driving vehicles can share and connect data with the lane departure warning system to perceive the surrounding vehicles and road conditions in real time, automatically adjust the driving trajectory and speed, and ensure safe and smooth driving.

[0097] See also Figure 2 In a lane departure warning method based on machine vision, the lane departure warning method includes the following steps:

[0098] Step 201: acquiring real-time image data of a vehicle in motion based on an image sensor, extracting lane image feature points in the real-time image data, and obtaining lane feature image data;

[0099] Step 202: Use the CNN neural network to perform lane line fitting on the lane feature image data to obtain real-time lane line data;

[0100] Step 203: Detect lane line deviation to determine whether the vehicle driving state deviates from the real-time lane line data;

[0101] Step 204: If the vehicle's driving state deviates from the real-time lane line data, a vehicle driving warning is issued to the driver.

[0102] Specifically, the present invention also includes:

[0103] The lane departure warning system uses a camera installed in the cab to perform scene recognition, and based on the road environment ahead and the vehicle's position, it determines the vehicle's behavior of deviating from the lane and promptly reminds the driver, thereby preventing lane departure accidents caused by driver negligence. The lane departure warning system consists of three parts: environmental perception, deviation decision, and warning release

[0104] System Hardware Box Figure 3 as follows:

[0105] The MCU is composed of MPC567xK and S32V2.

[0106] MPC567xK: Ultra-Reliable MPC567xK MCU for Automotive ADAS Applications

[0107] S32V2 is used for visual processing and machine deep intelligence AI learning;

[0108] The overall data processing flow is as follows:

[0109] 1. Lane detection

[0110] To detect and identify lane lines, we first need to use some image processing methods to extract the feature points of the lane image, and then select a suitable curve fitting method to fit the lane lines using the obtained feature points. Its main purpose is to remove noise interference in the image, preserve the road surface information of the road image to the greatest extent, and reduce irrelevant information in the road image, thereby improving the reliability of lane line detection and recognition. Image preprocessing only changes the form of information, but does not increase the amount of information in the image.

[0111] like Figure 4 As shown in Figure 1, we assume a flat road and use the camera intrinsic (focal length, optical center, offset parameter pixels, aspect ratio) and extrinsic (pitch angle and ground height) parameters to complete the transformation. The camera is installed at a distance h from the road plane and tilted downward. Therefore, the angle between the optical axis and the road plane axis is θ, defining the world coordinate system {F w}={X w ,Y w ,Z w ,1}, which is the intersection of the optical center of the camera and the road plane; define the camera coordinate system {F c}={X c ,Y c ,Z c ,1} and the image coordinate system {F i}={u i ,v i ,1};

[0112] Convert the road plane coordinates to the camera coordinates with a θ around X w Axis rotation, as -H / sinθ along Z w The camera calibration matrix contains the camera's intrinsic parameters, which are converted from camera coordinates to coordinates on the image plane. The 3D world coordinate point (X w ,Y w ,Z w ) and image projection (u i ,v i ) can be expressed as follows:

[0113] (u i ,v i ,1)=KTR(X w ,Y w ,Z w ,1) (1);

[0114] R represents the rotation matrix:

[0115]

[0116] T represents the transformation matrix:

[0117]

[0118] K represents the camera parameter matrix:

[0119]

[0120] f represents the focal length of the camera, s is the slope of the pixel, and (ku*kv) is the size ratio of the pixel. Formula (1) can also be expressed as:

[0121]

[0122] The real world coordinate system is the plane coordinate Y w =0, which can be simplified to:

[0123]

[0124] Using equation (6), we can project a window of interest on a real-world road to the image plane.

[0125] Grayscale of road images

[0126] The images output from the front camera or acquisition card are generally color images. Since the amount of information contained in color images is too large, its processing requires a lot of calculation and high hardware requirements, which affects the real-time performance of the system. Therefore, it is generally necessary to grayscale the images output by the camera in the system. Each pixel in the color image based on the RGB model is composed of three components: R, G, and B. The value range of each component is 0 to 255, so each pixel can represent more than 16 million (256*256*256) colors. If the values ​​of the three components R, G, and B are set equal, then each pixel can only represent 256 colors. This is the principle of grayscale.

[0127] Filtering enhancement of road images

[0128] The roads in the real environment are very complex. Affected by factors such as lighting and occlusion, the images output by the camera generally contain a lot of noise interference. In order to minimize these interference factors, an image enhancement link is added to the image preprocessing process. Image enhancement refers to the use of a series of technologies to process and process the original image to make it more suitable for specific application requirements, improve the visual effect of the image, or convert the image into a form that is more suitable for human or machine analysis and processing. Image enhancement is a basic image preprocessing method. It is not based on image fidelity. It can be a distortion process. Using it does not mean increasing the information of the image, and sometimes even loses some information. The purpose of image enhancement is to expand the differences between individuals with different features in the image, suppress areas of no interest, and enhance features of interest, so as to make the detection and recognition of these features easier.

[0129] 2. Lane detection module

[0130] In order to calculate the region of interest in the input image, the detection range of the system must be known. When the vehicle is on a straight road or a road with a large radius of curvature, the detection range will span from the vehicle to the horizon in the image.

[0131] like Figure 5 As shown, the vehicle encounters a right turn with radius r. We want to find l, the maximum detection range of the system. In a "worst case" situation, the system is at point c, close to the lane boundary like a tangent line, and the lane marking separator is within the detection range. This results in an error distance e from the potential point of lane deviation. As can be seen from the graphical method, ∠ABC = sin -1 (1 / r) and tan∠ABC=ac / r.

[0132] Combining these two formulas:

[0133]

[0134] Since triangle ABC is a right triangle, we can use the Pythagorean theorem to prove that (e+r) 2 =r 2 +ac 2 .

[0135] Substituting into (1) and simplifying, we can obtain the detection range l based on the curvature r and the edge error e.

[0136]

[0137] The most stringent regulations require that the system must work when the radius of curvature of the road is less than or equal to 250 meters. It requires the latest warning line to be 0.3m outside the lane boundary and the lowest earliest warning line to be 0.75m inside the lane boundary. This results in a warning being possible in an area of ​​1.05 meters width. By setting the warning threshold at the center of this area (0.225m inside the lane boundary), there is an edge distance of e=0.525m. Substituting these values ​​into (8), the result is a detection range of 16.18m on the road surface. The road surface images are cropped at this point for all images, including those of curves and straight roads.

[0138] Region of interest (ROI)

[0139] A method for establishing a search window in a region of interest (ROI) is proposed. When determining the width of the ROI, the width of the lane line is used as the reference, that is,

[0140] w i =αv i (9)

[0141] Where w i is the width of ROI at the i-th scan line; v i is the width of the lane marking line on the i-th scan line in the image coordinate system; α is a set constant, which represents the relationship between the ROI width and the marking line width. i The lane marking line width in the world coordinate system is obtained by perspective transformation. Assume that the lane marking line block structure width in the vehicle body coordinate system is v i ', then the transformation relationship between the two can be expressed as

[0142] v i =F(v i ',i)(10)

[0143] Where F represents the perspective transformation relationship of the width of the sign line from the vehicle body coordinate system to the image coordinate system. In the image coordinate system, the basic rule of ROI width change is wide at the bottom and narrow at the top. For the lane line edge points searched by the search window, in order to reduce the influence of noise such as shadows, a constant threshold is set, and pixels below the threshold are set to the constant.

[0144] Lane Detection

[0145] like Figure 6 As shown in the figure, lane lines are the most important traffic signs in road traffic, which can play a role in restraining and protecting the driving of vehicles. Whether in vehicle safety driving systems or in intelligent vehicle navigation based on machine vision, lane line detection and recognition is a basic and necessary functional module. The more important part of the early warning algorithm is to fit the lane lines by detecting points on the edge lines based on the collected image information. However, in this process, due to the noise on the image and the reduction in clarity of the lane lines due to wear and tear, it is often impossible to obtain accurate results in the end. The CNN neural network algorithm is used to quickly and concisely detect the lane lines.

[0146] First, the normalized intensity of the red, green, and blue channels is calculated for each frame of video. The negative second-order derivative of a one-dimensional Gaussian filter is convolved with each row of the region of interest, from bottom to top, giving a response to the intensity change of the corresponding lane marking. Due to the convolution, the pixel values ​​at the place of lane markings increase. As the maximum input pixel value is 1, the system determines that pixels above this value are lane markings. The resulting image is as follows Figure 6 As shown in b.

[0147] The LK algorithm is based on the following three assumptions: brightness constancy, temporal persistence, and spatial coherence. The first assumption states that a pixel in the image of an object in the field of view does not change its appearance as it moves from frame to frame. That is, the brightness of the pixel does not change as it moves from one continuous frame to another. The second assumption only means that when the image motion changes slowly, that is, the movement of the object is very slow relative to the frame rate. The third assumption states that adjacent points in an image are always adjacent points. The LK method uses tracking points from one frame to another to determine the position of the vehicle relative to the center of the lane and the direction of the vehicle's movement.

[0148] like Figure 7 As shown, perform Hough transform on the ROI image, find all vertical lines on the image and draw them as red lines.

[0149] like Figure 8As shown, Hough transform is a feature extraction technique used in image analysis and computer vision. Its original version detects straight lines, and later Hough transform has been extended to recognize arbitrary shapes, the most common of which is a circle. Using this transform to detect straight and vertical lines must have a specific score to be counted. This is called the line score. In order to get the line score, the image is divided into columns and the average of the pixel values ​​in each column is calculated. This average is the line score of the line. Hough transform generally uses a voting scheme to detect the most likely lane edge. If the line with the most votes is above the threshold (that is, it meets a certain score requirement), then the line is considered to be a lane marking. This is done once for the left half of the image and once for the right half of the image to obtain images of the left lane marking and the right lane marking. Lane tracking (Kalman filter method)

[0150] Only when the detected cluster is tracked for more than five consecutive frames can the system eliminate unreal and short-lived clusters, such as pedestrians crossing the road or road lettering. For structured roads, the lane line positions in two consecutive frames are not much different. Therefore, the correlation of lane line positions between adjacent frames can be used to guide the detection of lane lines in the next frame with the information obtained from the previous frame, so as to achieve real-time tracking of lane lines. Kalman filtering is a commonly used lane tracking method 3. Lane departure detection module

[0151] Warning according to standards

[0152] Lane departure is determined by the system by monitoring the position of the host vehicle relative to the detected lane boundary, and the system issues a warning when the host vehicle approaches the adjacent lane. By inverting the transformation matrix (6), the lane boundary coordinate mapping from the image plane to the road plane is calculated as follows.

[0153]

[0154] The resulting road plane coordinates are used to create the line equation of each lane boundary AX+BY+C=0. Using (12), the vertical distance d from the detected lane boundary to the vehicle origin (m, n) can be calculated.

[0155]

[0156] like Fig. 9 As shown, the origin of the vehicle is taken as a point on the road plane directly below the camera, where (M, N) = (0, 0). When the vehicle moves from the right lane to the left lane, in each forward image, the short lines represent the vehicle width and the long lines represent the detected lane boundaries. The digital display in the upper left corner indicates the (d-(w v / 2)) value, w v represents the wheelbase of the vehicle. When the vehicle is near the lane marking, its value is 0. The system issues a lane departure warning when (12) is satisfied.

[0157]

[0158] Lane departure warning based on time-space mechanism

[0159] A dual mechanism warning algorithm is used to avoid false alarms. The first warning mechanism only focuses on the distance to the lane boundary, which can be obtained directly from the image without complex calculations. The warning signal will be triggered only when the lane boundary is very close to the center of the image, which means that the vehicle is between two lanes. Moreover, in order to predict some lane departure situations, rather than lane changes, another warning mechanism based on the lane distance change rate is proposed.

[0160] Generally speaking, dangerous lane departure situations come from the following two situations: the driver is too close to the lane boundary; the vehicle maintains a fast departure speed, such as the vehicle approaches the lane boundary too quickly. Therefore, a spatial mechanism is proposed to detect these two lane departure situations separately.

[0161] Space Mechanism

[0162] like Fig.10 As shown in Figure 1, the spatial mechanism is used to identify whether the vehicle is close to a dangerous position on the road. First, define the warning box as a rectangle with the same width as the image and half the height of the image. Then, the midpoint of the upper edge of the warning box is the vanishing point obtained in the front. The definition of the danger zone is the area centered in the warning box, whose width is half of the image. Fig.10 Warning boxes are indicated in blue and danger areas in red.

[0163] Assumption d M and d S They are the distances to the interception points M and S at the bottom of the warning box, and the red cross point p wi It is the intersection of the main lane extension line and the warning box line. wc is the turning point at the bottom of the warning box, d M Defined as p wi and p wc The distance between S Defined as the distance between lanes, when the red intersection p wi When outside the warning box, no matter M Or d S are considered to be 0, so d M and d S It can be expressed by the following formula:

[0164]

[0165]

[0166] Space early warning mechanism in d M or S Greater than 1 / 4I w Start warning at 1:00 w and I h are the width and height of the image respectively.

[0167] Time mechanism

[0168] like Fig.11 As shown, the principle of the time mechanism is to detect d M and d S The value of t is changed. L(t) represents the value of t from time t to time tn. M and d S The expression of L(t) is as follows:

[0169]

[0170] n is the number of frames used to calculate L(t), and the variable ζ is defined as:

[0171] ζ=l(t 1 )-l(t 2 ) (15).

[0172] Describe at two different times t 1 and t 2 are different.

[0173] Consider a vehicle lane change approaching its left boundary, such as Fig.11 As shown. 1 and t 2 Select an appropriate time interval based on Fig.11 (b) to Fig.11 (e) Continuously analyze d M and d S At the beginning, if Fig.11 (a) and 11(b), the vehicle begins to drift towards its left lane boundary. Fig.11 (a) and 11(b), d S The value of d becomes 0, M The value of increases, but d M and d S are all positive. Then, Fig.11 (b) to 11(e)d M Continue to increase. In general, Fig.11 (b) to 11(e), we can observe that the vehicle lane change causes dM and d S The heuristic rule used is to calculate ζ every nine frames, that is, n = 9. The speed of acquiring images from the experimental camera is 25 frames / s. Therefore, the proposed time warning mechanism is updated every 0.36s. Therefore, t 1 and t 2 The relationship between is defined as: 1 +0.36 = t 2 .

[0174] The time warning mechanism is designed as follows: if ζ>T l , then the system emits a buzzing warning sound. In this case, the vehicle quickly changes to another lane. In (13), the system triggers a warning when the vehicle approaches the lane boundary, relying on the distance threshold of the spatial warning mechanism. However, if the vehicle approaches the lane boundary quickly despite not reaching the distance threshold, the driver will not have enough time to adjust to prevent touching and crossing the lane boundary. In this case, the spatial warning mechanism will be triggered too late to ensure driving safety. Therefore, the time warning mechanism is used to provide an emergency warning before the normal warning of the spatial mechanism. Through (14)(15) and as described above Fig.11 In the triggering method of (a)-(e), if the ζ updated every 0.36s is greater than the threshold, the warning relying on the time mechanism will be triggered to avoid possible accidental collision.

[0175] The actual test results are as follows Fig.12 , Fig.13 Shown

[0176] Warning trigger test: The starting position in the lane should be approximately in the center of the lane. After the vehicle enters the lane and tracks smoothly, and the vehicle posture is stabilized, the vehicle should drift slowly inside and outside the track, while entering the curve at a speed of 20m / s to 22m / s (Class I) and 17m / s to 19m / s (Class II) according to the curve classification. The vehicle should leave once to the left and right in two departure rate ranges (0 to 0.4m / s and 0.4m / s to 0.8m / s) on the right and left curves, for a total of 8 departures, as shown in the ISO-17361 LDW system design requirements and Table 1 below.

[0177]

[0178] Table 1

[0179] Repeatability Test: The repeatability test shall be carried out on a straight road. The vehicle shall travel in a straight line at a speed of 20 m / s to 22 m / s. When traveling straight along the straight section, the vehicle can travel in the center of the lane or along the lane line, and the lane line is opposite to the lane marking to be crossed when deviating from the lane. For example, when deviating to the right from the lane, the vehicle can travel along the left lane line, and vice versa, as shown in Table 2 below. While maintaining the specified speed at the relevant level and the vehicle tracking the route smoothly and keeping its attitude stable, the vehicle shall turn and slowly drive out of the lane at a speed of 0.1 m / s < V1 + 0.05 < 0.3 m / s, and conduct 8 tests, 4 times to the left (Group 1) and 4 times to the right (Group 2). On the right (Group 2), and conduct another 8 tests at a speed of 0.6 m / s < (V2 ± 0.05) ≤ 0.8 m / s [4 times to the left (Group 3) and 4 times to the right (Group 4), so a total of 16 tests are carried out. V1 and V2 are selected by the OEM. The tester shall conduct the lane departure test according to the deviation rate tolerance given in Table 2 until four tests are achieved in each group.

[0180]

[0181] Table 2

[0182] Warning Failure Test: When driving in the no-warning zone, the system shall not give any warning within a total distance of 1000 meters on the straight road (which can be carried out in a 1000-meter section or two 500-meter sections).

[0183] Test Evaluation Warning Generation Test: The system shall give a warning before crossing the latest warning line, but shall not give a warning before crossing the earliest warning line of each test case. Repeatability Test: The system shall give a warning within an area with a width of 30 cm in each test group. No warning shall be given outside the warning threshold placement area. If a specific test group includes more than four tests within the specified speed tolerance range, only the first four tests within the specified speed tolerance range shall be considered. Warning Error Test: No warning shall occur between the two earliest warning lines.

[0184] Remarks: According to GB 5768 - 2009, the relevant parameters of Chinese roads are as follows: China - Lane boundary related parameters The lane width shall be between 3.0 meters and 3.75 meters. The width of the lane boundary should be 10 cm, 15 cm or 20 cm. The broken marking line should be -4 m (section) + 6 m (space) for the opposite direction. -2 m (section) + 4 m (space) for the same direction in urban areas. - 6 m (section) + 9 m (space) for the same direction on expressways.

[0185] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A lane departure warning system based on machine vision, characterized in that: The lane departure warning system includes the following modules: An environment perception module, used to obtain real-time image data of a vehicle in motion based on an image sensor, extract lane image feature points from the real-time image data, and obtain lane feature image data; A lane line detection module is used to perform lane line fitting on the lane feature image data using a CNN neural network to obtain real-time lane line data; A deviation decision module is used to detect lane line deviation and determine whether the vehicle driving state deviates from the real-time lane line data; The warning issuing module is used to issue a vehicle driving warning to the driver if the vehicle driving state deviates from the real-time lane line data.

2. A lane departure warning method based on machine vision, characterized in that: The lane departure warning method comprises the following steps: Acquire real-time image data of the vehicle while it is traveling based on an image sensor, extract lane image feature points from the real-time image data, and obtain lane feature image data; Using a CNN neural network to perform lane line fitting on the lane feature image data to obtain real-time lane line data; Detecting lane line deviation to determine whether the vehicle's driving state deviates from the real-time lane line data; If the vehicle's driving state deviates from the real-time lane line data, a vehicle driving warning is issued to the driver.

3. The lane departure warning method based on machine vision as claimed in claim 2, characterized in that: The method of acquiring real-time image data of a vehicle in motion based on an image sensor, extracting lane image feature points in the real-time image data, and obtaining lane feature image data comprises the following steps: Using the intrinsic and extrinsic parameters of the image sensor for data conversion, the camera is installed at a distance h from the road plane, tilted downward; The angle between the optical axis and the road plane axis is θ, and the world coordinate system {F w }={X w ,Y w ,Z w ,1}, which is the intersection of the optical center of the camera and the road plane; define the camera coordinate system {F c }={X c ,Y c ,Z c ,1} and the image coordinate system {F i }={u i ,v i ,1}; Convert the road plane coordinates to the camera coordinates with a θ around X w Axis rotation, as -H / sinθ along Z w The camera calibration matrix contains the camera's intrinsic parameters, which are converted from camera coordinates to coordinates on the image plane. The 3D world coordinate point (X w ,Y w ,Z w ) and image projection (u i ,v i ) can be expressed as follows: (u i ,v i ,1)=K.T.R(X w ,Y w ,Z w ,1)(1); R represents the rotation matrix: T represents the transformation matrix: K represents the camera parameter matrix: f represents the focal length of the camera, s is the slope of the pixel, and (ku*kv) is the size ratio of the pixel. Formula (1) can also be expressed as: The real world coordinate system is the plane coordinate Y w =0, which can be simplified to: Use equation (6) to project a window of interest on a real-world road onto the image plane.

4. The lane departure warning method based on machine vision as claimed in claim 2, characterized in that: The method of acquiring real-time image data of a vehicle in motion based on an image sensor, extracting lane image feature points in the real-time image data, and obtaining lane feature image data comprises the following steps: Acquire real-time image data of the vehicle while it is traveling based on an image sensor, and perform grayscale processing on the real-time image data to obtain grayscale real-time image data; Performing filtering and enhancement processing on the grayscale real-time image data to obtain enhanced real-time image data; Extracting lane image feature points from the enhanced real-time image data to obtain lane feature image data; The lane feature extraction formula is as follows: ∠ABC=sin -1 (1 / r) and tan∠ABC=ac / r. Assume that triangle ABC is a right triangle. Use the Pythagorean theorem to prove that (e+r) 2 =r 2 +ac 2 , substitute into (1) and simplify, according to the curvature r and edge error e, the detection range l is obtained; The method of establishing a search window in the region of interest ROI is to determine the width of the ROI based on the width of the lane line. The expression is as follows: w i =αv i (9); Where w i is the width of ROI at the i-th scan line; v i is the width of the lane marking line on the i-th scan line in the image coordinate system; α is a set constant, which represents the relationship between the ROI width and the marking line width, v i The lane marking line width in the world coordinate system is obtained by perspective transformation. Assuming that the lane marking line block structure width in the vehicle body coordinate system is v i ', then the transformation relationship between the two can be expressed as: v i =F(v i ',i)(10); Where F represents the perspective transformation relationship of the width of the sign line from the vehicle body coordinate system to the image coordinate system. In the image coordinate system, the basic rule of ROI width change is wide at the bottom and narrow at the top. For the lane line edge points searched by the search window, a constant threshold is set, and pixels below the threshold are set to the constant.

5. The lane departure warning method based on machine vision as claimed in claim 2, characterized in that: The lane line fitting is performed on the lane feature image data using the CNN neural network to obtain real-time lane line data. The following steps are involved: Acquire lane feature image data, and perform Hough transform on the ROI image in the lane feature image data; Calculate the normalized intensity of the red, green and blue channels of the lane feature image data using the CNN neural network algorithm; Obtain all vertical lines in the ROI image, and draw the vertical lines as red lines; Kalman filtering is used to calculate the correlation between lane line positions between adjacent frames, and the information obtained from the previous frame image is used to guide the detection of lane lines in the next frame, thereby performing real-time tracking of lane lines.

6. The lane departure warning method based on machine vision as claimed in claim 2, characterized in that: The lane line deviation detection and judging whether the vehicle driving state deviates from the real-time lane line data comprises the following steps: Lane departure is determined by the system by monitoring the position of the vehicle relative to the detected lane boundary; The system issues a warning when the vehicle approaches the adjacent lane. By inverting the transformation matrix (6), the lane boundary coordinate mapping from the image plane to the road plane is calculated as follows: The obtained road plane coordinates are used to create the line equation of each lane boundary AX+BY+C=0. Using (12), the vertical distance d from the detected lane boundary to the vehicle origin (m, n) is calculated; 7. The lane departure warning method based on machine vision as claimed in claim 2, characterized in that: If the vehicle driving state deviates from the real-time lane line data, the vehicle driving warning to the driver includes the following steps: The origin of the vehicle is taken as a point on the road plane directly below the camera, where (M, N) = (0, 0). When the vehicle moves from the right lane to the left lane, in each forward image, the short lines represent the vehicle width, and the long lines represent the detected lane boundaries. The digital display in the upper left corner indicates the (d-(w) of each detected lane boundary. v / 2)) value, w v represents the vehicle wheelbase. When the vehicle is near the lane marking, its value is 0. The system issues a lane departure warning, that is, when (12) is satisfied; 8. The lane departure warning method based on machine vision as claimed in claim 2, characterized in that: If the vehicle driving state deviates from the real-time lane line data, the vehicle driving warning to the driver also includes the following steps: The warning mechanism only focuses on the distance to the lane boundary, which is obtained directly from the image without complex calculations. The warning signal will only be triggered when the lane boundary approaches the center of the image.

9. The lane departure warning method based on machine vision as claimed in claim 2, characterized in that: If the vehicle driving state deviates from the real-time lane line data, the vehicle driving warning to the driver also includes the following steps: The spatial mechanism is used to identify whether the vehicle is close to a dangerous position on the road. The implementation principle is as follows: Define the warning box as a rectangle with the same width as the image and half the height of the image. The midpoint of the upper edge of the warning box is at the vanishing point obtained in the front. The definition of the danger zone is the area centered in the warning box and half the width of the image. Assumption d M and d S They are the distances to the interception points M and S at the bottom of the warning box, and the red cross point p wi It is the intersection of the main lane extension line and the warning box line. wc is the turning point at the bottom of the warning box, d M Defined as p wi and p wc The distance between S Defined as the distance between lanes, when the red intersection p wi When outside the warning box, no matter M Or d S are considered to be 0, so d M and d S It can be expressed by the following formula: Space early warning mechanism in d M or S Greater than 1 / 4I w Start warning at w and I h are the width and height of the image respectively.

10. The lane departure warning method based on machine vision as claimed in claim 2, characterized in that: If the vehicle driving state deviates from the real-time lane line data, the vehicle driving warning to the driver also includes the following steps: The principle of the time mechanism is to detect d M and d S The value change of , L(t) represents the change of d from time t to time tn M and d S The mean of the sum, L(t) is expressed as follows: n is the number of frames used to calculate L(t), and the variable ζ is defined as: ζ=l(t1)-l(t2)(15).

Images