A vehicle lane departure warning method and system in nighttime scenes

Abstract

This invention discloses a lane departure warning method and system for use in nighttime scenarios. Developed primarily based on monocular vision and deep learning methods, the system includes: using an on-board camera to capture images of the road ahead and using the Res2Net50-VHA network to detect lane marking feature points on both sides of the current lane; using a Kalman filter to track lane marking feature points in preceding and following frame images; applying a combined fitting method of quadratic polynomials and linear equations to fit the upper and lower portions of the lane markings; calculating relevant parameters, including road curvature, lane centerline equation, vehicle lateral distance from the lane centerline, and yaw angle; and combining these parameters to determine the vehicle's driving state and implement a lane departure warning strategy. The algorithm employed in this invention boasts high recognition accuracy, high computational efficiency, and strong anti-interference capabilities, and the program outputs intuitive, real-time visualizations.

Description

Technical Field

[0001] The present invention relates to the field of driving safety assistance technology, and in particular to a vehicle lane departure warning method and system in nighttime scenarios. Background Art

[0002] According to statistics, traffic accident deaths in my country total approximately 90,000 people annually, accounting for 1.5% of the total death toll. Traffic accidents are caused by a variety of factors, with driver-related factors accounting for the vast majority. Driver fatigue or inattention can limit driver perception, making subjective factors prone to bias and misoperation, leading to accidents. Relevant data indicates that over 80% of traffic accidents are caused by drivers.

[0003] With the advancement of technology, some cars are now equipped with lane departure warning systems. When the system is activated, a camera mounted on the vehicle captures images of the road ahead. The video stream is then fed into a computing platform for processing, which determines the vehicle's position in the current lane. Vehicle status sensors collect information such as speed and steering status, and the control unit's decision-making algorithm determines whether the vehicle is drifting. If this occurs, an alarm is triggered.

[0004] The survey found that the current lane departure warning system has the following problems:

[0005] (1) Some lane departure warning systems use multiple sensors, which is costly;

[0006] (2) The lane line detection accuracy is high under good lighting conditions, but it is significantly reduced in night scenes;

[0007] (3) For driving situations such as changing lanes and overtaking, the warning system does not provide relevant response strategies. Some lane departure warning systems only use a single warning method, resulting in an inflated false alarm rate, drivers turning a deaf ear to the warning signals, and poor warning effects. Summary of the Invention

[0008] In order to overcome the shortcomings and deficiencies of the existing technology, the present invention provides a method and system for warning lane departure of vehicles in night scenes, which solves the problems of poor lane departure warning effect and incomplete decision-making in night scenes.

[0009] The present invention adopts the following technical solutions:

[0010] A lane departure warning method for a vehicle in a nighttime scenario, comprising:

[0011] Collect the road ahead image, and after preprocessing, use the Res2Net50-VHA network to detect the feature points of the lane lines on both sides of the current lane;

[0012] Linear interpolation is used to supplement the feature points of the lane lines on both sides, and Kalman filtering is used to track the lane feature points of the front and back frame images;

[0013] Using a combination of straight line equations and quadratic polynomials, the upper and lower parts of the lane line are fitted separately to obtain the left and right lane line equations.

[0014] Calculate relevant parameters based on the obtained left and right lane lines, including road curvature, lane centerline equation, lateral distance of the vehicle relative to the lane centerline, and yaw angle;

[0015] According to the parameters obtained above, the vehicle driving status is judged and the deviation warning strategy is executed.

[0016] Furthermore, the Res2Net50-VHA network is used to detect the feature points of the lane lines on both sides of the current lane, including the following:

[0017] Constructing a Res2Net50 network as a backbone feature extraction network for extracting features of the input image; the Res2Net50 network is composed of multiple Res2Net modules connected in series, and each Res2Net module constructs a hierarchical residual connection;

[0018] The VHA vertical and horizontal attention module is embedded in the last three output layers of the Res2Net50 network to form the Res2Net50-VHA network. The network downsamples and extracts the features of the lane line image, and finally concatenates them into a probability matrix of size h×(w+1)×n. The expected position of each lane line feature point in each row is solved.

[0019] The grid classification method is used for feature point detection. The lane line area in the image is divided into several grids. If the pixels of the lane line in the grid are greater than a given threshold, the center point of the grid is used as the coordinate of the lane line feature point.

[0020] Furthermore, each Res2Net module constructs a hierarchical residual connection, specifically:

[0021] After the first 1×1 convolution, the feature map is divided into s subsets according to the channel, and the subset is defined as x i , i∈{1,2,...,s}, calculate the output y as follows i :

[0022]

[0023] The feature scale of each subset is the same, but the channel is 1 / s of the input feature. Except for x1, all other sub-features are calculated by 3×3 convolution, defined as K i (x), sub-feature x i With Ki-1 (x) is added and input into K i That is, the n-channel convolution kernel (n = s × w) is replaced by s w-channel convolution kernels, and the Res2Net module is embedded in ResNet50, resulting in Res2Net50, which is used as the backbone feature extraction network for extracting lane line features in night scenes.

[0024] Furthermore, the VHA vertical and horizontal attention module is used to enhance the feature extraction capability. The output feature map of the nth layer of the network is Y n ∈R h×w×c , generate attention feature map A through VHA n ∈R h×w×c :

[0025] The specific calculation is: n Perform maximum pooling and average pooling operations on the vertical and horizontal directions respectively to obtain H n ∈R h ×1×c and W n ∈R 1×w×c :

[0026]

[0027]

[0028] Among them, K h =[1,w],K w =[h,1], representing the pooling window size.

[0029] Then, H n With W n It is sent to a shared module consisting of two convolutional layers, and the horizontal and vertical features are generated by the shared module and the sigmoid activation function respectively to generate H′ n With W n ′, where a 1×1 convolutional layer and a dimensionality reduction factor r are added to reduce the number of channels:

[0030] H′ n =f(Θ(H n ))

[0031] W n ′=f(Θ9W n ))

[0032] Among them, f is the sigmoid activation function, Θ is the shared module;

[0033] Finally, the feature map (H′ n With W n ′) and Y n Multiply to get the attention feature map An ;

[0034] Alpha n =Y n ×H′ n ×W n ′.

[0035] Furthermore, the image is divided into a grid of 48 rows × 300 columns.

[0036] Furthermore, the feature point supplementation and tracking are specifically as follows:

[0037] For the obtained lane feature points, determine whether the feature points are detected within the 48 rows of the grid. If not, linear interpolation is performed using the detected feature points of the lane line.

[0038] After adding feature points, Kalman filtering is applied to track the feature points of the two lane lines. It is divided into two parts: prediction and update. In the prediction stage, the system uses the optimal state of the previous frame image t-1 to The optimal state for the next frame image t Make a prediction and use the coordinates of the feature points detected in each frame to update the filter. The optimal position of the feature point at time t is calculated through iteration, which is used to obtain the input value of the optimal feature point in the next frame.

[0039] Furthermore, the combined fitting method of the straight line equation and the quadratic polynomial is applied to fit the upper and lower parts of the lane line respectively, and the equations of the left and right lane lines are obtained, specifically:

[0040] Use a quadratic curve to fit the 30 lane feature points located above the grid, and use a straight line equation to fit the 18 feature points at the bottom of the grid. Output the polar radius and polar angle of the lane equation, as well as the coordinates of the lower endpoint of each lane line. Based on this, calculate the yaw angle β and the lateral distance D between the left lane line and the vehicle's centerline. L and the lateral distance D between the right lane line and the center axis of the vehicle R , by combining the left and right lane line equations, we can get the equation of the center line of this lane.

[0041] Furthermore, the deviation warning strategy is specifically as follows:

[0042] When the vehicle's turn signal is on, lane departure warning is not performed by default. When the turn signal is not on, the system will issue a warning for crossing lanes or leaving lanes during driving. Specifically:

[0043] Using lateral distance as the primary warning parameter, the system compares the lateral distance the vehicle's centerline deviates from the lane centerline with the safety threshold T, and issues warnings for the following situations:

[0044] When D L ≥T, the system gives a left deviation warning;

[0045] When D R ≥T, the system issues a right deviation warning;

[0046] When D L 、D R ≤T, enter the auxiliary warning process;

[0047] The vehicle's yaw angle β is used as the system's auxiliary warning parameter, and the yaw angle is compared with the safety threshold β i , give warnings for the following situations:

[0048] When more than two consecutive frames of image β≥β are detected i , the system gives a left deviation warning;

[0049] When more than two consecutive frames of image β≤-β are detected i , the system gives a right deviation warning;

[0050] When-β i ≤β≤β i , it is believed that the vehicle is in a safe driving state at this time and no lane deviation occurs.

[0051] Furthermore, if the vehicle changes lanes without turning on the turn signal and fails to adjust to a safe state in the lane changing lane, the system will still determine that the vehicle is in a deviation state, thereby triggering an early warning. The system will not turn off the early warning information until the vehicle adjusts to a safe state.

[0052] A system for implementing the lane departure warning method in nighttime scenarios,

[0053] Lane feature extraction module: used to extract the feature points of the lane lines on both sides of the front road image;

[0054] Feature point position judgment module: used to detect the lane feature point positions of lane lines on both sides;

[0055] Lane feature point tracking module: uses linear interpolation to supplement the feature points of the lane lines on both sides, and then applies Kalman filtering to track the feature points of the two lane lines;

[0056] Lane feature point fitting module: uses quadratic polynomials and straight line equations to fit lane feature points to obtain left and right lane lines;

[0057] Parameter calculation module: Based on the obtained left and right lane lines, the calculated parameters include lane curvature, road direction, lateral distance of the vehicle relative to the lane centerline, and yaw angle;

[0058] Lane departure judgment and warning module: judges the vehicle's driving status and implements lane departure warning strategy.

[0059] The beneficial effects of Ming:

[0060] 1. This invention uses deep learning and vertical and horizontal attention methods to accurately identify lane lines and infer the position of lane lines in night environments.

[0061] 2. The present invention uses a Kalman filter to track lane feature points, making lane detection more accurate;

[0062] 3. The present invention adopts a combination of primary warning and auxiliary warning, comprehensively utilizes the yaw angle and lateral distance of the vehicle during driving to formulate a warning strategy, and provides more comprehensive warning information.

[0063] 4. The algorithm used in the present invention has the advantages of high recognition accuracy, high computational efficiency, and strong anti-interference ability, and the program outputs intuitive visualization effects in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A flow chart of the method of the present invention;

[0065] Figure 2 The structure diagram of the Res2Net module of the present invention;

[0066] Figure 3 The network structure diagram of Res2Net50-VHA of the present invention;

[0067] Figure 4 Schematic diagram of the VHA module structure of the present invention;

[0068] Figure 5 Schematic diagram of the lane feature point detection principle of the present invention;

[0069] Figure 6 Schematic diagram of the present invention deviating from the lane centerline;

[0070] Figure 7 A schematic diagram of the yaw angle of a vehicle traveling according to the present invention;

[0071] Figure 8 A schematic diagram of the lane departure warning strategy of the present invention;

[0072] Figure 9 Schematic diagram of the interface of the lane departure warning system of the present invention. DETAILED DESCRIPTION

[0073] The present invention will be further described in detail below with reference to the embodiments and the accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0074] Example

[0075] The vehicle-mounted camera used in the present invention is installed in the front of the vehicle, flush with the vehicle's center axis. The overall process is as follows Figure 1As shown, the present invention is a lane departure warning method for intelligent connected vehicles in night scenes, comprising the following steps:

[0076] S1: The onboard camera collects images of the road ahead and, after preprocessing, uses the Res2Net50-VHA network to detect the feature points of the lane lines on both sides of the current lane;

[0077] S11. Construct Res2Net50 as the backbone feature extraction network, which is composed of Res2Net modules connected in series to extract the features of the input image. Figure 2 As shown in the figure, in a single residual block, the Res2Net module constructs hierarchical residual connections, which enables the network to obtain details and global features at a finer granularity level, which is beneficial for lane line feature extraction and position estimation in lane line occlusion, missing and nighttime environments, and improves the network's feature expression ability. Specifically, after the first 1×1 convolution, the input feature map is divided into s subsets by channel, defined as x i , i∈{1,2,...,s}. Each feature has the same scale, but the channel is 1 / s of the input feature. Except for x1, other sub-features have corresponding 3×3 convolution kernels, defined as K i (x), whose output is y i . Sub-feature x i With K i-1 (x) is added and input into K i (x).

[0078] Calculate the output y according to formula (1) i .

[0079]

[0080] Res2Net50 is used to extract the features of the input image. The network structure is as follows Figure 3 As shown in the figure, assuming that the input feature map resolution is 288×800, it is first downsampled to 1 / 2 to obtain a feature map of 144×400. It then goes through four stages, and the feature maps extracted in the four different stages correspond to 1 / 4, 1 / 8, 1 / 16 and 1 / 32 resolutions of the input feature map.

[0081] The feature scale of each subset is the same, but the channel is 1 / s of the input feature. Except for x1, the other sub-features have corresponding 3×3 convolution kernels, defined as K i (x), sub-feature x i With K i-1 (x) is added and input into K i (x). That is, the n-channel convolution kernel (n = s × w) is replaced by s w-channel convolution kernels.

[0082] The Res2Net module is embedded into ResNet50 to obtain Res2Net50, which is used as the backbone feature extraction network to extract lane line features in night scenes.

[0083] S12, embed the Vertical and Horizontal Attention (VHA) module into the last three output levels of Res2Net50 to obtain Res2Net50-VHA, as shown in Figure 3 As shown in the figure, the network downsamples and extracts the features of the lane line image, and finally splices it into a probability matrix of size h×(w+1)×n to solve the expected position of each lane line feature point in each row.

[0084] VHA integrates lane marking texture and position information from vertical and horizontal pixels without requiring extensive computation. Furthermore, by embedding it into the appropriate location of Res2Net50, it improves lane marking detection accuracy in nighttime environments.

[0085] Further: the VHA vertical and horizontal attention module, its details include: the output feature map of the nth layer of the network is Y n ∈R h ×w×c , the attention feature map A can be generated through VHA n ∈R h×w×c× , A n With Y n Have the same number of channels. The specific calculation process is: n Perform maximum pooling and average pooling operations on the vertical and horizontal directions respectively to obtain H n ∈R h×1×c and W n ∈R 1×w×c As shown in formula (2) and formula (3), where K h =[1,w],K w = [h, 1], representing the pooling window size. Then, H n With W n It is sent to a shared module consisting of two convolutional layers, as shown in Equations (4) and (5), where f is the sigmoid activation function and Θ is the shared module. The horizontal and vertical features are generated by the shared module and the sigmoid activation function respectively to generate H′ n and W′ n , where a 1×1 convolution layer and a dimensionality reduction factor r are added to reduce the number of channels. Finally, the feature map (H′ n and W′ n ) and Y n Multiply to get the attention feature map A n .

[0086]

[0087]

[0088] H′ n =f(Θ(H n )) (4)

[0089] W′ n =f(Θ(W n )) (5)

[0090] Alpha n =Y n ×H′ n ×W′ n (6)

[0091] The lane line detection principle is as follows Figure 5 As shown. Divide the fixed area of the image into hw grids, classify each grid on each row, and find the lane feature points. Figure 5 Take the row in the dotted box on the left as an example. We traverse n times to find the positions of the n lane feature points in the row. The dark part in the middle on the right represents the position of the lane feature points in the row (black block). Since there are n lane lines, the probability matrix is n-dimensional.

[0092] Further: The grid classification method used in this embodiment is specifically: lane line detection is regarded as a process of row-by-row classification and grid-by-grid selection using global image features. The search process is: divide the fixed area of the image into h×w grids, perform target classification on each grid on each row, and find the position of the lane line feature points. The present invention only focuses on the lane lines on both sides of the lane. For each row of grids, it traverses twice to find the position of the two lane line feature points in that row. In the actual process, the lane line area in the image is divided into a grid of 48 rows × 300 columns, and the coordinates of the grid center point are used as the coordinates of the feature point position.

[0093] The grid classification method used in this invention divides the lane line area in the image into several grids. By classifying the grids, the positions of the lane line feature points are determined. Compared with semantic segmentation, this method can greatly reduce the amount of calculation and improve the detection speed.

[0094] S13, filter the night scene images in the open source dataset CULane, and build a lane line dataset under night scenes as training data for the network, so that it can learn the lane line features under night environment. During the network training phase, activate the auxiliary training branch. Figure 3 As shown in the dotted box in , Res2Net outputs feature maps of four levels with different sizes. The last three layers of feature maps are taken out and fused after upsampling to calculate the semantic loss of the lane line. This branch is not activated during prediction.

[0095] S2. After adding the feature points, apply Kalman filtering to track the feature points of the two lane lines, specifically:

[0096] For the obtained lane feature points, determine whether the feature points are detected within the 48 rows of the grid. If not, linear interpolation is performed using the detected feature points of the lane line.

[0097] After adding feature points, Kalman filtering is applied to track the feature points of the two lane lines. It is divided into two parts: prediction and update. In the prediction stage, the system uses the optimal state of the previous frame image t-1 to The optimal state for the next frame image t Make a prediction and use the coordinates of the feature points detected in each frame to update the filter. The optimal position of the feature point at time t is calculated through iteration, which is used to obtain the input value of the optimal feature point in the next frame.

[0098] The calculation process is shown in equations (7) to (9).

[0099]

[0100]

[0101] z t =Hx t +v() (9)

[0102] Where F is the state transfer matrix, which represents the current state inferred from the previous state; B is the control matrix, which represents the way the control variable u acts on the current state; t represents the system noise, which obeys the Gaussian distribution. Equation (8) represents the transfer of the noise covariance matrix, where represents the covariance corresponding to time t; P t-1 represents the covariance corresponding to time t-1; Q is the covariance of the system process noise. Formula (9) is the observation equation, z t is the observation value, H is the observation matrix, x t is the current state of the system, v is the measurement noise, and it obeys Gaussian distribution.

[0103] In the update phase, the Kalman coefficient K is calculated first. t , to determine the weights of the prediction model and the observation model. Calculate the optimal update value through formula (11) and formula (12) P t .

[0104]

[0105]

[0106]

[0107] Where R is the covariance matrix of the measurement noise. In formula (11), by calculating the actual observation value z t Expected observed value The residual between them is multiplied by the Kalman coefficient to correct the predicted value The coordinates of the feature points detected in each frame are used to update the filter, and the optimal position of the feature point at time t is calculated iteratively, which is used to obtain the input value of the optimal feature point for the next frame.

[0108] S3. Fit lane feature points. Because lane lines near the bottom of the image have less distortion and curvature, a linear fitting model is used to fit the 18 rows of feature points at the bottom of the grid. The left and right lane line calculation formulas are shown in Equations (13) and (14).

[0109] Left lane line: ρ1=x cosθ1+y sinθ1 (13)

[0110] Right lane line: ρ2=x cosθ2+y sin θ2 (14)

[0111] The lane lines located 30 rows above the grid have obvious perspective effects, so a quadratic curve is used to fit the curved lanes. The approximate equation of the lane lines is obtained using the least squares method, as shown in Equation (15). The coordinates of each feature point on the lane line are (x i ,y i ), calculate the deviations from all feature points to the target curve, and calculate the sum of squares of the deviations, as shown in formula (16). In order to make the target curve close to the actual road curve, the sum of squares of the deviations is required to be as small as possible. When the value reaches the minimum, the target curve is the curve closest to the actual lane line.

[0112] y=a0+a1x+a2x 2 (15)

[0113]

[0114] S4. Calculate relevant parameters, including lane curvature, road direction, lateral distance of the vehicle relative to the lane centerline, and yaw angle.

[0115] Road Curvature Radius: Combining the left and right lane equations, we can obtain the equation for the lane centerline, as shown in Equation (17). The inverse of the curvature at a point on a curve is called the curvature radius of the curve at that point. The curvature radius is calculated as shown in Equation (18). Substituting the lane centerline equation into the curvature radius calculation formula yields Equation (19).

[0116]

[0117]

[0118]

[0119] Yaw angle: The yaw angle is the angle between the longitudinal center axis of the vehicle and the bisector of the angle between the two lane lines, which can reflect the directional deviation of the vehicle during driving. Based on the lane line feature points detected by the above algorithm, for the feature points of the 18 rows of grids below, a straight line equation is used to fit the polar radius and polar angle of the lane line equation, as well as the coordinates of the lower endpoint of each lane line. In this way, the yaw angle β and the lateral distance D between the left lane line and the center axis of the vehicle can be calculated. L and the lateral distance D between the right lane line and the center axis of the vehicle R The equation for the bisector of the angle between the left and right lane lines is shown in equation (20), and the yaw angle is shown in equation (21).

[0120]

[0121] in,

[0122]

[0123] S5. Determine the vehicle's driving state based on the lateral distance between the vehicle and the lane centerline and the vehicle's yaw angle, specifically:

[0124] This invention combines primary and secondary warnings, utilizing the vehicle's yaw angle and lateral distance to formulate a warning strategy. When the vehicle's turn signal is on, lane departure warning is disabled by default. When the turn signal is off, the system issues warnings for lane crossing or lane departure, ensuring driver safety.

[0125] First, lateral distance is used as the primary warning parameter, and pixel distance is used to represent the actual distance. The lateral distance of the vehicle's centerline from the lane centerline is compared with the safety threshold T, and warnings are issued for the following situations:

[0126] When D L ≥T, the system gives a left deviation warning;

[0127] When D R ≥T, the system issues a right deviation warning;

[0128] When D L 、D R ≤T, enter the auxiliary warning process.

[0129] The vehicle's yaw angle β is used as the system's auxiliary warning parameter, and the yaw angle is compared with the safety threshold β i , give warnings for the following situations:

[0130] ●When more than two consecutive frames of images β≥β are detected i , the system gives a left deviation warning;

[0131] ●When more than two consecutive frames of images are detected, β≤-β i , the system gives a right deviation warning;

[0132] ●When-β i ≤β≤β i , it is believed that the vehicle is in a safe driving state at this time and no lane deviation occurs.

[0133] If the vehicle changes lanes without turning on the turn signal and fails to adjust to a safe state within the lane, the system will still determine that the vehicle is in a deviation state, thereby triggering an early warning. The system will not turn off the early warning information until the vehicle adjusts to a safe state.

[0134] Vehicle lateral distance and yaw angle calculation as follows Figure 6 and Figure 7 As shown, deviation from the decision-making process is as follows Figure 8 shown.

[0135] S6. The vehicle, detected lane lines, and road conditions are displayed in real time in the program, and a voice reminder is given when the vehicle is about to deviate. Specifically:

[0136] The present invention is based on PyQt and integrates the above method into the "Lane Departure Warning System", which mainly includes 5 functional areas, namely the video display area, the status prompt area, the parameter selection area, the parameter display area and the system control area. Among them, the video display area displays the image captured by the camera during driving and the processed lane line image in real time; the status prompt area uses an intuitive schematic diagram to represent the driving status of the vehicle; the parameter display area is used to select and set the current parameters; the parameter display area is used to display the system analysis results; the system control area is used to control the system, mainly including interactive buttons such as system startup and shutdown, camera calibration, video testing, simple mode and data saving. The system interface is as follows Figure 9 shown.

[0137] This embodiment also provides a system for implementing the method, including:

[0138] Lane feature extraction module: used to extract the feature points of the lane lines on both sides of the front road image; using a vehicle-mounted camera, the camera is installed in front of the motor vehicle, and there is at least one camera.

[0139] Feature point position judgment module: used to determine the lane feature point positions of lane lines on both sides;

[0140] Lane feature point tracking module: uses linear interpolation to supplement the feature points of the lane lines on both sides, and then applies Kalman filtering to track the feature points of the two lane lines;

[0141] Lane feature point fitting module: uses quadratic polynomials and straight line equations to fit lane feature points to obtain left and right lane lines;

[0142] Parameter calculation module: Based on the obtained left and right lane lines, the calculated parameters include lane curvature, road direction, lateral distance of the vehicle relative to the lane centerline, and yaw angle;

[0143] Lane departure judgment and warning module: judges the vehicle's driving status and implements lane departure warning strategy.

[0144] This invention solves the safety issues of intelligent connected vehicles driving on roads at night. It takes into account the characteristics of lane lines in night scenes and applies Res2Net50-VHA to effectively improve the accuracy of lane line detection. At the same time, grid classification is used instead of semantic segmentation to improve the detection speed of lane line feature points. A Kalman filter is used to track lane line feature points to ensure the stability of lane line detection. A comprehensive warning strategy based on vehicle yaw angle and lateral distance is adopted to provide lane departure warning information and reduce the risk of lane deviation. Finally, the program outputs intuitive visualization effects in real time.

[0145] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A lane departure warning method for vehicles in nighttime scenes, characterized in that: include: Collect the road ahead image, and after preprocessing, use the Res2Net50-VHA network to detect the feature points of the lane lines on both sides of the current lane; Linear interpolation is used to supplement the feature points of the lane lines on both sides, and Kalman filtering is used to track the lane feature points of the front and back frame images; Using a combination of straight line equations and quadratic polynomials, the upper and lower parts of the lane line are fitted separately to obtain the left and right lane line equations. Calculating relevant parameters based on the obtained left and right lane lines, the relevant parameters including road curvature, lane centerline equation, lateral distance of the vehicle relative to the lane centerline, and driving yaw angle; According to the above parameters, the vehicle driving status is judged and the deviation warning strategy is implemented; The Res2Net50-VHA network is used to detect the feature points of the lane lines on both sides of the current lane, including the following: Constructing a Res2Net50 network as a backbone feature extraction network for extracting features of the input image; the Res2Net50 network is composed of multiple Res2Net modules connected in series, and each Res2Net module constructs a hierarchical residual connection; The VHA vertical and horizontal attention module is embedded in the last three output layers of the Res2Net50 network to form the Res2Net50-VHA network. The network downsamples and extracts the features of the lane line image, and finally concatenates them into a probability matrix of size h×(w+1)×n. The expected position of each lane line feature point in each row is solved. The grid classification method is used for feature point detection. The lane line area in the image is divided into several grids. If the pixels of the lane line in the grid are greater than a given threshold, the center point of the grid is used as the coordinate of the lane line feature point.

2. The lane departure warning method for vehicles in nighttime scenarios according to claim 1, characterized in that: Each Res2Net module constructs a hierarchical residual connection, specifically: After the first 1×1 convolution, the feature map is divided into s subsets according to the channel, and the subset is defined as x i , i∈{1,2,...,s}, calculate the output y as follows i : The feature scale of each subset is the same, but the channel is 1 / s of the input feature. Except for x1, all other sub-features are calculated by 3×3 convolution, defined as K i (x), sub-feature x i With K i-1 (x) is added and input into K i (x); that is, the n-channel convolution kernel n = s × w is replaced by s w-channel convolution kernels, and the Res2Net module is embedded in ResNet50 to obtain Res2Net50, which is used as the backbone feature extraction network to extract lane line features in night scenes.

3. The vehicle lane departure warning method in nighttime scenarios according to claim 1, characterized in that: The VHA vertical and horizontal attention module is used to enhance the feature extraction capability. The output feature map of the nth layer of the network is Y n ∈R h×w×c , generate attention feature map A through VHA n ∈R h×w×c : The specific calculation is: n Perform maximum pooling and average pooling operations on the vertical and horizontal directions respectively to obtain H n ∈R h×1×c and W n ∈R 1×w×c : Among them, K h =[1,w],K w =[h,1], represents the pooling window size; Then, H n With W n It is sent to a shared module consisting of two convolutional layers, and the horizontal and vertical features are generated by the shared module and the sigmoid activation function respectively to generate H′ n With W n ′, where a 1×1 convolutional layer and a dimensionality reduction factor r are added to reduce the number of channels: H′ n =f(Θ(H n )) W′ n =f(Θ(W n )) Among them, f is the sigmoid activation function, Θ is the shared module; Finally, the feature map is n Multiply to obtain the attention feature map A n ; TO n =And n ×H′ n ×W′ n 。 4. The method for warning lane departure of a vehicle at night according to claim 1, characterized in that: Divide the image into a grid of 48 rows by 300 columns.

5. The vehicle lane departure warning method in nighttime scenarios according to claim 4, characterized in that: The feature point supplementation and tracking are specifically as follows: For the obtained lane feature points, determine whether the feature points are detected within the 48 rows of the grid. If not, use the detected feature points of the lane line for linear interpolation. After adding feature points, Kalman filtering is applied to track the feature points of the two lane lines. It is divided into two parts: prediction and update. In the prediction stage, the system uses the optimal state of the previous frame image t-1 to The optimal state for the next frame image t Make a prediction and use the coordinates of the feature points detected in each frame to update the filter. The optimal position of the feature point at time t is calculated through iteration, which is used to obtain the input value of the optimal feature point in the next frame.

6. The vehicle lane departure warning method in nighttime scenarios according to claim 1, characterized in that: Applying the combined fitting method of the straight line equation and the quadratic polynomial, the upper and lower parts of the lane line are fitted separately to obtain the left and right lane line equations, specifically: Use a quadratic curve to fit the 30 lane feature points located above the grid, and use a straight line equation to fit the 18 feature points at the bottom of the grid. Output the polar radius and polar angle of the lane equation, as well as the coordinates of the lower endpoint of each lane line. Based on this, calculate the yaw angle β and the lateral distance D between the left lane line and the vehicle's centerline. L and the lateral distance D between the right lane line and the center axis of the vehicle R , by combining the left and right lane line equations, we can get the equation of the center line of this lane.

7. The vehicle lane departure warning method in nighttime scenarios according to claim 1, characterized in that: The deviation warning strategy is specifically as follows: When the vehicle's turn signal is on, lane departure warning is not performed by default. When the turn signal is not on, the system will issue a warning for crossing lanes or leaving lanes during driving. Specifically: Using lateral distance as the primary warning parameter, the system compares the lateral distance the vehicle's centerline deviates from the lane centerline with the safety threshold T, and issues warnings for the following situations: When D L ≥T, the system issues a left deviation warning; When D R ≥T, the system issues a right deviation warning; When D L 、D R ≤T, enter the auxiliary warning process; The vehicle's yaw angle β is used as the system's auxiliary warning parameter, and the yaw angle is compared with the safety threshold β i , give warnings for the following situations: When more than two consecutive frames of image β≥β are detected i , the system gives a left deviation warning; When more than two consecutive frames of image β≤-β are detected i , the system gives a right deviation warning; When-β i ≤β≤β i , it is believed that the vehicle is in a safe driving state at this time and no lane deviation occurs.

8. The vehicle lane departure warning method in nighttime scenarios according to claim 7, characterized in that: If the vehicle changes lanes without turning on the turn signal and fails to adjust to a safe state within the lane, the system will still determine that the vehicle is in a deviation state, thereby triggering an early warning. The system will not turn off the early warning information until the vehicle adjusts to a safe state.

9. A system for implementing the vehicle lane departure warning method in nighttime scenarios according to any one of claims 1 to 8, characterized in that: Lane feature extraction module: used to extract the feature points of the lane lines on both sides of the front road image; Feature point position judgment module: used to detect the lane feature point positions of lane lines on both sides; Lane feature point tracking module: uses linear interpolation to supplement the feature points of the lane lines on both sides, and then applies Kalman filtering to track the feature points of the two lane lines; Lane feature point fitting module: uses quadratic polynomials and straight line equations to fit lane feature points to obtain left and right lane lines; Parameter calculation module: Based on the obtained left and right lane lines, the calculated parameters include lane curvature, road direction, lateral distance of the vehicle relative to the lane centerline, and yaw angle; Lane departure judgment and warning module: judges the vehicle's driving status and implements lane departure warning strategy.

Images