A lane line fusion recognition method based on semantic segmentation and edge detection algorithm

Abstract

This invention provides a lane line fusion recognition method based on semantic segmentation and edge detection algorithms. The method comprises: obtaining a road image containing lane lines and preprocessing the road image to obtain an input image; then, performing lane line edge detection on the input image using an edge detection algorithm to output a first lane line detection result; then, establishing a semantic segmentation model, performing knowledge distillation and quantization-aware training based on the semantic segmentation model to obtain a fully trained lightweight semantic segmentation model; then, performing lane line detection on the input image based on the lightweight semantic segmentation model to output a second lane line detection result; finally, fusing the first and second lane line detection results using a sliding window algorithm to output a target lane line detection result. This invention can address the low detection accuracy and poor stability issues of existing technologies.

Description

Technical Field

[0001] The present invention relates to the technical field of lane line detection, and in particular to a lane line fusion recognition method based on semantic segmentation and edge detection algorithms. Background Art

[0002] Autonomous driving has attracted significant attention from researchers in academia and industry. To ensure vehicle safety, autonomous driving systems must keep the vehicle on the road and along lane lines, which requires accurate lane perception. Lane detection plays a crucial role in autonomous driving systems, particularly in Advanced Driver Assistance Systems (ADAS). Lane detection is typically defined as detecting the location of lane lines in videos or images of a moving vehicle.

[0003] Existing lane detection technologies primarily fall into two categories: traditional image-based methods and deep learning-based methods. Traditional image-based methods segment lane areas through methods such as edge detection and filtering. Mainstream approaches include: lane detection based on the Hough transform; lane detection based on LSD lines; lane detection based on top-view transformations; lane detection based on fitting; and lane detection based on parallel perspective vanishing points. Traditional image-based lane detection methods generally suffer from issues such as unstable detection, susceptibility to external interference, and limited application scenarios. Deep learning methods have rapidly gained attention due to their robustness and real-time performance. They are broadly categorized into four types: segmentation-based methods, detection-based methods, parametric curve-based methods, and keypoint-based methods. The most common of these is the semantic segmentation-based lane detection method, which models lane detection as a pixel-by-pixel classification problem, classifying each pixel as a lane area or background. However, deep learning-based methods based on semantic segmentation also suffer from poor performance and low accuracy in the presence of severe image occlusion.

[0004] Therefore, in order to solve the problems of lane line detection methods using traditional image methods, such as unstable detection, susceptibility to external interference, and limited application scenarios, and the problem of low accuracy when deep learning methods are used for lane line recognition when the picture is severely occluded, the present invention provides a lane line fusion recognition method based on semantic segmentation and edge detection algorithm, which is used to solve the problems of low detection accuracy and poor stability in the existing technology. Summary of the Invention

[0005] In view of this, the present invention provides a lane line fusion recognition method based on semantic segmentation and edge detection algorithm, which is used to solve the problems of low detection accuracy and poor stability in the existing technology due to external factors such as occlusion and shadow in the recognition image.

[0006] To solve the above problems, the present invention proposes a lane line fusion recognition method based on semantic segmentation and edge detection algorithm, comprising:

[0007] In some possible implementations, a road image including lane lines is obtained, and the road image is preprocessed to obtain an input image;

[0008] Performing lane line edge detection on the input image based on an edge detection algorithm to output a first lane line detection result;

[0009] Establishing a semantic segmentation model, performing knowledge distillation and quantization perception training based on the semantic segmentation model to obtain a fully trained lightweight semantic segmentation model, and performing lane line detection on the input image based on the fully trained lightweight semantic segmentation model to output a second lane line detection result;

[0010] Based on the sliding window algorithm, the first lane line detection result and the second lane line detection result are fused and identified to output the target lane line detection result.

[0011] In some possible implementations, the fully trained lightweight semantic segmentation model is trained based on the Deeplabv3p-ResNet18 model.

[0012] In some possible implementations, the fully trained lightweight semantic segmentation model includes:

[0013] Encoding module and decoding module;

[0014] The encoding module includes the backbone network DCNN and the cascaded atrous spatial pyramid pooling module ASPP;

[0015] The atrous spatial pyramid pooling module ASPP consists of a cascaded atrous convolution module and an average pooling layer;

[0016] The dilated convolution module consists of three dilated convolution layers with dilated convolution rates arranged from small to large.

[0017] The decoding module consists of two convolutional layers and two four-fold bilinear interpolation upsampling modules.

[0018] In some possible implementations, performing lane line detection on the input image based on the fully trained lightweight semantic segmentation model to output a second lane line detection result includes:

[0019] Perform feature extraction on the input image based on the backbone network DCNN to obtain a first feature map;

[0020] Based on the dilated convolution module in the dilated spatial pyramid pooling module ASPP, the first feature map is subjected to dilated convolution processing, and then the average pooling layer is used to perform pooling processing to obtain a second feature map;

[0021] Based on the decoding module, the second feature map is up-sampled by four times bilinear interpolation to obtain a second lane line detection result.

[0022] In some possible implementations, obtaining a road image containing lane lines and preprocessing the road image to obtain an input image includes:

[0023] Get a road image containing lane lines;

[0024] The input image is obtained by performing resolution adjustment, image correction, and image enhancement on the road image containing lane lines.

[0025] In some possible implementations, performing lane edge detection on the input image based on an edge detection algorithm to output a first lane line detection result includes:

[0026] Converting the input image into a grayscale image, and performing image masking on the grayscale image to obtain a masked image;

[0027] Performing Gaussian filtering on the masked image to obtain a preliminary lane line image;

[0028] Lane line detection is performed on the preliminary lane line image based on the edge detection algorithm to obtain the first lane line detection result.

[0029] In some possible implementation methods, a semantic segmentation model is established, and knowledge distillation and quantization-aware training are performed based on the semantic segmentation model to obtain a fully trained lightweight semantic segmentation model, including:

[0030] Establish a semantic segmentation model and perform knowledge distillation training based on the semantic segmentation model to obtain a preliminary optimized semantic segmentation model;

[0031] Based on the preliminary optimized semantic segmentation model, quantization-aware training is performed to obtain a fully trained lightweight semantic segmentation model.

[0032] In some possible implementations, a semantic segmentation model is established, and knowledge distillation training is performed based on the semantic segmentation model to obtain a preliminary optimized semantic segmentation model, including:

[0033] Obtain road images containing lane lines as training sets;

[0034] Get a fully trained teacher model;

[0035] Performing lane recognition on a road image containing lane lines based on the fully trained teacher model to obtain a teacher lane recognition result;

[0036] The road image containing lane lines is identified based on the semantic segmentation model to obtain the student lane line recognition result;

[0037] The conventional loss is obtained by calculating the student lane line recognition results and the true recognition results based on the cross entropy loss function;

[0038] Based on the relative entropy loss, the student lane recognition results and the teacher lane recognition results are calculated to obtain the distillation loss;

[0039] The semantic segmentation model is iteratively trained based on distillation loss and conventional loss as loss parameters to obtain a preliminary optimized semantic segmentation model.

[0040] Some possible implementations of performing quantization-aware training based on a preliminary optimized semantic segmentation model to obtain a fully trained lightweight semantic segmentation model include:

[0041] Insert a pseudo-quantization operator before the convolution operator of the preliminary optimized semantic segmentation model to simulate the simulation error caused by the quantization of the preliminary optimized semantic segmentation model;

[0042] The model parameters are optimized according to the simulation error to obtain a fully trained lightweight semantic segmentation model.

[0043] In some possible implementations, fusing the first lane line detection result and the second lane line detection result based on a sliding window algorithm to output a target lane line detection result includes:

[0044] Calculating, based on a sliding window algorithm, each window feature value of the first lane line detection result and each window feature value of the second lane line detection result for the first lane line detection result and the second lane line detection result;

[0045] Calculating the distance between the feature values ​​of each window of the first lane line detection result and the feature values ​​of each window of the second lane line detection result and the expected feature value based on the feature values ​​of each window of the first lane line detection result and the feature values ​​of each window of the second lane line detection result;

[0046] Based on the distances of the feature values ​​of each window of the first lane line detection result and the feature values ​​of each window of the second lane line detection result to the expected feature value, the windows with smaller distances to the expected feature value are selected and fused into one image, and the target lane line detection result is outputted through optimization.

[0047] Compared with the prior art, the beneficial effects of the present invention include: the present invention obtains an input image by acquiring a road image containing lane lines and preprocessing the road image; then, lane line edge detection is performed on the input image based on an edge detection algorithm to output a first lane line detection result; secondly, a semantic segmentation model is established, and knowledge distillation and quantization perception training are performed based on the semantic segmentation model to obtain a fully trained lightweight semantic segmentation model, and lane line detection is performed on the input image based on the lightweight semantic segmentation model to output a second lane line detection result; finally, the first lane line detection result and the second lane line detection result are fused and identified based on a sliding window algorithm to output a target lane line detection result. The present invention improves the adaptability and robustness of the algorithm by performing lane line recognition through the edge detection algorithm. Secondly, a well-trained lightweight semantic segmentation model is obtained through knowledge distillation and quantized perception training. Lane lines are detected based on the lightweight semantic segmentation model to ensure the recognition accuracy of the model and improve the running speed. Finally, a sliding window algorithm is used to fuse the first lane line detection result and the second lane line detection result to output the target lane line detection result. This helps to overcome the problem of unstable detection based on the edge detection algorithm, and improves the shortcomings of the semantic segmentation network that does not fully utilize the prior knowledge of the lane line, overcomes the limitations of the single lane line detection algorithm, and improves the accuracy and stability of lane line detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of an embodiment of a lane line fusion recognition method based on semantic segmentation and edge detection algorithm provided by the present invention;

[0049] Figure 2 The lightweight data diagram of the model provided by the present invention;

[0050] Figure 3 This is a schematic diagram of the pixel statistical histogram provided by the present invention. DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0052] It should be understood that the schematic drawings are not drawn to scale. The flowcharts used in the present invention illustrate operations implemented according to some embodiments of the present invention. It should be understood that the operations in the flowcharts may be implemented out of sequence, and steps that do not have a logical contextual relationship may be reversed or performed simultaneously. In addition, those skilled in the art, guided by the present disclosure, may add one or more additional operations to the flowcharts or remove one or more operations from the flowcharts.

[0053] Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor systems and / or microcontroller systems.

[0054] Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0055] Before describing the specific embodiments, it should be noted that occlusion includes but is not limited to dirt on the recognition lens, shadow occlusion, etc., and the presence of occlusion, shadow and other external factors in the recognition image may lead to low detection accuracy and poor stability.

[0056] An embodiment of the present invention provides a lane line fusion recognition method based on semantic segmentation and edge detection algorithm, which is described below.

[0057] Figure 1 This is a flowchart of an embodiment of a lane line fusion recognition method based on semantic segmentation and edge detection algorithm provided by the present invention, which includes the following steps:

[0058] S101, obtaining a road image including lane lines, and preprocessing the road image to obtain an input image;

[0059] S102, performing lane line edge detection on the input image based on an edge detection algorithm and outputting a first lane line detection result;

[0060] S103: Establish a semantic segmentation model, perform knowledge distillation and quantization perception training based on the semantic segmentation model to obtain a fully trained lightweight semantic segmentation model, and perform lane line detection on the input image based on the fully trained lightweight semantic segmentation model to output a second lane line detection result;

[0061] S104: Based on a sliding window algorithm, the first lane line detection result and the second lane line detection result are fused and identified to output a target lane line detection result.

[0062] Compared with the existing technology, the present invention obtains an input image by obtaining a road picture containing lane lines and preprocessing the road picture; then, lane line edge detection is performed on the input image based on the edge detection algorithm to output a first lane line detection result; secondly, a semantic segmentation model is established, and knowledge distillation and quantization perception training are performed based on the semantic segmentation model to obtain a fully trained lightweight semantic segmentation model, and lane line detection is performed on the input image based on the lightweight semantic segmentation model to output a second lane line detection result; finally, the first lane line detection result and the second lane line detection result are fused and identified based on the sliding window algorithm to output a target lane line detection result. The present invention improves the adaptability and robustness of the algorithm by performing lane line recognition through the edge detection algorithm. Secondly, a well-trained lightweight semantic segmentation model is obtained through knowledge distillation and quantized perception training. Lane lines are detected based on the lightweight semantic segmentation model to ensure the recognition accuracy of the model and improve the running speed. Finally, a sliding window algorithm is used to fuse the first lane line detection result and the second lane line detection result to output the target lane line detection result. This helps to overcome the problem of unstable detection based on the edge detection algorithm, and improves the shortcomings of the semantic segmentation network that does not fully utilize the prior knowledge of the lane line, overcomes the limitations of the single lane line detection algorithm, and improves the accuracy and stability of lane line detection.

[0063] It should be noted that the edge detection algorithm is based on the OpenCV framework and the Canny edge detection operator with adaptive threshold for first lane line detection.

[0064] OpenCV, short for Open Source Computer Vision Library, is a cross-platform, open-source computer vision processing library. Initiated, maintained, and participated in by Intel's Russian team, it supports numerous algorithms related to computer vision and machine learning. Released under a BSD license, it is free for commercial and research use. OpenCV can be used to develop real-time image processing, computer vision, and pattern recognition programs, and can also utilize Intel's IPP for accelerated processing.

[0065] In a preferred embodiment of the present invention, the lightweight semantic segmentation model is trained based on the Deeplabv3p-ResNet18 model.

[0066] It should be noted that Deeplabv3p-ResNet18 is an improvement based on the Deeplabv3+ model, and the ResNet18 network is used as the backbone network in Deeplabv3+.

[0067] Furthermore, it should be noted that Deeplabv3+ is one of a series of semantic segmentation algorithms proposed by the Google team.

[0068] In a preferred embodiment of the present invention, the fully trained lightweight semantic segmentation model includes:

[0069] Encoding module and decoding module;

[0070] The encoding module includes a backbone network (DCNN, Deeplabv Convolutional Neural Network) and a cascaded atrous spatial pyramid pooling module (ASPP, Atrous Spatial Pyramid Pooling);

[0071] The atrous spatial pyramid pooling module ASPP consists of a cascaded atrous convolution module and an average pooling layer;

[0072] The dilated convolution module consists of three dilated convolution layers with dilated convolution rates arranged from small to large.

[0073] The decoding module consists of two convolutional layers and two four-fold bilinear interpolation upsampling modules.

[0074] In a specific embodiment, the backbone network (DCNN) uses the ResNet18 network as the backbone network.

[0075] In a preferred embodiment of the present invention, performing lane line detection on the input image based on the fully trained lightweight semantic segmentation model to output a second lane line detection result includes:

[0076] Perform feature extraction on the input image based on the backbone network DCNN to obtain a first feature map;

[0077] Based on the dilated convolution module in the dilated spatial pyramid pooling module ASPP, the first feature map is subjected to dilated convolution processing, and then the average pooling layer is used to perform pooling processing to obtain a second feature map;

[0078] Based on the decoding module, the second feature map is up-sampled by four times bilinear interpolation to obtain a second lane line detection result.

[0079] It should be noted that ASPP (Atrous Spatial Pyramid Pooling) is a dilated spatial convolutional pooling pyramid.

[0080] In a preferred embodiment of the present invention, a road image including lane lines is obtained and preprocessed to obtain an input image, including:

[0081] Get a road image containing lane lines;

[0082] The road image containing lane lines is subjected to resolution adjustment, image correction, and image enhancement to obtain an input image.

[0083] In a preferred embodiment of the present invention, performing lane line edge detection on an input image based on an edge detection algorithm to output a first lane line detection result includes:

[0084] Convert the input image into a grayscale image, and perform image mask processing on the grayscale image to obtain a masked image;

[0085] Perform Gaussian filtering on the masked image to obtain a preliminary lane line image;

[0086] Lane line detection is performed on the preliminary lane line image based on the edge detection algorithm to obtain the first lane line detection result.

[0087] In a specific embodiment, the input image is converted into a grayscale image, and the grayscale image is subjected to image mask processing to obtain a masked image;

[0088] Perform Gaussian filtering on the masked image to obtain a preliminary lane line image;

[0089] Among them, the input image is filtered using a Gaussian first-order derivative filter, that is, the preliminary lane line image f(x,y) is filtered using a Gaussian function to obtain a smoothed image data matrix, where (x,y) are the image pixel coordinates respectively;

[0090] The preliminary lane line image is convolved with a Gaussian filter function to obtain a smoothed image. The specific formula is:

[0091] s(x,y)=f(x,y)*G(x,y,σ)

[0092] In the formula, s(x,y) is the smoothed image and G(x,y,σ) is the Gaussian filter function.

[0093] The Gaussian filter function is:

[0094] G(x,y,σ) is a two-dimensional Gaussian function, which is the first-order directional reciprocal of G(x,y) in a certain direction n. Then: in

[0095] Where n is the direction vector, is the gradient vector, which is the gradient vector of the original image f(x,y) and G n Perform convolution and change the direction of n at the same time, then G n *n when f(x,y) reaches its maximum value and the direction of the detected edge. The value range of σ is 0.3-3.

[0096] Furthermore, lane line detection is performed on the preliminary lane line image based on an edge detection algorithm to obtain a first lane line detection result, including extracting lane lines from the road image using a Canny edge detection algorithm based on the OpenCV framework and an adaptive threshold, and obtaining a binary image output of the lane lines and the background, namely the first lane line detection result.

[0097] Performing lane line detection on the preliminary lane line image based on an edge detection algorithm to obtain a first lane line detection result, specifically comprising: calculating a gradient magnitude and a gradient direction of each pixel point in the preliminary lane line image based on the edge detection algorithm to determine a gradient value and a gradient direction of each pixel point in the preliminary lane line image;

[0098] Based on the edge detection algorithm, the gradient value of each pixel in the preliminary lane line image is non-maximum suppressed and the image after non-maximum suppression is output;

[0099] Based on the edge detection algorithm, double threshold detection is performed on the image after non-maximum suppression to obtain the image edge data, that is, the first lane line detection result.

[0100] The step of calculating the gradient magnitude and gradient direction of each pixel in the input image based on the edge detection algorithm to determine the gradient value and gradient direction of each pixel in the input image includes:

[0101] The gradient value and gradient direction of each pixel in the input image are calculated based on the gradient value calculation formula and the gradient direction angle calculation formula.

[0102] The gradient magnitude M and direction angle A of each pixel are calculated using the following formula:

[0103] M=‖f(x,y)*G(x,y,σ)‖

[0104]

[0105] Based on the edge detection algorithm, the gradient value of each pixel in the preliminary lane line image is non-maximum suppressed and the output image after non-maximum suppression is output, including:

[0106] Compare the gradient strength of the current pixel with the gradient strength of the two pixels along the positive and negative gradient directions. If the current pixel is found to be the largest, then the current pixel can be saved and used as an edge point, that is, the point where M reaches a local maximum at the direction angle A. If this is not the case, then the current pixel is suppressed, and finally the non-maximum suppressed image is output based on the edge points.

[0107] Based on the edge detection algorithm, double threshold detection is performed on the image after non-maximum suppression to obtain image edge data, namely the first lane line detection result, specifically including: obtaining the adaptive threshold of the Canny edge detection algorithm.

[0108] The specific process is as follows: the segmentation threshold of the foreground and background of the image after non-maximum suppression is recorded as T, the number of pixels in the image whose grayscale value is less than the threshold T is recorded as N0, the number of pixels whose grayscale value is greater than the threshold T is recorded as N1, the proportion of the number of pixels belonging to the foreground in the entire image is recorded as w0, and the proportion of the number of pixels belonging to the background in the entire image is recorded as w1, then:

[0109]

[0110]

[0111] Among them, the size of the grayscale image is M×N, that is:

[0112] N0+N1=M×N (3)

[0113] w0+w1=1 (4)

[0114] The total average grayscale of the image is recorded as μ, the average grayscale of the foreground is recorded as μ0, and the average grayscale of the background is recorded as μ1, then:

[0115] μ=w0×μ0+w1×μ1 (5)

[0116] Therefore, the inter-class variance g of foreground and background is:

[0117] g=w0×(π0-μ) 2 +w1×(π1-μ) 2 (6)

[0118] The traversal method is used to find the threshold T that maximizes the inter-class variance g. This threshold is the adaptive threshold of the grayscale image. This threshold is multiplied by a certain ratio as the high and low thresholds of the Canny edge detection algorithm.

[0119] Double threshold detection is performed based on high and low thresholds to obtain image edge data, namely the first lane line detection result.

[0120] It should be noted that Canny edge detection is a very popular edge detection algorithm proposed by John Canny in 1986.

[0121] In a preferred embodiment of the present invention, a semantic segmentation model is established, and knowledge distillation and quantization-aware training are performed based on the semantic segmentation model to obtain a fully trained lightweight semantic segmentation model, including:

[0122] Establish a semantic segmentation model and perform knowledge distillation training based on the semantic segmentation model to obtain a preliminary optimized semantic segmentation model;

[0123] Based on the preliminary optimized semantic segmentation model, quantization-aware training is performed to obtain a fully trained lightweight semantic segmentation model.

[0124] In a specific embodiment, a fully trained lightweight semantic segmentation model is obtained by performing knowledge distillation and quantization-aware training on the semantic segmentation model. The knowledge distillation and quantization-aware training on the semantic segmentation model reduce the running time and model size of the fully trained lightweight semantic segmentation model while ensuring the accuracy of the model. For details, please refer to Figure 2 , Figure 2 This is a lightweight data diagram of the model provided by the present invention.

[0125] In the figure, Deeplabv3p-ResNet50 is the teacher model for knowledge distillation training, Deeplabv3p-ResNet18 is the semantic segmentation model, and Deeplabv3p-ResNet18-quantizede is a fully trained lightweight semantic segmentation model.

[0126] In a preferred embodiment of the present invention, a semantic segmentation model is established, and knowledge distillation training is performed based on the semantic segmentation model to obtain a preliminary optimized semantic segmentation model, including:

[0127] Obtain road images containing lane lines as training sets;

[0128] Get a fully trained teacher model;

[0129] Based on the teacher model, lane recognition is performed on the road image containing lane lines to obtain the teacher lane recognition result;

[0130] The road image containing lane lines is identified based on the semantic segmentation model to obtain the student lane line recognition result;

[0131] The conventional loss is obtained by calculating the student lane line recognition results and the true recognition results based on the cross entropy loss function;

[0132] Based on the relative entropy loss, the student lane recognition results and the teacher lane recognition results are calculated to obtain the distillation loss;

[0133] The semantic segmentation model is iteratively trained based on distillation loss and conventional loss as loss parameters to obtain a preliminary optimized semantic segmentation model.

[0134] In a specific embodiment, during the implementation of knowledge distillation, the teacher model selects the more complex and larger Deeplabv3p-ResNet50 model, and the student model selects the smaller Deeplabv3p-ResNet18 model.

[0135] In a preferred embodiment of the present invention, a fully trained lightweight semantic segmentation model is obtained by performing quantization-aware training based on a preliminary optimized semantic segmentation model, including:

[0136] Insert a pseudo-quantization operator before the convolution operator of the preliminary optimized semantic segmentation model to simulate the simulation error caused by the quantization of the preliminary optimized semantic segmentation model;

[0137] The model parameters are optimized according to the simulation error to obtain a fully trained lightweight semantic segmentation model.

[0138] In a specific embodiment, a pseudo quantization operator is inserted before the quantization operation node (i.e., the convolution operator);

[0139] Execute the "quantization-dequantization" logic in advance to obtain the simulation error;

[0140] The model parameters are optimized according to the simulation error, and the pseudo-quantization operator is deleted to obtain a fully trained lightweight semantic segmentation model.

[0141] In a preferred embodiment of the present invention, the first lane line detection result and the second lane line detection result are fused and identified based on a sliding window algorithm to output a target lane line detection result, including:

[0142] Calculating, based on a sliding window algorithm, each window feature value of the first lane line detection result and each window feature value of the second lane line detection result for the first lane line detection result and the second lane line detection result;

[0143] Calculating the distance between the feature values ​​of each window of the first lane line detection result and the feature values ​​of each window of the second lane line detection result and the expected feature value based on the feature values ​​of each window of the first lane line detection result and the feature values ​​of each window of the second lane line detection result;

[0144] Based on the distances of the feature values ​​of each window of the first lane line detection result and the feature values ​​of each window of the second lane line detection result to the expected feature value, the windows with smaller distances to the expected feature value are selected and fused into one image, and the target lane line detection result is outputted through optimization.

[0145] In a specific embodiment, the sliding window algorithm is processed as follows: the first lane line detection result and the second lane line detection result are binary black and white images. For example, the background is black and the lane line is white. Assume that the image resolution is x*y, that is, y rows and x columns. The number of white points in each column is counted to obtain x values. These x values ​​are plotted in a coordinate system with the horizontal axis ranging from 1 to x and the vertical axis representing the number of white points in each column. A histogram is thus obtained. For details, please refer to Figure 3 , Figure 3 The pixel statistical histogram provided by the present invention shows that the column number corresponding to the maximum value of the left half of the histogram is the approximate position of the left lane line; the column number corresponding to the maximum value of the right half of the histogram is the approximate position of the right lane line. These two approximate positions are used as starting points. Define a rectangular area, called a "window", and use the two starting points as the midpoints of the lower edge of the window. Store the horizontal coordinates of all white points in the square, then take the average of the stored horizontal coordinates, and use the column where the mean is located and the position of the upper edge of the first "window" as the midpoint of the lower edge of the next "window" to continue searching. Repeat this process until all rows have been searched, and the horizontal coordinates of the midpoints of the lower edge of each window are obtained as the horizontal coordinates of the eigenvalues.

[0146] For the previous n lane line detection outputs, the sliding window algorithm is calculated and divided into m windows by row. The horizontal coordinate x of the bottom midpoint of the lane line window on the left and right sides is recorded as vector x. l and x r , according to the definition x l ,x r ∈R m ,Right now:

[0147] x l =[x l1 x l2 x l3 …x lm ] T (7)

[0148] x r =[x r1 x r2 x r3 …x rm ] T (8)

[0149] For n lane line detection outputs, it can be recorded as an m-row n-column matrix A, A∈R m×n , the lane line matrices on the left and right sides are:

[0150] A l =[x l1 x l2 …x ln ] (9)

[0151] A r =[x r1 x r2 …x rn ] (10)

[0152] Taking the left lane line as an example, l Expanded to:

[0153]

[0154] Calculate the mean coordinates of the midpoints of each window for n lane line detection outputs as the expectation of the current lane line detection output. Taking the left lane line as an example, A l Calculate the arithmetic mean of each row of the matrix to obtain the vector e, e∈R m , that is, the expected eigenvalue, the specific formula is as follows:

[0155]

[0156] The first lane line detection result and the second lane line detection result are also processed by windowing to obtain the horizontal coordinate output of the midpoint at the bottom of the window, which are recorded as x and o and x d , x l ,x r ∈R m ,Right now:

[0157] x o =[x o1 x o2 x o3 …x om ] T (13)

[0158] x d =[x d1 x d2 x d3 …x dm ] T (14)

[0159] Find x respectively o with x d The distance from e is denoted as d o with d d , with do For example, there are:

[0160] d o =x o -e=[x o1 -e1x o2 -e2…x om -e m ] T (15)

[0161] Compare the distance between the feature values ​​of each window of the first lane line detection result and the second lane line detection result and the expected value, and fuse all windows with smaller distances to the expected feature value into a new image. The current lane line detection output after the fusion decision is obtained as shown in (16).

[0162] F(OpenCV|DeepLab)=min(d o , d d ) (16)

[0163] Finally, based on the eigenvalues ​​of each window of the first lane line detection result and the distances of the eigenvalues ​​of each window of the second lane line detection result to the expected eigenvalue, the windows with the smallest distances to the expected eigenvalue are selected and fused into one image. The optimized output of the target lane line detection result includes:

[0164] The windows with the smallest distance to the expected eigenvalue are selected and fused into one image. Based on this image, Hough transform is used to detect straight lines in the image, and the lane line slope is calculated. Lane lines with unreasonable slopes are removed, and a quadratic function is used to fit and optimize the current lane line detection result to output the target lane line detection result.

[0165] Hough transform detection is used to eliminate lane lines with unreasonable slopes and reduce the false recognition rate, further improving the fitting accuracy. A quadratic function is used to fit the current lane line detection results to reduce the false recognition rate and further improve the accuracy.

[0166] Compared with the existing technology, the present invention obtains an input image by obtaining a road picture containing lane lines and preprocessing the road picture; then, lane line edge detection is performed on the input image based on the edge detection algorithm to output a first lane line detection result; secondly, a semantic segmentation model is established, and knowledge distillation and quantization perception training are performed based on the semantic segmentation model to obtain a fully trained lightweight semantic segmentation model, and lane line detection is performed on the input image based on the lightweight semantic segmentation model to output a second lane line detection result; finally, the first lane line detection result and the second lane line detection result are fused and identified based on the sliding window algorithm to output a target lane line detection result. The present invention improves the adaptability and robustness of the algorithm by performing lane line recognition through the edge detection algorithm. Secondly, a well-trained lightweight semantic segmentation model is obtained through knowledge distillation and quantized perception training. Lane lines are detected based on the lightweight semantic segmentation model to ensure the recognition accuracy of the model and improve the running speed. Finally, a sliding window algorithm is used to fuse the first lane line detection result and the second lane line detection result to output the target lane line detection result. This helps to overcome the problem of unstable detection based on the edge detection algorithm, and improves the shortcomings of the semantic segmentation network that does not fully utilize the prior knowledge of the lane line, overcomes the limitations of the single lane line detection algorithm, and improves the accuracy and stability of lane line detection.

[0167] The above is a detailed introduction to the lane line fusion recognition method based on semantic segmentation and edge detection algorithm proposed in the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. The above is only a preferred specific implementation method of the present invention, but the protection scope of the present invention is not limited to this. Any changes or replacements that can be easily thought of by any technical personnel familiar with this technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A lane line fusion recognition method based on semantic segmentation and edge detection algorithm, characterized in that: include: Obtaining a road image containing lane lines, and preprocessing the road image to obtain an input image; Performing lane line edge detection on the input image based on an edge detection algorithm to output a first lane line detection result; Establish a semantic segmentation model, perform knowledge distillation and quantization perception training based on the semantic segmentation model to obtain a fully trained lightweight semantic segmentation model, and perform lane line detection on the input image based on the fully trained lightweight semantic segmentation model to output a second lane line detection result. The fully trained lightweight semantic segmentation model is trained based on the Deeplabv3p-ResNet18 model. The fully trained lightweight semantic segmentation model includes: an encoding module and a decoding module; the encoding module includes a backbone network DCNN and a dilated spatial pyramid pooling module ASPP cascaded therewith; the dilated spatial pyramid pooling module ASPP includes a cascaded dilated convolution module and an average pooling layer; the dilated convolution module includes three dilated convolution layers with dilated convolution rates arranged from small to large; the decoding module includes two convolution layers and two fourfold bilinear interpolation upsampling modules; The first lane line detection result and the second lane line detection result are fused and identified based on a sliding window algorithm to output a target lane line detection result, including: Calculating, based on a sliding window algorithm, each window feature value of the first lane line detection result and each window feature value of the second lane line detection result for the first lane line detection result and the second lane line detection result; Calculating the distance between the feature values ​​of each window of the first lane line detection result and the feature values ​​of each window of the second lane line detection result and the expected feature value based on the feature values ​​of each window of the first lane line detection result and the feature values ​​of each window of the second lane line detection result; Based on the distances of the feature values ​​of each window of the first lane line detection result and the feature values ​​of each window of the second lane line detection result to the expected feature value, the windows with smaller distances to the expected feature value are selected and fused into one image, and the target lane line detection result is outputted through optimization.

2. The lane line fusion recognition method based on semantic segmentation and edge detection algorithm according to claim 1 is characterized in that: Based on the fully trained lightweight semantic segmentation model, lane line detection is performed on the input image to output a second lane line detection result, including: Perform feature extraction on the input image based on the backbone network DCNN to obtain a first feature map; Based on the dilated convolution module in the dilated spatial pyramid pooling module ASPP, the first feature map is subjected to dilated convolution processing, and then the average pooling layer is used to perform pooling processing to obtain a second feature map; Based on the decoding module, the second feature map is up-sampled by four times bilinear interpolation to obtain a second lane line detection result.

3. The lane line fusion recognition method based on semantic segmentation and edge detection algorithm according to claim 1, characterized in that: Obtain a road image containing lane lines and preprocess the road image to obtain an input image, including: Get a road image containing lane lines; The input image is obtained by performing resolution adjustment, image correction, and image enhancement on the road image containing lane lines.

4. The lane line fusion recognition method based on semantic segmentation and edge detection algorithm according to claim 1, characterized in that: Performing lane line edge detection on the input image based on an edge detection algorithm and outputting a first lane line detection result, including: Converting the input image into a grayscale image, and performing image masking on the grayscale image to obtain a masked image; Performing Gaussian filtering on the masked image to obtain a preliminary lane line image; Lane line detection is performed on the preliminary lane line image based on the edge detection algorithm to obtain the first lane line detection result.

5. The lane line fusion recognition method based on semantic segmentation and edge detection algorithm according to claim 1, characterized in that: Establish a semantic segmentation model, perform knowledge distillation and quantization-aware training based on the semantic segmentation model to obtain a fully trained lightweight semantic segmentation model, including: Establish a semantic segmentation model and perform knowledge distillation training based on the semantic segmentation model to obtain a preliminary optimized semantic segmentation model; Based on the preliminary optimized semantic segmentation model, quantization-aware training is performed to obtain a fully trained lightweight semantic segmentation model.

6. The lane line fusion recognition method based on semantic segmentation and edge detection algorithm according to claim 5, characterized in that: Establish a semantic segmentation model and perform knowledge distillation training based on the semantic segmentation model to obtain a preliminary optimized semantic segmentation model, including: Obtain road images containing lane lines as training sets; Get a fully trained teacher model; Performing lane recognition on a road image containing lane lines based on the fully trained teacher model to obtain a teacher lane recognition result; The road image containing lane lines is identified based on the semantic segmentation model to obtain the student lane line recognition result; The conventional loss is obtained by calculating the student lane line recognition results and the true recognition results based on the cross entropy loss function; Based on the relative entropy loss, the student lane recognition results and the teacher lane recognition results are calculated to obtain the distillation loss; The semantic segmentation model is iteratively trained based on distillation loss and conventional loss as loss parameters to obtain a preliminary optimized semantic segmentation model.

7. The lane line fusion recognition method based on semantic segmentation and edge detection algorithm according to claim 5, characterized in that: Based on the preliminary optimized semantic segmentation model, quantization-aware training is performed to obtain a fully trained lightweight semantic segmentation model, including: Insert a pseudo-quantization operator before the convolution operator of the preliminary optimized semantic segmentation model to simulate the simulation error caused by the quantization of the preliminary optimized semantic segmentation model; The model parameters are optimized according to the simulation error to obtain a fully trained lightweight semantic segmentation model.

Images