Enhanced backbone network real-time lane line detection method

Abstract

The invention relates to the technical field of lane line detection, and particularly provides an enhanced backbone network real-time lane line detection method, which comprises the steps of inputting a to-be-detected target image into an enhanced backbone network structure for image feature extraction to obtain a first high-level feature, a first low-level feature and a first middle-level feature; and inputting the first high-level feature into the feature pyramid for up-sampling, and fusing the first high-level feature with the first middle-level feature and the first low-level feature to obtain a fused feature map. And extracting feature information around the priori lane line from the fused feature map, and calculating to obtain lane line features. And processing the lane line features, outputting a posterior lane line, and carrying out post-processing to obtain a final lane line as a lane line detection result. According to the method, the C3k2 enhanced SPPF backbone network is arranged, the parameter quantity and the calculation complexity are remarkably reduced, the reasoning time delay of the model is greatly reduced, and the memory occupation is optimized. The curved prior lane line is introduced to fit the real road condition, and the detection precision and accuracy are improved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of lane line detection, in particular to a real-time lane line detection method based on an enhanced backbone network. BACKGROUND

[0002] Lane lines, as one of the important indication information of the road surface, include solid lines, dashed lines and arrow markings. Lane line detection, as the basis of the intelligent driving field, uses collected road image information combined with image processing and deep learning algorithms to quickly identify lane lines on the road, providing key environmental perception information for path planning, lane keeping and automatic lane changing of vehicles, helping vehicles accurately determine their position in the lane during driving and ensuring driving safety and stability.

[0003] Most existing network models for lane line detection use classic convolutional neural networks as backbone networks. In order to obtain higher detection accuracy, multiple convolutional layers and pooling layers are stacked to balance the details and semantic understanding capabilities, which leads to a sharp increase in memory usage, a large-scale computation and a long inference delay, making it difficult to meet real-time detection requirements. Reducing the number of model layers will result in insufficient feature extraction levels and weak semantic information expression capabilities, leading to a significant decrease in detection accuracy. SUMMARY

[0004] The enhanced backbone network real-time lane line detection method provided by the embodiments of the application at least solves the problem that the number of model layers and the amount of calculation cannot be balanced, and it is difficult to improve detection accuracy with less time delay.

[0005] In a first aspect, the embodiments of the application provide an enhanced backbone network real-time lane line detection method, which includes:

[0006] inputting a target image to be detected into an enhanced backbone network structure for image feature extraction to obtain first high-level features; wherein the enhanced backbone network structure also outputs first low-level features and first middle-level features;

[0007] inputting the first high-level features into a feature pyramid sampling and sequentially fusing the first middle-level features and the first low-level features to obtain a fused feature map;

[0008] extracting feature information around a prior lane line from the fused feature map to calculate lane line features; wherein the prior lane line is obtained by clustering analysis of a lane line image dataset, and the prior lane line includes curved lane lines;

[0009] processing the lane line features to output a posterior lane line, and post-processing the posterior lane line to obtain a final lane line as a lane line detection result.

[0010] The enhanced backbone network real-time lane line detection method provided by the embodiment of the present invention inputs the target image to be detected into the enhanced backbone network structure to extract image features, and the first high-level features obtained include:

[0011] Input the target image into the convolution operation layer of the enhanced backbone network for convolution processing, and output a convolution feature map;

[0012] Inputting the convolutional feature map into the C3k2 module group of the enhanced backbone network, and sequentially outputting the first low-level features and the first mid-level features; wherein the C3k2 module group includes five C3k2 modules arranged in series;

[0013] Inputting the first middle-level features into the fast spatial pyramid pooling layer for pooling to obtain a pooled output feature map;

[0014] The pooled output feature map is input into the feature optimization layer to output the first high-level feature.

[0015] The enhanced backbone network real-time lane line detection method provided by the embodiment of the present invention inputs the convolutional feature map into the C3k2 module group of the enhanced backbone network, and sequentially outputs the first low-level feature and the first middle-level feature, including:

[0016] The convolution feature map is used as the input of the C3k2 module, and is convolved and divided into a first branch and a second branch; wherein the number of channels of the first branch and the second branch is the same;

[0017] When the C3k parameter is true, the first tributary is processed through multiple C3k layers and then feature-joined with the second tributary; when the C3k parameter is not true, the first tributary is processed through a bottleneck structure and then feature-joined with the second tributary;

[0018] The result of splicing the second tributary with the processed first tributary features is convolutionally fused to obtain the output of the C3k2 module; wherein the output of each C3k2 module is set as the input of the next C3k2 module; the output of the third C3k2 module arranged in series is set as the first low-level feature, and the output of the fifth C3k2 module arranged in series is set as the first middle-level feature.

[0019] The enhanced backbone network real-time lane line detection method provided by the embodiment of the present invention inputs the first middle-level features into the fast spatial pyramid pooling layer for pooling, and obtains a pooled output feature map including:

[0020] Convolution processing is performed on the first middle-layer features, and a multi-scale maximum pooling operation is performed on the convolution result, and pooling results of different sizes are obtained corresponding to pooling kernels of different sizes; wherein the pooling kernel sizes for the multi-scale maximum pooling operation include 1×1, 3×3, 5×5, and 7×7;

[0021] The pooling result is restored to its size by upsampling and then element-by-element addition is performed to obtain the pooling output feature map.

[0022] The enhanced backbone network real-time lane line detection method provided by the embodiment of the present invention inputs the first high-level feature into the feature pyramid, and sequentially fuses it with the first middle-level feature and the first low-level feature to obtain a fused feature map, including:

[0023] Input the first high-level features into the feature pyramid and perform upsampling to obtain the second middle-level features;

[0024] Fusing the second mid-level features with the first mid-level features to obtain a mid-level fusion result; wherein the resolution of the second mid-level features is greater than the resolution of the first high-level features;

[0025] Upsampling the middle-level fusion result in the feature pyramid to obtain a second low-level feature;

[0026] The second low-level features are fused with the first low-level features to obtain the fused feature map; wherein the resolution of the second low-level features is greater than the resolution of the second middle-level features.

[0027] The enhanced backbone network real-time lane line detection method provided by an embodiment of the present invention further includes, before extracting feature information around the prior lane line in the fused feature map:

[0028] Collecting lane line image data, and annotating the lane line image data to obtain the lane line image dataset;

[0029] Input the lane line image dataset into the clustering algorithm, set the clustering target and the number of clusters, and initialize the cluster centers;

[0030] Execute a clustering algorithm to update the cluster center and output the prior lane line.

[0031] The enhanced backbone network real-time lane line detection method provided by the embodiment of the present invention extracts feature information around the prior lane line in the fused feature map, and calculates the lane line features including:

[0032] Positioning the a priori lane line in the fused feature map, and defining a region of interest around the positioning result;

[0033] extracting features in the region of interest and performing feature compression to obtain an intermediate vector;

[0034] performing attention calculation on the fusion feature map through the intermediate vector to obtain an attention weight matrix;

[0035] performing weighted summation on pixel features of the fusion feature map corresponding to elements in the attention weight matrix to obtain the lane line feature.

[0036] The enhanced backbone network real-time lane line detection method provided by the embodiment of the application includes:

[0037] inputting the lane line feature into a first fully connected layer for feature transformation and activating using an activation function;

[0038] inputting the activated feature into a second fully connected layer for feature transformation and activating again using the activation function to obtain an intermediate feature;

[0039] inputting the intermediate feature into a third fully connected layer to output a confidence degree; wherein the confidence degree corresponds to the posterior lane line one by one;

[0040] inputting the intermediate feature into a fourth fully connected layer to output a lane line abscissa;

[0041] obtaining a sequence coordinate of the posterior lane line according to the lane line abscissa and a lane line ordinate; wherein the lane line ordinate is preset at equal intervals in the range of 1 to 0;

[0042] taking the sequence coordinate of the posterior lane line and the confidence degree corresponding to the posterior lane line as an output result of the posterior lane line.

[0043] The enhanced backbone network real-time lane line detection method provided by the embodiment of the application includes:

[0044] performing elimination and arrangement according to the sequence coordinate of the posterior lane line to retain an effective posterior lane line;

[0045] processing the effective posterior lane line through a non-maximum suppression algorithm to output the posterior lane line with the highest confidence degree and remove other posterior lane lines;

[0046] taking the posterior lane line output by the non-maximum suppression algorithm as the final lane line to output a lane line detection result.

[0047] In a second aspect, the embodiments of the present application also provide an electronic device, comprising a processor and a memory storing a program, wherein the program comprises instructions which, when executed by the processor, cause the processor to perform the enhanced backbone network real-time lane line detection method according to the embodiments described above.

[0048] The enhanced backbone network real-time lane line detection method provided by the embodiments of the present application solves the technical problem that increasing the number of model layers to improve the detection effect will be accompanied by an increase in detection time overhead, sets a C3k2 enhanced SPPF backbone network, and compared with other common backbone networks, the required parameter amount and computational complexity are significantly reduced when the same number of layers is reached, greatly reducing the inference delay of the model and optimizing the memory occupation, and realizing real-time detection. By pre-training and clustering in large-scale lane line image data, the curved prior lane line is obtained, which can provide better initialization effect than the artificially defined straight lane line, fit the real road conditions, and improve the detection precision and accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other embodiments can also be obtained from these drawings without creative labor.

[0050] Figure 1 is a flowchart of the enhanced backbone network real-time lane line detection method of the embodiments of the present application.

[0051] Figure 2 is an implementation schematic diagram of the enhanced backbone network real-time lane line detection method of the embodiments of the present application.

[0052] Figure 3 is a structure diagram of the enhanced backbone network of the embodiments of the present application.

[0053] Figure 4 is a network structure diagram of a C3k2 module of the embodiments of the present application.

[0054] Figure 5 is a network structure diagram of a fast spatial pyramid pooling layer of the embodiments of the present application.

[0055] Figure 6 is a feature pyramid structure diagram of the embodiments of the present application.

[0056] Figure 7 is a prior lane line schematic diagram of the embodiments of the present application.

[0057] Figure 8 Schematic diagram of the implementation of generating lane line features in an embodiment of the present invention.

[0058] Figure 9 This is a schematic diagram of the implementation of obtaining the final lane line in step S400 of the embodiment created by the present invention.

[0059] Figure 10 It is a structural schematic diagram of the electronic device created by the present invention. DETAILED DESCRIPTION

[0060] The following describes embodiments of the present invention in more detail with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0061] The network structure of the lane detection model includes the following three designs in related technologies:

[0062] One is the Spatial Convolutional Neural Network (SCNN). SCNN extends traditional layer-by-layer convolution to slice-by-slice convolution within the feature map, enabling information transfer between pixels in the same layer. It is suitable for processing long, continuous shapes or large objects with strong spatial relationships but few appearance cues, such as traffic lanes, utility poles, and walls. SCNN treats the rows or columns of the feature map as "layers" and sequentially applies convolution, nonlinear activation, and summation operations to form a deep neural network. In this way, information can be propagated between neurons in the same layer, thereby enhancing spatial information.

[0063] Given a three-dimensional tensor of size C × H × W, SCNN divides it into H slices. The first slice is fed into a convolutional layer, whose output is added to the next slice, and the cycle repeats until the last slice is updated. This structure allows each pixel to receive information propagating in a specific direction. However, SCNN detection accuracy is low and latency is long.

[0064] The second is the Conditional Convolution-based Lane Detection Network (CondLaneNet). CondLaneNet is a top-down lane detection framework that first detects lane instances and then dynamically predicts the shape of each instance. Inspired by CondInst and SOLOv2, a conditional lane detection strategy based on conditional convolution and a row-by-row formula is proposed. The optimization focuses on specifying lane shapes based on the row-by-row formula. To address the problem of detecting lanes with complex topologies, a RIM was designed. Based on long short-term memory, RIM cyclically predicts the state vector and kernel parameter vector, allowing the same proposal point to predict multiple lane shapes. In its implementation, the backbone network uses a residual network (ResNet) and the bottleneck layer uses a Transformer FPN. Considering the long length of lanes, which requires global attention, self-attention is performed on the features output by the backbone network before the basic FPN constructs the pyramid. Although CondLaneNet is computationally efficient, its detection accuracy is insufficient.

[0065] The third is the Cross Layer Recurrent Network (CLRerNet). It uses straight lines as prior lane lines and improves the accuracy and robustness of lane line detection through cross-layer refinement. In the feature extraction layer, ResNet or Deep Layer Aggregation (DLA) is used as the backbone network to extract multi-scale features of the image. CLRNet combines high-level semantic information with low-level detail information through a cross-layer refinement mechanism to gradually optimize the lane line detection results. This mechanism can effectively solve the problem of lane line detection at different scales. The disadvantage of CLRNet is its long latency.

[0066] In summary, in the field of lane line detection, related detection methods are difficult to balance the number of stacked layers of convolutional neural networks and detection accuracy, and are unable to control the amount of calculation and delay while ensuring detection accuracy. To this end, the present invention creates an embodiment 1 that provides an enhanced backbone network real-time lane line detection method. The backbone network is set to a C3k2 enhanced SPPF backbone network. Compared with other backbone networks, the number of parameters and computational complexity required to achieve the same number of layers are lower. While ensuring the accuracy of lane line detection, the model's inference delay is greatly reduced, memory usage is optimized, and it is more conducive to the smooth operation of the vehicle's intelligent assisted driving system.

[0067] In related technologies, lane detection models primarily use various classic convolutional neural networks as the backbone network, with multi-scale feature fusion as the bottleneck layer. A fully connected output layer is superimposed on this layer as the mainstream detection method. The output vector is a sequence of lane line coordinates in the image. To achieve high detection accuracy, multiple layers of convolutional neural networks must be stacked, which increases memory usage, computational complexity, and latency, making real-time detection difficult. Reducing the number of model layers to reduce computational complexity and latency also results in reduced detection accuracy.

[0068] For this purpose, refer to Figure 1 and Figure 2 As shown, the first embodiment of the present invention provides an enhanced backbone network real-time lane line detection method, which can avoid delay and excessive computational complexity while maintaining detection accuracy, and realize real-time lane line detection while taking into account detection efficiency.

[0069] Specifically, refer to Figure 1 As shown in FIG, the enhanced backbone network real-time lane line detection method includes the following steps:

[0070] Step S100: Input the target image to be detected into the enhanced backbone network structure to extract image features and obtain first high-level features. The enhanced backbone network structure also outputs first low-level features and first mid-level features.

[0071] In the first embodiment of the present invention, the backbone network is set to a C3k2 enhanced SPPF backbone network. Compared with the technical solutions in the related art that use residual networks (ResNet), deep layer aggregation networks (Deep Layer Aggregation, DLA) or visual geometry group networks (Visual Geometry Group Network, VGG) as backbone networks, the C3k2 enhanced SPPF backbone network requires fewer parameters and computational complexity to achieve the same number of layers; based on the traditional spatial pyramid pooling (Spatial Pyramid Pooling, SPP), the optimized fast spatial pyramid pooling (Spatial Pyramid Pooling Fast, SPPF) is integrated into the C3k2 network backbone as a key component for feature extraction. SPPF can enhance feature expression through the attention layer, fuse different features, and enhance the model's ability to detect targets of different scales.

[0072] Reference Figure 3As shown in the figure, the C3k2-enhanced SPPF backbone network consists of eight layers. Specifically, it includes a convolutional layer (conv), a C3k2 module repeated five times, a fast spatial pyramid pooling layer (SPPF), and a feature optimization layer (C2PSA). The C3k2 module is the core of the C3k2-enhanced SPPF backbone network.

[0073] As an implementable method, step S100 includes the following steps:

[0074] In step S110, the target image X0 to be detected is input into the convolution operation layer conv of the C3k2 enhanced SPPF backbone network for convolution processing, and a convolution feature map X1 = conv(X0) is output.

[0075] In step S120, the convolutional feature map X1 is input into the C3k2 module group of the C3k2 enhanced SPPF backbone network, which sequentially outputs the first low-level feature conv (X4) and the first middle-level feature conv (X6). The C3k2 module group includes C3k2 modules arranged in series and repeated five times.

[0076] Further, refer to Figure 4 As shown in Figure 1, the C3k2 module is an improved convolutional neural network structure. The input feature map is cleverly split into two parts within the C3k2 module. Within the C3k2 module group, the convolutional feature map X1 is used as the input to the first C3k2 module. After convolution, the convolutional feature map X1 is split into the first branch Y1 and the second branch Y2. Thus, Y1, Y2 = split(X1). The first branch Y1 and the second branch Y2 have the same number of channels.

[0077] The features of the second branch, Y2, are directly transmitted, preserving shallow features of the target image, including important information such as basic texture and edges. The features of the first branch, Y1, undergo more complex processing through multiple bottleneck structures or variable convolution kernels to extract deep features of the target image. These deep features can better capture the semantic information and high-level structure of the image.

[0078] Specifically, when the C3k parameter is true, the first tributary Y1 is processed through multiple C3k layers to obtain deep features, which are then concatenated with the features of the second tributary Y2 to obtain X2. In this case, X2 = concat(C3k(…(C3k(Y1))), Y2). When the C3k parameter is false, the first tributary Y1 is processed through a bottleneck structure to obtain deep features, which are then concatenated with the features of the second tributary Y2 to obtain X2.

[0079] The output of the first C3k2 module, conv(X2), is obtained by concatenating the second branch Y2 with the processed first branch features through a 1×1 convolution. The two parts are processed separately and then concatenated to form a feature representation that combines shallow and deep information.

[0080] It should be noted that when the number of input and output channels of the C3k2 module is c and the convolution kernel size k is 3, the number of parameters of the C3k2 module with 4 layers of convolution is 2c 2 (1+k 2 )=20c 2 . Set the size of the target image to (h, w), then the computational cost of the C3k2 module is 2c 2 (1+k 2 )*hw=20c 2 hw.

[0081] Correspondingly, the number of parameters of the residual network with four convolution layers is 36c 2 , for the target image of the same size, the computational cost is 36c 2 The computational complexity and parameter count of the C3k2 module in the first embodiment of the present invention are only 56% of those of the residual network.

[0082] Set the output conv(X2) of the first C3k2 module as the input of the next C3k2 module. The processing in each C3k2 module refers to Figure 4 As shown, the above steps are repeated to perform convolution, segmentation, feature splicing, and then fusion convolution to obtain the output. The output conv(X3) of the second C3k2 module is used as the input of the third C3k2 module, the output conv(X4) of the third C3k2 module is used as the input of the fourth C3k2 module, the output conv(X5) of the fourth C3k2 module is used as the input of the fifth C3k2 module, and the output conv(X6) of the fifth C3k2 module is used as the output of the entire C3k2 module group. Among them, the output conv(X4) of the third C3k2 module set in series is set as the first low-level feature, and the output conv(X6) of the fifth C3k2 module set in series is set as the first mid-level feature.

[0083] In step S130 , the first middle-level features are input into a fast spatial pyramid pooling layer SPPF for pooling to obtain a pooled output feature map.

[0084] Fast Spatial Pyramid Pooling (SPPF) is an improved version of traditional Spatial Pyramid Pooling (SPP). Spatial Pyramid Pooling (SPP) is a structure that effectively processes information at different scales. Fast Spatial Pyramid Pooling (SPPF) inherits the advantages of SPP and further optimizes and improves on them. Through the Fast Spatial Pyramid Pooling layer, the backbone network can better process extracted features, thereby enhancing the feature extraction capabilities of the entire network, enabling the network to more accurately capture key information in the image and providing stronger support for subsequent tasks such as lane detection.

[0085] Specifically, refer to Figure 5 As shown in the figure, the first middle layer feature is convolved, and the convolution result X7 is subjected to a multi-scale maximum pooling operation, and pooling results of different sizes are obtained corresponding to pooling kernels of different sizes; among them, the pooling kernels for the multi-scale maximum pooling operation include pooling kernels of sizes 1×1, 3×3, 5×5, and 7×7.

[0086] The convolution result X7 is subjected to a maximum pooling operation with a convolution kernel size of 1×1 to obtain X8; the convolution result X7 is subjected to a maximum pooling operation with a convolution kernel size of 3×3 to obtain X9; the convolution result X7 is subjected to a maximum pooling operation with a convolution kernel size of 5×5 to obtain X10; the convolution result X7 is subjected to a maximum pooling operation with a convolution kernel size of 7×7 to obtain X11.

[0087] When the size of the first middle-level feature input to the fast spatial pyramid pooling layer SPPF is H×W×C, the size of the pooling result X8 obtained by the multi-scale maximum pooling operation is H×W×C; the size of the pooling result X9 is [H / 3]×[W / 3]×C; the size of the pooling result X10 is [H / 5]×[W / 5]×C; and the size of the pooling result X11 is [H / 7]×[W / 7]×C.

[0088] The pooling result X8 remains unchanged, the pooling result X9 is restored to the original size H×W×C through upsampling, the pooling result X10 is restored to the original size H×W×C through upsampling, and the pooling result X11 is restored to the original size H×W×C through upsampling. Then, perform element-by-element addition to fuse the features of the multi-scale pooling results after the restoration to obtain the pooled output feature map.

[0089] In step S140, the pooled output feature map obtained in step S130 is input to the feature optimization layer C2PSA to output the first high-level feature. The first low-level feature conv (X4), the first mid-level feature conv (X6), and the first high-level feature output by the C3k2 enhanced SPPF backbone network each include feature information at different levels. Low-level features typically have higher resolution and can capture detailed information in the image, such as lane edges and textures. High-level features have a wider field of view and stronger semantic expression capabilities, and can understand the overall structure and context of the image.

[0090] In step S200, the first high-level feature is input into the feature pyramid, and is sequentially fused with the first middle-level feature conv(X6) and the first low-level feature conv(X4) to obtain a fused feature map P.

[0091] The output of the C3k2-enhanced SPPF backbone network includes both low-level features such as lane edges and textures, as well as high-level features such as image structure and contextual information. However, a single level of features cannot meet the dual requirements of detail and semantics in lane detection. Therefore, step S200 introduces a feature pyramid to further optimize feature extraction. This feature pyramid is used to fuse multi-level features from the output of the C3k2-enhanced SPPF backbone network.

[0092] Specifically, refer to Figure 6 As shown, in step S100, the C3k2 enhanced SPPF backbone network outputs the first low-level feature conv (X4), the first middle-level feature conv (X6) and the first high-level feature obtained in sequence through the downsampling process in resolution from high to low.

[0093] The first high-level features are input into the feature pyramid and upsampled to obtain the second middle-level features, where the resolution of the second middle-level features is greater than that of the first high-level features.

[0094] The second middle-level features obtained by upsampling are fused with the first middle-level features conv(X6) output by the C3k2 enhanced SPPF backbone network to obtain the middle-level fusion result.

[0095] The middle-level fusion result is upsampled in the feature pyramid to obtain the second-lowest-level features, and the resolution of the second-lowest-level features is greater than that of the second-middle-level features.

[0096] The second low-level features are fused with the first low-level features conv(X4) output by the C3k2 enhanced SPPF backbone network to obtain the fused feature map P.

[0097] Through upsampling and downsampling operations, the semantic information of high-level features is transferred to low-level features, while the detailed information of low-level features is supplemented to high-level features. Feature maps at different levels are integrated together through specific fusion strategies, which can be set to weighted summation or splicing. The resulting fused feature map P retains the high resolution and detailed information of low-level features while integrating the semantic information of high-level features, which can more comprehensively represent the characteristics of lane lines. All features from low to high levels are effectively integrated into these fused feature maps, providing the subsequent lane detection module with a richer, more accurate, and hierarchical feature representation, thereby improving the accuracy and robustness of lane detection.

[0098] Step S300: extract feature information around the prior lane line from the fused feature map P obtained in step S200, and calculate the lane line feature.

[0099] Specifically, before step S300, the lane line detection method implemented by the enhanced backbone network further includes obtaining a priori lane lines.

[0100] In the first embodiment of the present invention, the a priori lane lines are obtained by clustering the lane line image data set. When obtaining the a priori lane lines, the lane line image data is first collected and annotated to obtain the lane line image data set.

[0101] Input the lane image dataset into the clustering algorithm, set the clustering target and number of clusters, and initialize the cluster centers. The clustering algorithm can be set to K-means, K-medoids, or other clustering algorithms.

[0102] A clustering algorithm is executed to update the cluster centers and output the lane line priors. By performing in-depth cluster analysis on a large-scale lane line image dataset, the best lane line priors can be extracted from a large number of lane line image samples.

[0103] It's important to note that the lane image dataset used for cluster analysis includes curved lanes, straight lines, and other complex shapes. The lane priors derived from clustering also include curved lanes, straight lines, and other complex shapes. These priors encompass a variety of possible lane features, more closely resembling real-world road conditions and providing a more accurate initialization template for subsequent lane detection tasks.

[0104] Reference Figure 7The figure shows a schematic diagram of lane line priors. When selecting lane line priors, it's important to prioritize diversity and coverage. A diverse set of lane line priors can cover all possible lane line situations on real roads, while also encompassing a wide range of areas where lane lines might appear. After selecting such lane line priors, regardless of the actual lane line's position on the road or its shape, the lane line priors provide a reference template that matches the target image's actual lane line. This approach significantly improves lane line detection accuracy and robustness in complex road environments and with varying lane line shapes.

[0105] Next, the lane prior extraction process is based on statistical analysis of large-scale data, rather than relying solely on manual definition. Clustering algorithms leverage the natural distribution and patterns in the data, avoiding the biases and limitations that can arise from manual definition. The lane priors extracted in this way are more adaptable to lane characteristics in diverse scenarios, providing a foundation for lane detection models.

[0106] Specifically, after obtaining the prior lane line, step S300 includes:

[0107] Reference Figure 8 As shown in Figure 1, the prior lane lines are located in the fused feature map P, and the region of interest is delineated around the positioning results.

[0108] Extracting features closely related to lane detection in the region of interest helps the model better understand the lane's shape, orientation, and relationship to other traffic elements. For example, extracting features around a priori lane lines can more accurately determine their continuity, whether they are occluded, and their relative position to other lane lines.

[0109] The vectors corresponding to the features extracted from the region of interest are concatenated, and then the dimension is compressed using a fully connected network to obtain an intermediate vector x, the dimension of which is C.

[0110] The attention calculation is performed on the fused feature map P of size H×W×C through the intermediate vector x: the intermediate vector x needs to be inner-producted with each pixel channel vector in the fused feature map P and normalized. The attention weight matrix is ​​obtained by referring to the following formula.

[0111]

[0112] In the formula, weight ij is the attention weight, representing the pixel channel vector P in the fusion feature map P ij The importance of lane line features. C is the number of channels of the fusion feature map P. ij is the pixel channel vector P at the (i, j) position in the intermediate vector x and the fusion feature map Pij The dot product of is used to measure the similarity between the two. Used to scale the inner product results to avoid gradient vanishing. Softmax(·) is a normalization calculation that converts the similarities of all positions into weights in the range [0, 1], and the sum of all weights is 1.

[0113] Next, the global image information is fused according to the attention matrix. The element weight in the attention weight matrix is ij Pixel feature P corresponding to the fusion feature map P ij Perform weighted summation and refer to the following formula to obtain the lane line feature x1.

[0114]

[0115] Lane feature x1 is the lane feature after integrating the global features. Step S300 utilizes the rich hierarchical information in the fused feature map P and further improves the pertinence and effectiveness of the features by focusing on the local area around the prior lane lines.

[0116] Step S400 , processing the lane line feature x1 to output a posterior lane line, and post-processing the posterior lane line to obtain a final lane line as a lane line detection result.

[0117] Reference Figure 9 As shown, as an implementation method, step S400 includes:

[0118] Step S410: Process the lane line feature x1 to output the posterior lane line.

[0119] The output of step S410 is generated by the output layer. This layer converts the previously extracted lane features x1 into preliminary lane predictions, known as posterior lane lines. These predictions are generated based on the features learned by the network and prior knowledge. They contain key information such as the lane's position and shape, but may contain some inaccuracies or incompleteness.

[0120] Specifically, the lane line feature x1 is input into the first fully connected layer linear1 for feature transformation and activated using the activation function relu(·) to obtain relu(linear1(x1)).

[0121] The activated feature relu(linear1(x1)) is input into the second fully connected layer linear2 for feature transformation and activated again using the activation function relu(·) to obtain the intermediate feature x2=relu(linear2(relu(linear1(x1)))).

[0122] The intermediate feature x2 is input into the third fully connected layer linear3, and the confidence of the lane line is output as logits=linear3(x2); where the confidence corresponds one-to-one to the posterior lane line.

[0123] The intermediate feature x2 is input into the fourth fully connected layer linear4, and the lane line horizontal coordinate xs=linear4(x2) is output.

[0124] The sequence coordinates of the posterior lane line are obtained according to the lane line horizontal coordinate xs and the lane line vertical coordinate ys, xy = concat (xs, ys). The lane line vertical coordinate ys is preset at equal intervals between 1 and 0. In the first embodiment of the present invention, the vertical coordinates of the posterior lane line are set to 36 points at equal intervals between 1 and 0, that is,

[0125] The posterior lane line coordinate sequence xy is a sequence of points starting from the edge of the image. Each posterior lane line is assigned a corresponding confidence score. The output of the posterior lane line is the sequence of posterior lane line coordinates xy and the corresponding confidence score logits.

[0126] The confidence level reflects the reliability assessment of each lane line. A higher confidence level indicates greater confidence in the model's detection of that lane line. The confidence level is typically calculated based on a variety of factors, including the lane line's characteristic strength, its match with the prior lane line, and the consistency of the surrounding environment.

[0127] The output layer abandons the cross-layer refinement method, further reducing the computational complexity and thus significantly reducing the model's inference latency.

[0128] In step S420 , the posterior lane line is post-processed to obtain the final lane line as the lane line detection result.

[0129] Specifically, the lane lines are eliminated and sorted according to their sequence coordinates xy, and valid lane lines are retained.

[0130] After obtaining all valid posterior lane lines and their corresponding confidence scores, the non-maximum suppression (NMS) algorithm is used to process the valid posterior lane lines. First, all valid posterior lane lines are sorted by confidence score. Starting with the posterior lane line with the highest confidence score, the spatial relationship between that posterior lane line and the other lane lines is examined one by one. If two lane lines overlap or are very close in space, they are considered duplicate detections of the same lane line. In this case, only the lane line with the higher confidence score is retained, while the lane line with the lower confidence score is suppressed.

[0131] Finally, the non-maximum suppression algorithm outputs the most confident posterior lane lines based on the confidence scores and removes redundant posterior lane lines. Through this post-processing step, the posterior lane lines are further optimized and corrected, and the output posterior lane lines serve as the final lane lines, producing more accurate and reliable lane detection results.

[0132] The enhanced backbone network provided in the first embodiment of the present invention implements a lane line detection method, and performs pre-training and clustering in large-scale lane line image data to obtain curved prior lane lines. Compared with manually defined straight lane lines, it can provide a better initialization effect and fit the actual road conditions. A C3k2 enhanced SPPF backbone network is set. Compared with other common backbone networks, when the same number of layers is reached, the required number of parameters and computational complexity are significantly reduced. The fast spatial pyramid pooling layer SPPF enhances the feature expression capability with the help of the attention layer. The method provided in the first embodiment also abandons the cross-layer refinement method, further reducing the computational complexity, thereby greatly reducing the inference delay of the model. The present invention can improve the accuracy of lane line detection and reduce the amount of calculation and memory usage, which is beneficial to the smooth operation of the intelligent assisted driving system of the vehicle.

[0133] Embodiment 2 of the present invention also provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a computer processor, is used to enable the computer to execute the enhanced backbone network real-time lane line detection method provided in the above embodiment 1.

[0134] Embodiment 3 of the present invention also provides a computer program product, including a computer program, wherein the computer program, when executed by a computer processor, is used to enable the computer to execute the enhanced backbone network real-time lane line detection method provided in the above-mentioned embodiment 1.

[0135] Embodiment 4 of the present invention also provides an electronic device, comprising: a processor, and a memory for storing a program, wherein the program comprises instructions, which, when executed by the processor, enable the processor to execute the enhanced backbone network real-time lane line detection method provided in the above-mentioned embodiment 1.

[0136] refer to Figure 10, a structural block diagram of an electronic device that can be used as a server or client of an embodiment of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0137] like Figure 10 As shown, the electronic device includes a computing unit 101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 102 or a computer program loaded from a storage unit 108 into a random access memory (RAM) 103. Various programs and data required for the operation of the electronic device can also be stored in the RAM 103. The computing unit 101, the ROM 102, and the RAM 103 are connected to each other via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0138] Multiple components in the electronic device are connected to the I / O interface 105, including: an input unit 106, an output unit 107, a storage unit 108, and a communication unit 109. The input unit 106 can be any type of device capable of inputting information into the electronic device. The input unit 106 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 107 can be any type of device capable of presenting information and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 108 can include, but is not limited to, a magnetic disk and an optical disk. The communication unit 109 allows the electronic device to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and can include, but is not limited to, a modem, a network card, an infrared communication device, and / or a wireless communication transceiver, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0139] The computing unit 101 can be various general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 101 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing units, various computing units for running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 101 performs the various methods and processes described above. For example, in some embodiments, the method embodiments created by the present invention can be implemented as a computer program, which is tangibly contained in a machine-readable medium, such as a storage unit 108. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 102 and / or communication unit 109. In some embodiments, the computing unit 101 can be configured to perform the above-mentioned method in any other appropriate manner (e.g., by means of firmware).

[0140] The computer programs for implementing the methods of the embodiments of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0141] In the context of the present invention, a machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic or infrared systems, devices or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device or any suitable combination of the foregoing.

[0142] It should be noted that the term "comprising" and its derivations, as used in the embodiments of the present invention, are intended to be open-ended. The term "comprising" and its derivations are used in the sense of "including but not limited to". The term "based on" is intended to mean "based, at least in part, on". The term "one embodiment" is intended to mean "at least one embodiment". The term "another embodiment" is intended to mean "at least one additional embodiment". The term "some embodiments" is intended to mean "at least some embodiments". The terms "a" and "an", as used in the embodiments of the present invention, are intended to be interpreted to mean "one or more" unless otherwise indicated. The terms "the" and "the" preceding article are intended to be interpreted to mean "the one and only one" unless otherwise indicated by context.

[0143] The user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and corresponding operation portals are provided for the user to choose authorization or rejection.

[0144] The steps described in the method embodiments provided by the embodiments of the present invention can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of protection of the present invention is not limited in this respect.

[0145] The word "embodiment" in the specification refers to the specific features, structures or characteristics described in connection with the embodiment can be included in at least one embodiment of the present invention. The presence of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean independence or alternative to other embodiments. Each embodiment in the specification is described in a relevant manner, and the same or similar parts between each embodiment are referred to each other. In particular, for device, equipment, system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts are referred to the part of the method embodiment.

[0146] The above-described embodiments only express several embodiments of the present invention, and the description is more specific and detailed, but it cannot be understood as a limitation on the scope of protection. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present invention, a number of modifications and improvements can be made, which are within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be subject to the appended claims.

Claims

1. An enhanced backbone network real-time lane line detection method, characterized in that: include: Inputting the target image to be detected into the enhanced backbone network structure to extract image features and obtain first high-level features; wherein the enhanced backbone network structure also outputs first low-level features and first middle-level features; Input the first high-level feature into the feature pyramid for upsampling and sequentially fuse it with the first middle-level feature and the first low-level feature to obtain a fused feature map; Extracting feature information around the prior lane line from the fused feature map and calculating the lane line feature; wherein the prior lane line is obtained by clustering analysis of a lane line image dataset, and the prior lane line includes a curved lane line; The lane line features are processed to output a posterior lane line, and the posterior lane line is post-processed to obtain a final lane line as a lane line detection result.

2. The enhanced backbone network real-time lane line detection method according to claim 1, characterized in that: The target image to be detected is input into the enhanced backbone network structure for image feature extraction, and the first high-level features obtained include: Input the target image into the convolution operation layer of the enhanced backbone network for convolution processing, and output a convolution feature map; Inputting the convolutional feature map into the C3k2 module group of the enhanced backbone network, and sequentially outputting the first low-level features and the first mid-level features; wherein the C3k2 module group includes five C3k2 modules arranged in series; Inputting the first middle-level features into the fast spatial pyramid pooling layer for pooling to obtain a pooled output feature map; The pooled output feature map is input into the feature optimization layer to output the first high-level feature.

3. The enhanced backbone network real-time lane line detection method according to claim 2, characterized in that: Inputting the convolutional feature map into the C3k2 module group of the enhanced backbone network, and sequentially outputting the first low-level features and the first mid-level features includes: The convolution feature map is used as the input of the C3k2 module, and is convolved and divided into a first branch and a second branch; wherein the number of channels of the first branch and the second branch is the same; When the C3k parameter is true, the first tributary is processed through multiple C3k layers and then feature-joined with the second tributary; when the C3k parameter is not true, the first tributary is processed through a bottleneck structure and then feature-joined with the second tributary; The result of splicing the second tributary with the processed first tributary features is convolutionally fused to obtain the output of the C3k2 module; wherein the output of each C3k2 module is set as the input of the next C3k2 module; the output of the third C3k2 module arranged in series is set as the first low-level feature, and the output of the fifth C3k2 module arranged in series is set as the first middle-level feature.

4. The enhanced backbone network real-time lane detection method according to claim 2, characterized in that: The first middle-level features are input into the fast spatial pyramid pooling layer for pooling, and the obtained pooled output feature map includes: Convolution processing is performed on the first middle-layer features, and a multi-scale maximum pooling operation is performed on the convolution result, and pooling results of different sizes are obtained corresponding to pooling kernels of different sizes; wherein the pooling kernel sizes for the multi-scale maximum pooling operation include 1×1, 3×3, 5×5, and 7×7; The pooling result is restored to its size by upsampling and then element-by-element addition is performed to obtain the pooling output feature map.

5. The enhanced backbone network real-time lane detection method according to claim 1, characterized in that: The first high-level feature is input into the feature pyramid, and is sequentially fused with the first middle-level feature and the first low-level feature to obtain a fused feature map, including: Input the first high-level features into the feature pyramid and perform upsampling to obtain the second middle-level features; Fusing the second mid-level features with the first mid-level features to obtain a mid-level fusion result; wherein the resolution of the second mid-level features is greater than the resolution of the first high-level features; Upsampling the middle-level fusion result in the feature pyramid to obtain a second low-level feature; The second low-level features are fused with the first low-level features to obtain the fused feature map; wherein the resolution of the second low-level features is greater than the resolution of the second middle-level features.

6. The enhanced backbone network real-time lane detection method according to claim 1, characterized in that: Before extracting feature information around the prior lane line from the fused feature map, the method further includes: Collecting lane line image data, and annotating the lane line image data to obtain the lane line image dataset; Input the lane line image dataset into the clustering algorithm, set the clustering target and the number of clusters, and initialize the cluster centers; Execute a clustering algorithm to update the cluster center and output the prior lane line.

7. The enhanced backbone network real-time lane detection method according to claim 1, characterized in that: Extract feature information around the prior lane line from the fused feature map, and calculate the lane line features including: Positioning the a priori lane line in the fused feature map, and defining a region of interest around the positioning result; Extracting features from the region of interest and performing feature compression to obtain an intermediate vector; Performing attention calculation on the fused feature map using the intermediate vector to obtain an attention weight matrix; The elements in the attention weight matrix corresponding to the pixel features of the fusion feature map are weightedly summed to obtain the lane line feature.

8. The enhanced backbone network real-time lane detection method according to claim 1, characterized in that: Processing the lane line features to output a posterior lane line includes: Input the lane line features into the first fully connected layer for feature transformation and activation using an activation function; The activated features are input into the second fully connected layer for feature transformation and activated again using the activation function to obtain intermediate features; Inputting the intermediate features into a third fully connected layer and outputting a confidence score; wherein the confidence score corresponds one-to-one to the posterior lane line; Input the intermediate features into the fourth fully connected layer and output the horizontal coordinate of the lane line; Obtaining a sequence coordinate of the a posteriori lane line according to the lane line horizontal coordinate and the lane line vertical coordinate; wherein the lane line vertical coordinate is preset at equal intervals within a range of 1 to 0; The sequence coordinates of the posterior lane line and the confidence level corresponding to the posterior lane line are used as the output result of the posterior lane line.

9. The enhanced backbone network real-time lane detection method according to claim 8, characterized in that: Post-processing the posterior lane line to obtain a final lane line as a lane line detection result includes: Eliminate and sort the lane lines according to the sequence coordinates of the posterior lane lines, and retain the valid posterior lane lines; Processing valid posterior lane lines using a non-maximum suppression algorithm, outputting the posterior lane line with the highest confidence and removing the other posterior lane lines; The posterior lane line output by the non-maximum suppression algorithm is used as the final lane line, and the lane line detection result is output.

10. An electronic device comprising: A processor and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the enhanced backbone network real-time lane line detection method according to any one of claims 1 to 9.

Images