Lane line detection method based on SDAMamba and FFCDConv

Abstract

The invention discloses a lane line detection method based on SDAMamba and FFCDConv, and relates to the field of lane line detection. The method comprises the following steps: preprocessing a training set lane line image to obtain an updated training set; improving the DAMamba model to obtain an SDAMamba model for extracting multi-scale features of the updated training set image; extracting depth multi-scale features by using a feature pyramid neck network; mapping the depth multi-scale feature to a Hough parameter space by using DHT and selecting a Hough feature corresponding to a lane; jointly inputting the Huffcharacteristics and the depth multi-scale characteristics into an FFCDConv module, predicting the characteristics of different lane instances, and obtaining a predicted lane position map through a lane decoder; and inputting the predicted lane position map into a linear layer to determine a final lane position, and obtaining a lane line detection image. According to the method, the problem of breakage easily occurring in a lane line detection result can be repaired in a targeted manner, and the confusion rate of adjacent lanes is remarkably reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of lane line detection, and particularly relates to a lane line detection method based on SDAMamba (Sobel feature-guided Dynamic Adaptive Mamba) and FFCDConv (Fast Fourier Convolution and Dynamic Convolution). BACKGROUND

[0002] Lane line detection is a core perception technology of an automatic driving system, and its core function is to accurately identify lane lines (including solid lines, dashed lines, edge lines, etc.) in a road scene, so as to provide a bottom layer position reference for a lane keeping assistant, a lane departure warning, an automatic lane changing decision and other key control modules of a vehicle. The reliability of the technology directly determines the judgment accuracy of the automatic driving system on the driving boundary, and thus becomes a key factor affecting the driving safety of the vehicle.

[0003] At present, the mainstream technical path for realizing lane line detection in the industry is to analyze and process road images collected by a vehicle-mounted camera based on a deep learning model. Although this path can achieve high detection accuracy under normal road conditions (such as sunny days, straight lanes and non-sheltered scenes), there are still significant shortcomings in complex real road conditions (such as lane line wear, foreign object sheltering, extreme light or bad weather).

[0004] Firstly, in the selection of model architecture, in recent years, the visual Mamba model as a new architecture, with its core mechanism of selective state space model and image scanning strategy, has shown its potential in processing long sequence data in the field of computer vision. Its linear computational complexity is theoretically very suitable for processing high-resolution images and long-distance spatial dependence, such as modeling global context information of curved lane lines. However, the scanning mechanism of the visual Mamba is essentially to expand the image into a one-dimensional sequence for processing, which may weaken the inherent two-dimensional spatial relationship between pixels. This weakening of structural perception easily leads to two major problems: first, in the wear or sheltered area (such as the lane section sheltered by the shadow of the preceding vehicle or obstacles), the model is difficult to use the complete surrounding context for repair, resulting in a "break" in the detection result, which cannot form a complete lane contour; second, the model's ability to distinguish similar texture areas (such as adjacent lane lines and road stains) may decrease, as the sequence processing may blur the difference in their two-dimensional spatial distribution, leading to an increase in false detection rate (mistaking non-lane line areas as lane lines), which in turn affects the accuracy of subsequent driving decisions.

[0005] In the scene of multi-lane parallel, curve intersection or dense lane line (such as highway entrance, city roundabout), another core limitation of the prior art is that the image frequency domain information is not fully utilized to enhance the differentiation of the lane boundary. From the perspective of signal processing, the lane line appears as "alternating linear structure of light and dark" in the image, and its characteristics in the frequency domain mainly correspond to high-frequency components — these high-frequency information can strengthen the detail difference of the lane edge and help the model clearly distinguish the boundaries of adjacent lanes (such as the distance between two parallel lane lines or the curvature change of curved lane). However, the design of the current mainstream deep learning model still takes spatial domain convolution operation as the core: by sliding the convolution kernel in the spatial dimension (length, width) of the image to extract features, it ignores the value of the high-frequency features of the lane line in the frequency domain. This "emphasis on space and neglect of frequency domain" technical solution makes it difficult for the model to accurately distinguish the boundaries of adjacent lane lines (especially when the distance between them is close or there is slight deformation), and thus prone to "mistaking two adjacent lanes as a single lane instance", which directly threatens the safety of lane keeping for vehicles. SUMMARY

[0006] In view of the above shortcomings of the prior art, the present application provides a lane line detection method based on SDAMamba and FFCDConv.

[0007] The technical scheme provided by the present application is: A lane line detection method based on SDAMamba and FFCDConv, the method comprising the following steps: Step 1: obtaining a training set for lane line detection; Step 2: sequentially downsampling and image enhancing the image samples in the training set to obtain an updated training set; Step 3: improving the DAMamba Block in the existing DAMamba model, the improved DAMamba Block is named as SDAMamba Block, and the DAMamba model after the improvement of the DAMamba Block is called SDAMamba model; Step 4: taking the SDAMamba model as the backbone network, inputting the image samples in the updated training set into the backbone network to obtain multi-scale feature maps ; Step 5: taking the existing feature pyramid network as the neck network, inputting the multi-scale feature maps output by the SDAMamba into the neck network according to the order of , , , , , in turn to perform feature fusion and obtain deep multi-scale feature maps ; Step 6: Utilize the feature map and the updated image samples in the training set to determine the binary cross-entropy loss function , based on training the backbone network; Step 7: Utilize the existing deep Hough transform algorithm to map the deep multi-scale feature map output in Step 5 into the Hough parameter space, and then select the Hough features of each lane ; Step 8: Jointly input the Hough features and the feature map into the FFCDConv module to obtain features for different lane instances ; Step 9: obtain the predicted lane position map through the lane decoder; Step 10: Input the predicted lane position map into the linear layer to determine the final lane position, thereby obtaining the lane line detection image.

[0008] Further, according to the lane line detection method, the DAMambaBlock in the existing DAMamba model is improved, including: 1) The DAS module is improved, and the improved DAS module is referred to as an SDAS module; 2) The convolution feedforward network and the batch normalization layer in the existing DAMamba Block are deleted.

[0009] Further, according to the lane line detection method, the improvement of the DAS module is: introducing a Sobel edge detection algorithm to extract lane line edge features from the original feature map input into the DAS module offset prediction network, generating an edge feature map, and performing weighted average fusion on the edge feature map and the original feature map input into the DAS module offset prediction network, forming an enhanced feature map with geometric prior, and taking the enhanced feature map as the input of the offset prediction network.

[0010] Further, according to the lane line detection method, the binary cross-entropy loss function is determined by utilizing the feature map and the updated image samples in the training set , including: The feature map is decoded into a predicted lane probability map with the same size as the image samples in the updated training set through multi-layer convolution , specifically: the first layer of transposed convolution realizes 2 times upsampling while reducing the number of channels; the second layer of bilinear interpolation realizes fine upsampling; and the third layer of convolution generates ; The binary cross-entropy loss function is determined according to formula (1) ; (1) Wherein represents a ground truth lane probability map.

[0011] Further, according to the lane line detection method, the step 7 specifically comprises: Step 7.1: interpolating the deep multi-scale feature map output by step 5 to obtain Euclidean space features ; Step 7.2: using a deep Hough transform algorithm to map to a Hough parameter space to obtain Hough parameter features ; Step 7.3: aggregating the Hough parameter features obtained in step 7.2, and the aggregated Hough parameter features are denoted as G, and G is decoded to generate a Hough map; Step 7.4: selecting Hough features of each lane based on the Hough map . Further, according to the lane line detection method, the Hough features of each lane selected based on the Hough map in step 7.4 are located by performing non-maximum suppression and threshold filtering on the Hough map

[0012] , specifically: first, maximum pooling is applied to the Hough map to suppress non-maximum pixels, and then points with probability values exceeding a threshold value are retained as , the threshold value is set to 0.1, wherein the value of each pixel point on the Hough map represents the predicted probability that the point exists as a lane instance.

[0013] Further, according to the lane line detection method, the FFCDConv module is composed of a double-branch dynamic convolution and a feature fusion module in cascade; the double-branch dynamic convolution is composed of a frequency domain dynamic convolution of a global branch and a spatial domain dynamic convolution of a local branch in parallel; the frequency domain dynamic convolution of the global branch and the spatial domain dynamic convolution of the local branch both adopt a series structure design, wherein the frequency domain dynamic convolution of the global branch is composed of a global branch of the FFC and a first dynamic convolution module in cascade, and the spatial domain dynamic convolution of the local branch is composed of a local branch of the FFC and a second dynamic convolution module in cascade.

[0014] Further, according to the lane line detection method, the Hough features are combined with the feature map​​ The joint input into the FFCDConv module obtains features for different lane instances The process is as follows: Firstly, the input data is divided into data sent to the local branch and data sent to the global branch along the dimension of the feature channel , wherein represents the percentage of feature channels allocated to the global branch; then, the lane instance features with local details are obtained through two parallel convolution layers in the FFC local branch and the second dynamic convolution module and sent to the feature fusion module, the lane instance features with global context are obtained through parallel signal transformers and convolution layers in the FFC global branch and the first dynamic convolution module and sent to the feature fusion module, wherein the signal transformer uses 1x1 convolution and RELU to reduce the number of data channels; the kernel parameters of the 2D convolution layer in the dynamic convolution module are fed to the multilayer perceptron The processing process of the feature fusion module is as follows: the input data of the module is bilinearly interpolated to align to the spatial dimension ; then, a lightweight convolution is used to respectively calibrate the channel dimensions of the two aligned features to 0.75C and 0.25C; finally, the two aligned and calibrated features are spliced in the channel dimension, and the final lane instance features are output, wherein N is the total number of lanes, represents the lane.

[0015] Further, according to the lane line detection method, the lane decoder is composed of transposed convolution and up-sampling.

[0016] Further, according to the lane line detection method, the method for determining the final lane position by inputting the predicted lane position map into the linear layer in step 10 is as follows: firstly, the row features of the predicted lane position map are extracted and input into the linear layer to perform a binary classification operation to determine whether each row has a lane; then, the set of consecutive rows determined to have a lane is defined as the predicted vertical range, and the y coordinate of each lane can be determined by the vertical range and the image height H of the image sample in the training set , wherein H represents the total height of the image, represents the height of the image as the ​​​​​Finally, the softmax layer is applied to the row features in the vertical range to obtain the x coordinate of each lane, and the row features outside the vertical range are filled with default values to finally determine the lane position.

[0017] Compared with the prior art, the present application has the following beneficial effects: (1) The existing visual Mamba still has the problems of information transmission error or poor effect when performing preliminary multi-scale feature extraction on the input lane line image under complex road conditions such as lane line wear, occlusion and short distance between adjacent lane lines. The present application designs an SDAMamba model, the core of which is to improve the dynamic adaptive scanning module based on the existing DAMamba model. The improved dynamic adaptive scanning module is named as SDAS module, and the SDAS (Sobel feature-guided Dynamic Adaptive Scan) module innovatively introduces Sobel edge features as geometric priors, which are deeply integrated with the dynamic scanning mechanism, so that the module can accurately locate and strengthen the features of the key lane line region which is worn or occluded in the input feature map, thereby targetedly repairing the "breakage" problem prone to occur in the lane line detection result; (2) The present application uses FFC (Fast Fourier Convolution) combined with the existing dynamic convolution (Dynamic Convolution) module to build a double-branch dynamic convolution structure to enhance its ability to distinguish different lane instance features, in order to solve the problem of adjacent lane confusion in lane line detection: the frequency domain dynamic convolution of the global branch efficiently models the long-distance spatial dependence of the lane line, thereby strengthening the semantic distinction degree of different lane instances; the spatial domain dynamic convolution of the local branch captures the local detail geometric features of the lane boundary, providing pixel-level information for accurate positioning of each lane instance. The FFC DConv module expands the receptive field of the dynamic convolution, so that it can not only rely on the dynamic kernel to adaptively extract features, but also can simultaneously consider the global context and local details, thereby significantly reducing the confusion rate of adjacent lanes. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 The flowchart of the lane line detection method of the present embodiment based on SDAMamba and FFC DConv; Figure 2 The lane line detection model structure diagram of the present embodiment based on SDAMamba and FFC DConv; Figure 3 The network structure diagram of the SDAMamba model of the present embodiment; Figure 4is a structural schematic diagram of the existing DAMamba Block; Figure 5 is a structural schematic diagram of the SDAMamba Block of the present embodiment; Figure 6 is a structural schematic diagram of the FFCDConv of the present embodiment; Figure 7 is a comparison diagram of the visualization results of three existing lane line detection methods and the lane line detection method of the present application on the CULane dataset, and a ground truth diagram of the CULane dataset; wherein (a) is a result diagram of the existing Ultrafast lane line detection method, (b) is a result diagram of the existing LaneATT lane line detection method, (c) is a result diagram of the existing SGNet lane line detection method, (d) is a result diagram of the lane line detection method of the present application, and (e) is a ground truth diagram of the CULane dataset. DETAILED DESCRIPTION

[0019] In order to facilitate the understanding of the present application, the present application will be described more fully below with reference to the relevant drawings.

[0020] Figure 1 is a flowchart of the lane line detection method of the present embodiment based on SDAMamba and FFCDConv. As shown in Figure 1 and Figure 2 , the lane line detection method of the SDAMamba and FFCDConv includes the following steps: Step 1: Obtain a public dataset for the field of lane line detection, and the public dataset has been divided into a training set, a test set and a validation set; The public dataset obtained in the present embodiment is a large-scale lane detection dataset CULane, which contains a total of 133,235 labeled images, and the training set and the test set constituting the public lane line dataset CULane are obtained.

[0021] The CULane dataset covers lane line images in various road environments, including urban roads, highways and rural roads. At the same time, the dataset contains lane line images in various scenes, such as lane lines partially blocked by vehicles, curved lane lines, lane lines in poor light conditions at night and lane lines under shadows, and the dataset fully covers various complex road conditions that may be encountered in actual applications. The CULane dataset has a total of 133,235 images, of which 88,880 are training set images, 34,680 are test set images, and 9,675 are validation set images.

[0022] Step 2: sequentially downsample and image enhance the image samples in the training set to obtain an updated training set; In the present embodiment, the image samples in the training set are defined as where C=3 represents the RGB channel, and H and W represent the height and width of the image, respectively. All images in the training set are down-sampled to 360x640 in this step. Random transformations are applied to the down-sampled images for image augmentation, including RGB shift, hue saturation, blur, color jitter, and random brightness change, to simulate real-world lighting changes, complex conditions of blur. The image size after image augmentation is still 360x640.

[0023] Step 3: Improvement is made to the DAMamba Block in the existing DAMamba model, the improved DAMamba Block is named as SDAMamba Block, and the DAMamba model after the improvement of the DAMamba Block is called SDAMamba model. The DAMamba Block in the existing DAMamba model is improved in this embodiment, and the specific improvements include: 1. The DAS module is improved, and the improved DAS module is called SDAS module; the improvement of the DAS module in this embodiment is: considering that the lane line has obvious edge features, a simple and efficient Sobel edge detection algorithm is introduced to extract lane line edge features from the original feature map of the offset prediction network in the input DAS (Dynamic Adaptive Scan) module, to generate an edge feature map, and to perform weighted average fusion of the edge feature map and the original feature map of the offset prediction network in the input DAS module, to form an enhanced feature map with geometric prior, the size of which remains consistent with that of the input feature map, so as to take the enhanced feature map as the input of the offset prediction network. That is, the Sobel edge detection algorithm is introduced to constrain the scanning process of the DAS module with geometric prior features.

[0024] The DAS module after the improvement of the Sobel edge detection algorithm and the weighted fusion is called SDAS (Sobel feature-guided Dynamic Adaptive Scan) module.

[0025] 2、Delete the convolutional feedforward network and batch normalization layer in the existing DAMamba Block. The main reason for deleting the convolutional feedforward network is that the convolutional feedforward network enhances the function of detail perception through local convolution operation, which can be effectively replaced by the SDAS module by injecting strong geometric prior from the lane line edge feature extracted by the Sobel edge detection algorithm. This prior can more directly and effectively strengthen the local structural features of the lane line. At the same time, the batch normalization layer is mainly used to stabilize the feature distribution and accelerate the training convergence, and part of its function can also be compensated by the dynamic scanning mechanism of the SDAS module itself, which has the ability of adaptive feature calibration and can maintain the stability of the distribution to a certain extent. In addition, removing these two layers of structure also significantly reduces the parameter quantity and computational overhead of the model, which can improve the computational efficiency.

[0026] The improved DAMamba Block is named SDAMamba Block, and the improved DAMamba model is called SDAMamba model.

[0027] Figure 3 The overall framework of the SDAMamba model is shown, which is the same as the existing DAMamba model. SDAMamba is also composed of a stem module and a four-stage hierarchical architecture. Each of the four levels of stages is stacked by multiple SDAMamba Blocks. Figure 4 The structure diagram of the existing DAMamba Block is shown, Figure 5 The structure diagram of the SDAMamba Block of the present embodiment is shown. As Figure 5 shown, the SDAMamba Block extracts and transmits features through depth separable convolution, layer normalization, SDAS module, selective structured state space module and residual connection, and its core is the SDAS module. The SDAS module realizes dynamic focusing and adaptive relationship modeling of important regions in the input feature map of the module by introducing the edge geometric prior of the lane line and using the deformable convolution mechanism.

[0028] The specific process in the SDAS module is as follows: first, the Sobel edge detection algorithm is used to extract the lane line edge features in the input feature map of the SDAS module to generate an edge feature map; then, the edge feature map is weighted and fused with the original input feature map of the SDAS module to form an enhanced feature map with fused geometric priors, which has the same size as the input feature map; subsequently, the fused feature map is input into an offset prediction network (OPN) to generate offset amounts with geometric prior constraints for the position of each image block of the original input feature map of the SDAS module, and these offset amounts directly define the dynamic sampling positions of the deformable convolution kernel, so that the convolution kernel can adaptively adjust its receptive field according to the image content; in the following deformable convolution process, the offset amounts are combined with the original image block positions to calculate the dynamically adjusted sampling positions, the feature values at these positions are obtained on the input feature map through bilinear interpolation, and the deformable convolution operation is completed, so that more flexible spatial features can be captured.

[0029] Step 4: Taking the SDAMamba model as the backbone network, inputting the image samples in the updated training set into the backbone network to obtain a multi-scale feature map ; In this embodiment, the SDAMamba model is taken as the backbone network, and the image samples in the updated training set are input into the SDAMamba model. The processing process in the SDAMamba model is as follows: first, an initial feature map with a resolution of is generated by a stem module composed of 4 layers of overlapping 3×3 convolution ; subsequently , four levels are entered, each level is stacked by a plurality of SDAMamba blocks, the specific number is set to [2, 2, 8, 2], and the resolution of is sequentially reduced to by downsampling operation between the levels of the SDAMamba, while the number of channels is increased, and the specific number of channels is set to [64, 128, 256, 512].

[0030] The processing process of the first SDAMamba block in the first level of the SDAMamba is as follows: First, a lightweight spatial feature extraction is performed by a depth separable convolution module to obtain , After residual connection, we get , Layer normalization is performed to obtain , Subsequently, the core SDAS module is entered; in the SDAS module, first, the original is divided into blocks, specifically, the original Implicit partitioning Each non-overlapping tile corresponds to a grid cell, and the original position of the tile is... Mapping to normalized coordinate system Simultaneously, edge features of each patch are extracted using the Sobel operator. Then, the edge features of each patch are... The weighted average of the original tiles is then used to fuse them. Following that, OPN from Learn the 2D coordinate offset of the tile of interest relative to q from all tiles. , The adaptive sampling position is obtained by superimposing the original position. Deformable convolution utilizes right Dynamic feature extraction is performed, and bilinear interpolation is used to smoothly sample features at non-integer positions; the sampled features are then processed according to... The original tiles are arranged in a spatial order, from left to right and from top to bottom, forming a one-dimensional sequence. ; Feature extraction is performed in the input selective structured state space module, and the output sequence is ordered accordingly. The spatial order of the original tiles is reorganized into a two-dimensional feature map. ; The output characteristics of this SDAMamba Block are obtained through residual connection. The second SDAMamba Block in the first level of SDAMamba repeats the processing of the first SDAMamba Block to obtain the feature map of the first level of SDAMamba. Similarly, the remaining three stages of SDAMamba yield three different semantic levels of multi-scale feature maps. ,in The hierarchical feature map representing the lowest semantic level. The hierarchical feature map represents the highest semantic level.

[0031] Step 5: Using the existing feature pyramid network as the neck network, process the multi-scale feature maps output by SDAMamba. Simultaneously input into the neck network according to , , , Feature fusion is performed sequentially to obtain a deep multi-scale feature map. ; The 4-layer multi-scale feature map output by SDAMamba Meanwhile, the input is input into the feature pyramid network for feature fusion. First, the feature pyramid network is used to obtain a feature map by self-convolution . Then, F4 is doubled up-sampled and added to to obtain a feature map . In this way, the remaining three layers of deep multi-scale feature maps are obtained in turn. Among them, represents the lowest resolution level fusion feature map, represents the highest resolution level fusion feature map. That is, the feature pyramid network finally outputs a deep multi-scale feature map . In this embodiment, the input channel number of the feature pyramid network is set to [64, 128, 256, 512], and the output channel is set to 128.

[0032] Step 6: Determine the binary cross-entropy loss function using the feature map and the updated image samples in the training set, and train the backbone network based on ; In order to enable the SDAMamba model to quickly distinguish the pixel points corresponding to the lane in the input lane image, a constraint is imposed on . In this embodiment, the (90×160) is decoded into a predicted lane probability map (360×640) of the same size as the image samples in the updated training set through multi-layer convolution. Specifically, the first layer of transposed convolution realizes 2 times up-sampling while reducing the number of channels; the second layer of bilinear interpolation realizes fine up-sampling; and the third layer of convolution generates . Then, the binary cross-entropy loss function value is determined according to formula (1), and the binary cross-entropy loss is used to guide the training of the backbone network, i.e., the SDAMamba model.

[0033] (1) wherein represents the ground truth lane probability map. The CULane data set provides accurate coordinate point annotations for each lane. The coordinate point annotations are generated into continuous lane lines through interpolation, and then rasterized into a binary mask map with the same resolution as the original image.

[0034] Step 7: Map the deep multi-scale feature map output in step 5 into the Hough parameter space using the existing deep Hough transform (DHT) algorithm, and then select the Hough feature of each lane ; Step 7.1: Process the deep multi-scale feature map output from Step 5. Interpolation is performed to obtain the Euclidean space features. ; To save computational resources, this implementation only uses the depth multi-scale feature map output in step 5. Mapped into the Hough parameter space, due to the features The depth multi-scale feature map output in step 5 represents the position of the lane in Euclidean space. Perform interpolation and define its interpolated features. Features of Euclidean space.

[0035] Step 7.2: Euclidean space features As input to DHT, using DHT to Mapping to the Hough parameter space yields the Hough parameter features. ; Points in polar coordinate system A straight line in a two-dimensional image can be represented by the Hough transform. The Hough transform formula is as follows: (2) Before DHT, Θ and R are defined as follows: The quantization level of r is Θ = R = 360. DHT uses this quantization method and equation (2) to... Transform to parameter space to obtain the corresponding Hough parameter features. .by Taking DHT as an example, Initialized as a zero matrix, for Given a point (h, w), by iterating through all Θ values, we can obtain the result through equation (2). midpoint The series. DHT will The features of points belonging to the same straight line are superimposed in the parameter space to form A single point feature ultimately completes the entire process. arrive The mapping. Following the same method, Also mapped to via DHT .

[0036] Step 7.3: Characterize the Hough parameters obtained in Step 7.2 The aggregation is performed, and the aggregated Hough parameter features are denoted as G. G is then decoded to generate a Hough map, and focus loss is used. Constrain the Hough map; This step first involves analyzing the Hough parameter characteristics obtained in step 7.2. Upsample to uniform size 360x360 and stitch as Hough parameter feature G. Then decode G by a map decoder composed of convolution and upsampling to generate Hough map with the same resolution as ground truth Hough map. Hough map is a probability distribution map, where each pixel value of the map represents the predicted probability of the point existing lane instance. Ground truth Hough map is used to supervise the generated Hough map, which is constructed in the form of binary parameter space label: connect the equally spaced sampling key points on the bottom of the real lane line labeled on the image samples in the updated training set to line segments, then map the line segments to parameter space by Hough transform, take the mean value of each lane mapping point coordinate as ground truth point, and assign 1 to the corresponding position in the 360x360 matrix, indicating that the lane exists, and assign 0 to the rest of the positions.

[0037] Since the number of positive samples representing real lanes in the Hough map is much less than that of negative samples, the focal loss shown in formula (3) is used to constrain the Hough map.

[0038] (3) where N is the number of lanes; is the weight of controlling positive samples; is the weight of adjusting difficult samples, which refers to samples that are difficult to distinguish as positive or negative samples; represents the true probability value of point .

[0039] Step 7.4: Select Hough feature of each lane based on Hough map , where binary cross-entropy loss is used to guide the selection of ; The present embodiment locates by performing Non-Maximum Suppression (NMS) and threshold filtering on the Hough map. Specifically: first apply max pooling to the Hough map to suppress non-maximal pixels, then retain points with probability value exceeding the threshold as , and the threshold is set to 0.1. The resolution of

[0040] After adding focal loss to constrain the Hough map, ​Since the graph hovers around the ground truth point, the reverse Hough transform (RHT) is used to transform G back to Euclidean space to obtain the probability map of the predicted line. And add binary cross-entropy loss To predict the linear chart.

[0041]

[0042] in The ground truth line probability map is obtained by performing a Relative Human Touch (RHT) on the ground truth points. Through the geometric mapping relationship of the RHT, the linear error in the image space is fed back to the Hough space features. A large value indicates that the map decoder predicts... The inaccurate gradient signal is used to optimize the parameters of the map decoder through backpropagation, improving the map decoder's predictions. It converges towards the ground truth point.

[0043] Step 8: Hough features With feature map The combined inputs are fed into the FFCDConv module to obtain features specific to different lane instances. ; like Figure 6 As shown, the FFCDConv module consists of a cascaded dual-branch dynamic convolution and a feature fusion module. The dual-branch dynamic convolution is composed of a frequency-domain dynamic convolution of the global branch and a spatial-domain dynamic convolution of the local branch connected in parallel. Both the frequency-domain dynamic convolution of the global branch and the spatial-domain dynamic convolution of the local branch adopt a cascaded structure design. Specifically, the frequency-domain dynamic convolution of the global branch is composed of the global branch of FFC and the first dynamic convolution module cascaded sequentially, and the spatial-domain dynamic convolution of the local branch is composed of the local branch of FFC and the second dynamic convolution module cascaded sequentially. When using the FFCDConv module, first... Data is divided into local branches along the dimension of its feature channels. and data sent to the global branch ,in This represents the percentage of feature channels allocated to the global branch, in this embodiment. ;Then, Lane instance features with local details are obtained through two parallel convolutional layers and a second dynamic convolutional module in the local branch of FFC. And send it to the feature fusion module, Lane instance features with global context are obtained through parallel signal transformers and convolutional layers in the FFC global branch, as well as the first dynamic convolutional module. and sent to the feature fusion module, wherein the signal transformer uses 1x1 convolution and RELU to reduce the number of data channels The dynamic convolution module contains a multi-layer perceptron and a 2D convolution layer, and the kernel parameters of the 2D convolution layer are mainly obtained by feeding into the multi-layer perceptron.

[0044] The processing procedure of the feature fusion module is as follows: the input of the module is aligned to the spatial dimension of and by bilinear interpolation, and then the channel dimension of the two aligned features is calibrated to 0.75C and 0.25C respectively by using a lightweight convolution. Finally, the two aligned and calibrated features are spliced in the channel dimension, and the final lane instance feature is output. N is the total number of lanes, and represents the lane.

[0045] Step 9:The predicted lane position map is obtained by passing through the lane decoder, and a position-weighted binary cross-entropy loss function is used to constrain the predicted lane position map. In this embodiment, the lane instance feature is fed into the lane decoder composed of transposed convolution and up-sampling to obtain N predicted lane position sub-maps, and the N predicted lane position sub-maps are spliced together along the channel dimension to obtain the predicted lane position map. The pixel value represents the probability that a point on the predicted lane position map belongs to a lane. The position-weighted binary cross-entropy loss function shown in formula (5) is used to constrain the predicted lane position map.

[0046]

[0047] wherein is the ground truth lane probability map, which is a binary mask generated by interpolating the manually labeled lane point coordinates; represents the positive class weight.

[0048] Step 10: The predicted lane position map is input into a linear layer to determine the final lane position, thereby obtaining a lane line detection image, and a Softmax cross-entropy loss function is used to constrain the process.

[0049] ​In this step, first, the predicted lane position map is extracted for the line features and input into the linear layer to perform a binary classification operation to determine whether each line exists or not; then, the set of consecutive lines determined to exist is defined as the predicted vertical range, and the image height H of the image sample in the training set can be used to determine the y coordinate of each lane , H represents the total height of the image, represents the height of the image in the first line. The softmax layer is applied to the line features in the vertical range to obtain the x coordinate of each lane, and the line features outside the vertical range are filled with default values to finally determine the lane position and obtain the lane line detection image. In this embodiment, Softmax cross-entropy is used as the vertical range loss to supervise the process and ensure the correctness of the lane vertical range.

[0050] In this embodiment, F1-measure is used as a measure of detection performance, and the lane line detection model obtained by the training process of steps 4 to 8 is tested using the test set. The lane line detection model parameter weight file with the best test result performance is saved, and the trained lane line detection model based on SDAMamba and FFCDConv is obtained, and the lane line detection image of the test set is obtained.

[0051] All the loss terms above are combined to define the overall training loss of the lane line detection model as follows:

[0052] The weights of different loss functions are as follows: = 100, = 1000, = 100, = 100, = 10.

[0053] When training the lane line detection model, the AdamW optimizer is used, and the learning rate is first set to , and the step decay learning rate strategy and constant warm-up strategy are used, and then the learning rate is reduced by 0.9 every 1 epoch in CULane. The training set in CULane is trained for 20 epochs, and the batch size is 2.

[0054] The F1-measure is based on the IoU (Intersection over Union) of the predicted lane and the ground truth lane. The lane with IoU greater than 0.5 is considered as correct prediction. The calculation method of F1-measure is as follows:

[0055] where TP is the number of correctly predicted lane points, FP and FN are the number of false positive and false negative lane points, respectively.

[0056] The embodiment compares the method of the application with the existing deep learning-based lane line detection methods of SCNN, Ultrafast, LaneAF, LaneATT and SGNet in performance, and the experimental results are shown in Table 1.

[0057] Table 1 Comparison results of objective evaluation of different lane line detection methods on CULane test data As shown in Table 1, the lane line detection method based on SDAMamba and FFCDConv provided by the embodiment achieves high values on the seven indexes of Total, Normal, Crowded, Shadow, No line, Arrow and Night, which proves the effectiveness and practicability of the method of the application.

[0058] Figure 7 FIG. 1 is a comparison diagram of visual results of three existing lane line detection methods and the lane line detection method of the application on the CULane dataset, and a ground truth map of the CULane dataset; wherein (a) is a result map of the existing Ultrafast lane line detection method, (b) is a result map of the existing LaneATT lane line detection method, (c) is a result map of the existing SGNet lane line detection method, (d) is a result map of the lane line detection method of the application, and (e) is a ground truth map of the CULane dataset. Through the comparison of (a) to (e), it can be intuitively seen that the method of the application can accurately detect all lane lines in real complex road conditions, and has great advantages in detection accuracy compared with other existing methods, and is almost completely consistent with the true value annotation of the CULane dataset. Figure 7

[0059] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope defined by the application.​

Claims

1. A lane detection method based on SDAMamba and FFCDConv, characterized in that, The method includes the following steps: Step 1: Obtain the training set for lane line detection; Step 2: Perform downsampling and image enhancement on the image samples in the training set sequentially to obtain an updated training set; Step 3: Improve the DAMamba Block in the existing DAMamba model. The improved DAMamba Block is named SDAMamba Block, and the DAMamba model after the improvement of DAMamba Block is called the SDAMamba model. Step 4: Using the SDAMamba model as the backbone network, input the image samples from the updated training set into the backbone network to obtain multi-scale feature maps. ; Step 5: Using the existing feature pyramid network as the neck network, process the multi-scale feature maps output by SDAMamba. Simultaneously input into the neck network according to, , , Feature fusion is performed sequentially to obtain a deep multi-scale feature map. ; Step 6: Utilize feature maps The binary cross-entropy loss function is determined by updating the image samples in the training set. ,based on Train the backbone network; Step 7: Use the existing deep Hough transform algorithm to transform the deep multi-scale feature map output in Step 5. Mapping to the Hough parameter space, and then selecting the Hough features for each lane. ; Step 8: Hough features With feature map The combined inputs are fed into the FFCDConv module to obtain features specific to different lane instances. ; Step 9: The lane decoder obtains the predicted lane location map; Step 10: Input the predicted lane position map into the linear layer to determine the final lane position, thereby obtaining the lane line detection image.

2. The lane line detection method according to claim 1, characterized in that, The improvements to the DAMamba Block in the existing DAMamba model include: 1) Improve the DAS module and call the improved DAS module the SDAS module; 2) Remove the convolutional feedforward network and batch normalization layer from the existing DAMamba Block.

3. The lane line detection method according to claim 2, characterized in that, The improvement to the DAS module is as follows: the Sobel edge detection algorithm is introduced to extract lane line edge features from the original feature map of the offset prediction network in the DAS module to be input, generate an edge feature map, and then the edge feature map is fused with the original feature map of the offset prediction network in the DAS module by weighted average to form an enhanced feature map with fused geometric prior, which is used as the input of the offset prediction network.

4. The lane line detection method according to claim 1, characterized in that, The use of feature maps The binary cross-entropy loss function is determined by updating the image samples in the training set. ,include: Feature maps are processed through multiple convolutions. Decoded into a predicted lane probability map of the same size as the image samples in the updated training set. Specifically: the first layer of transposed convolution achieves 2x upsampling while reducing the number of channels; the second layer of bilinear interpolation achieves fine upsampling; the third layer of convolution generates... ; The binary cross-entropy loss function is determined according to equation (1). ; （1） in This represents a probability map of actual lane conditions on the ground.

5. The lane line detection method according to claim 1, characterized in that, Step 7 specifically includes: Step 7.1: Process the deep multi-scale feature map output from Step 5. Interpolation is performed to obtain the Euclidean space features. ; Step 7.2: Use the deep Hough transform algorithm to... Mapping to the Hough parameter space yields the Hough parameter features. ; Step 7.3: Characterize the Hough parameters obtained in Step 7.2 The aggregation is performed, and the aggregated Hough parameter features are represented as G. G is then decoded to generate a Hough map. Step 7.4: Select Hough features for each lane based on the Hough map. .

6. The lane line detection method according to claim 5, characterized in that, Step 7.4 involves selecting Hough features for each lane based on the Hough map. The localization is achieved by applying nonmaximum suppression and threshold filtering to the Hough map. Specifically, first, max pooling is applied to the Hough map to suppress non-maximum pixels, and then the probability values ​​are retained. Points exceeding the threshold are considered as The threshold is set to 0.1, where each pixel on the Hough map... value This indicates the predicted probability that a lane instance exists at that point.

7. The lane line detection method according to claim 1, characterized in that, The FFCDConv module is composed of a cascaded dual-branch dynamic convolution and a feature fusion module. The dual-branch dynamic convolution is composed of a parallel frequency domain dynamic convolution of the global branch and a spatial domain dynamic convolution of the local branch. Both the frequency domain dynamic convolution of the global branch and the spatial domain dynamic convolution of the local branch adopt a serial structure design. The frequency domain dynamic convolution of the global branch is composed of the global branch of FFC and the first dynamic convolution module cascaded in sequence, and the spatial domain dynamic convolution of the local branch is composed of the local branch of FFC and the second dynamic convolution module cascaded in sequence.

8. The lane line detection method according to claim 7, characterized in that, Hough characteristics With feature map The combined input is fed into the FFCDConv module to obtain features for different lane instances. The process is as follows: First of all Data is divided into local branches along the dimension of its feature channels. and data sent to the global branch ,in This represents the percentage of feature channels allocated to the global branch; then, Lane instance features with local details are obtained through two parallel convolutional layers and a second dynamic convolutional module in the local branch of FFC. And send it to the feature fusion module, Lane instance features with global context are obtained through parallel signal transformers and convolutional layers in the FFC global branch, as well as the first dynamic convolutional module. It is then sent to the feature fusion module, where the signal transformer uses 1×1 convolution and ReLU to reduce... The number of data channels; the kernel parameters of the 2D convolutional layers in the dynamic convolution module are determined by... The data is fed into the multilayer sensor. The feature fusion module processes the data as follows: It takes data input to the module... and Perform bilinear interpolation to align to a spatial dimension of Then, lightweight convolutions are used to calibrate the channel dimensions of the two aligned features to 0.75C and 0.25C respectively. Finally, the two aligned and calibrated features are concatenated by channel dimension to output the final lane instance features. N is the total number of lanes. Representing the Lane.

9. The lane line detection method according to claim 1, characterized in that, The lane decoder is composed of transposed convolution and upsampling concatenation.

10. The lane line detection method according to claim 1, characterized in that, Step 10 describes the method of inputting the predicted lane position map into the linear layer to determine the final lane position as follows: First, extract row features from the predicted lane position map and input them into the linear layer to perform a binary classification operation to determine whether a lane exists in each row; then, define the set of rows that are continuously determined to have lanes as the predicted vertical range. The y-coordinate of each lane can be determined by the vertical range and the image height H of the updated image samples in the training set. H represents the total height of the image. The height of the image is represented by the first... The process involves applying a softmax layer to the row features within the vertical range to obtain the x-coordinate of each lane, and filling the row features outside the vertical range with default values ​​to ultimately determine the lane position.

Images