Video lane detection method based on multi-source feature fusion

Abstract

The present invention relates to a video lane line detection method based on multi-source feature fusion, which belongs to the research field of computer vision and image processing. It comprises the following steps: using a phase branch to obtain a phase map feature of a current frame, and refining the phase map feature into a phase-defogging feature and a phase-lane line detection feature; a color-defogging branch extracts the color-defogging feature of the current frame, and after splicing and reducing the dimension with the phase-defogging feature, decomposes it into a color-defogging-defogging feature and a color-defogging-lane line feature; a color-lane line branch extracts and splices the features of an ordered video sequence, fuses and reduces the dimension, and obtains a color-lane line feature through a spatial attention module; and a mask-based multimodal self-attention fusion module is used to enhance the color-lane line feature by combining the phase-lane line detection feature and the color-defogging-lane line feature. The present invention can improve the lane line detection accuracy in foggy videos with reduced visibility and degraded quality.

Description

Technical Field

[0001] The present invention relates to a lane line detection method, and in particular to a video lane line detection method based on multi-source feature fusion, which belongs to the research field of computer vision and image processing. Background Art

[0002] With the maturity of autonomous driving technology, lane detection has attracted more and more attention from researchers. Commonly used video lane detection datasets based on deep learning include Caltech Lanes, ApolloScape, BDD100K, and VIL-100. Among them, Caltech-Lanes only contains 4 videos with a total of 1224 images, which is a small number and is usually not used to train deep convolutional neural networks. BDD100K and ApolloScape are two large video datasets for autonomous driving. However, these two datasets do not provide instance-level annotations of lane lines. VIL-100 is a video dataset for instance-level lane detection. Although VIL-100 provides lane detection scenes in foggy weather, the amount of this type of video data is too small (3 / 100), which is far from enough for studying lane detection tasks in foggy weather.

[0003] Lane detection algorithms based on deep learning can be divided into detection methods based on key point detection, anchors, and parameter curves. The key point detection method detects lane line instances and uses post-processing technology to divide the detected instances into lane lines. However, in foggy environments, this method is difficult to construct global information, which affects the detection accuracy. The anchor-based lane detection algorithm usually uses a top-down method to detect lane lines. This method simplifies the pixel-by-pixel classification method of the segmentation method to anchor-by-anchor classification, which simplifies the complexity of the problem. However, there are positioning errors in the detection of some anchor points, which reduces the detection accuracy. The parameter curve-based method uses a pre-set parameter curve to predict the shape of the lane line. This method solves the problem of difficulty in fitting using polynomial curves and reduces the difficulty of model building and training. However, some lane lines are difficult to fit using parameter curves, so the accuracy is not high.

[0004] Therefore, to address the problem that lane line detection algorithms have low detection accuracy, especially in foggy videos with reduced visibility and degraded quality, a video lane line detection method based on multi-source feature fusion is proposed. Summary of the invention

[0005] The purpose of the present invention is to provide a video lane line detection method based on multi-source feature fusion to solve the problem of low lane line detection accuracy in foggy videos with reduced visibility and quality degradation in the prior art.

[0006] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0007] A video lane detection method based on multi-source feature fusion includes the following steps:

[0008] S1. Construct the VIL-hazing67 lane detection dataset in foggy weather;

[0009] S2. Construct a phase branch module. The phase branch extracts the current frame of the video in the dataset through the phase encoder and obtains the phase map feature through fast Fourier transform. The spatial attention module is used to refine the phase map feature into phase-defogging feature and phase-lane line detection feature.

[0010] S3. Construct a color-defogging branch module, wherein the color-defogging branch extracts the color-defogging feature of the current frame, concatenates and reduces the dimension of the color-defogging feature with the phase-defogging feature, and decomposes the reduced-dimensional phase-defogging feature into a color-defogging-defogging feature and a color-defogging-lane line feature;

[0011] S4. Construct a color-lane branch module. The color-lane branch extracts the features of the past n frames of ordered video sequences and concatenates them. It uses 1×1 convolution for fusion and dimensionality reduction. Finally, the spatial attention module is applied to the features to obtain the color-lane features.

[0012] S5. Construct a mask self-attention module, and use the phase-lane line detection feature and the color-desmog-lane line feature to enhance the color-lane line feature through the mask self-attention module, so as to obtain a lane line detection result.

[0013] Furthermore, step S1 specifically includes:

[0014] S1-1. According to the existing public dataset VIL-100, there are 100 videos, including different scenes such as sunny days, cloudy days, nights, heavy fog, and crowded traffic. The scenes with obvious shadows in VIL-100 and the three foggy scenes without fog-free scene labels are removed, and 67 videos that meet the conditions are obtained;

[0015] S1-2. Construct an atmospheric scattering model, obtain depth maps of the 67 video scenes through the existing robust_cvd method, and generate foggy videos of different concentrations for each clean video in the depth map by adjusting the ambient light A and the scattering factor β in the atmospheric scattering model to obtain the final VIL-hazing67 dataset. The ambient light A and the scattering factor β are set between [1.0, 1.5] and [1.8, 2.4] respectively.

[0016] Furthermore, step S2 includes the following specific steps:

[0017] S2-1. The phase encoder in the phase branch module extracts the current frame The phase map is obtained by fast Fourier transform Phase characteristics , the formula is as follows:

[0018] ,

[0019] ,

[0020] in, represents the fast Fourier transform, represents a phase encoder;

[0021] S2-2. Phase characteristics Phase-dehazing features are obtained through two spatial attention modules in the phase branch module. and phase-lane detection features , the formula is as follows:

[0022] ,

[0023] ,

[0024] in, Represents the phase branch module used to obtain the phase-defog feature The spatial attention module, Represents the phase branch module used to obtain phase-lane detection features The spatial attention module.

[0025] Furthermore, step S3 includes the following specific steps:

[0026] S3-1. The encoder in the color-defogging branch module extracts the current frame Color-defog feature , the color-defog feature The color-dehazing features obtained by the spatial attention module in the color-dehazing branch module , the formula is as follows:

[0027] ,

[0028] ,

[0029] in, Represents the color-defog encoder, Represents the spatial attention module in the color-dehazing branch module;

[0030] S3-2. The phase-defog feature and color-dehazing features Splicing is performed to obtain splicing features, and the splicing features are reduced in dimension using a 1×1 convolution kernel to obtain features after dimensionality reduction. , the formula is as follows:

[0031]

[0032] in, represents the feature concatenation operation, Represents the convolution kernel as Convolution operation;

[0033] S3-3. Reduce the dimension of the features Decompose into color-dehaze-dehaze features and color-dehazing-lane line features .

[0034] Furthermore, step S4 includes the following specific steps:

[0035] S4-1. The encoder in the color-lane branch module extracts adjacent features of the past n frames of ordered video sequence and sequentially splices the adjacent features together to obtain the splicing features of the past n frames of ordered video sequence , the formula is as follows:

[0036] ,

[0037] ,

[0038] ,

[0039] ,

[0040] in, Represents the color-lane encoder, Represents feature concatenation operation;

[0041] S4-2. Use 1×1 convolution kernel to stitch features of the past n frames of ordered video sequence Fusion and dimensionality reduction to obtain features , the formula is as follows:

[0042] ,

[0043] in, Represents the convolution kernel as Convolution operation;

[0044] S4-3. The characteristics The color-lane line features are obtained by the spatial attention module in the color-lane line branch module , the formula is as follows:

[0045] ,

[0046] in, Represents the spatial attention module in the color-lane branch module.

[0047] Furthermore, step S5 includes the following specific steps:

[0048] S5-1. The phase-lane detection feature , color-defogging-lane line features and color-lane features As input features of the masked self-attention module;

[0049] S5-2. The input features are dynamically and adaptively generated into S tokens by the masked self-attention module, denoted as [ ];

[0050] S5-3. Based on color-lane line features Intrinsic self-similarity, and as well as and The similarity between the two modalities is set as: , calculate the similarity, and then get the features composed of S tokens. The formula is as follows:

[0051] ,

[0052] in, = ;

[0053] S5-4. The features composed of the S tokens are input into the modified TokenFuser module for restoration to obtain the restored features;

[0054] S5-5. Send the restored features to the decoder to predict the lane line results.

[0055] Furthermore, the modified TokenFuser module described in step S5-4 removes the layer normalization operations before and after the linear layer in the TokenFuser module.

[0056] The advantages of the present invention are:

[0057] 1. A dataset VIL-hazing67 was constructed for the task of lane line detection in foggy weather videos.

[0058] 2. A lane detection network based on masked color-phase feature fusion is proposed, including a phase branch module, a color-defogging branch module, a color-lane branch module and a masked self-attention module. The phase branch is used to obtain phase-defogging features and phase-lane detection features; the color-defogging branch extracts the color-defogging features of the current frame, and after splicing and dimension reduction with the phase-defogging features, it is decomposed into color-defogging-defogging features and color-defogging-lane features; the color-lane branch extracts the features of the ordered video sequence and splices them, fuses and reduces the dimension, and obtains the color-lane features through the spatial attention module; the masked multimodal self-attention fusion module is used to enhance the color-lane features by combining the phase-lane detection features and the color-defogging-lane features, effectively improving the accuracy of lane detection.

[0059] 3. A multi-feature self-attention fusion module with mask is proposed to achieve the effective fusion of multi-source features of phase features, color-dehazing features and color-lane line features. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0061] Figure 1 This is a video lane detection model diagram based on multi-source feature fusion of the present invention.

[0062] Figure 2 A model diagram showing the structure of a masked self-attention module (a) and TokenFuser (b) of the present invention.

[0063] Figure 3 The VIL-hazing67 dataset constructed for the present invention is shown.

[0064] Figure 4 This is the result of lane line detection by the video lane line detection method TGC-Net for foggy and non-fog images of the same scene.

[0065] Figure 5 A visual comparison of the method of the present invention and other methods on the VIL-hazing67 test set. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Example 1

[0068] In this embodiment, Figure 1 As shown, the present invention provides a video lane detection method based on multi-source feature fusion, and the specific steps include:

[0069] S1. Construct the VIL-hazing67 lane detection dataset in foggy weather;

[0070] S2. Construct a phase branch module. The phase branch extracts the current frame of the video in the dataset through the phase encoder and obtains the phase map feature through fast Fourier transform. The spatial attention module is used to refine the phase map feature into phase-defogging feature and phase-lane line detection feature.

[0071] S3. Construct a color-defogging branch module. The color-defogging branch extracts the color-defogging feature of the current frame, concatenates and reduces the dimension with the phase-defogging feature, and decomposes the reduced-dimensional phase-defogging feature into color-defogging-defogging feature and color-defogging-lane line feature;

[0072] S4. Construct a color-lane branch module. The color-lane branch extracts the features of the past n frames of ordered video sequences and concatenates them. It uses 1×1 convolution for fusion and dimensionality reduction. Finally, the spatial attention module is applied to the features to obtain the color-lane features.

[0073] S5. Construct a mask self-attention module, and use the phase-lane line detection feature and color-dehazing-lane line feature to enhance the color-lane line feature through the mask self-attention module to obtain the lane line detection result.

[0074] In this embodiment, step S1 specifically includes:

[0075] S1-1. The existing public dataset VIL-100 has a total of 100 videos, including different scenes such as sunny days, cloudy days, nights, heavy fog, and crowded traffic. Based on the phenomenon that there is usually no sun in foggy weather and no shadows, the present invention removes scenes with obvious shadows in VIL-100; the three foggy scenes included in VIL-100 do not have labels for their corresponding fog-free scenes. From the perspective of the defogging task, due to the lack of fog-free labels, the present invention also chooses to remove these three foggy scenes. So far, the screening of the VIL-100 dataset scenes has been completed, and a total of 67 videos that meet the conditions have been screened out, named IL-hazing67, as shown in Figure 1. Figure 3 As shown, the VIL-hazing67 dataset constructed by the present invention includes foggy images, lane line labels and clean images;

[0076] S1-2. Construct an atmospheric scattering model and obtain depth maps of 67 video scenes through the existing robust_cvd method. For each clean video in the depth map, generate foggy videos of different concentrations by adjusting the ambient light A and scattering factor β in the atmospheric scattering model to obtain the final VIL-hazing67 dataset. The ambient light A and scattering factor β are set between [1.0, 1.5] and [1.8, 2.4] respectively.

[0077] In this embodiment, step S2 includes the following specific steps:

[0078] S2-1. The phase encoder in the phase branch module extracts the current frame The phase map is obtained by fast Fourier transform Phase characteristics , the formula is as follows:

[0079] ,

[0080] ,

[0081] in, represents the fast Fourier transform, represents a phase encoder;

[0082] S2-2. Phase characteristics Phase-dehazing features are obtained through two spatial attention modules in the phase branch module. and phase-lane detection features , the resolution is , the formula is as follows:

[0083] ,

[0084] ,

[0085] in, Represents the phase branch module used to obtain the phase-defog feature The spatial attention module, Represents the phase branch module used to obtain phase-lane detection features The spatial attention module.

[0086] In this embodiment, step S3 includes the following specific steps:

[0087] S3-1. The encoder in the color-defogging branch module extracts the current frame Color-defog feature , color-dehazing feature The color-dehazing features obtained by the spatial attention module in the color-dehazing branch module , with a resolution of , the formula is as follows:

[0088] ,

[0089] ,

[0090] in, Represents the color-defog encoder, Represents the spatial attention module in the color-dehazing branch module;

[0091] S3-2. Phase-defog feature (resolution is ) and color-defog features (resolution is ) are stitched together to obtain a resolution of The splicing features are reduced in dimension using a 1×1 convolution kernel to obtain a resolution of The features after dimensionality reduction , the formula is as follows:

[0092]

[0093] in, represents the feature concatenation operation, Represents the convolution kernel as Convolution operation;

[0094] S3-3. Reduce the dimension of the features Decompose into color-dehaze-dehaze features and color-dehazing-lane line features .

[0095] In this embodiment, step S4 includes the following specific steps:

[0096] S4-1. The encoder in the color-lane branch module extracts adjacent features of the past n frames of ordered video sequence and sequentially splices the adjacent features together to obtain a resolution of The splicing features , the formula is as follows:

[0097] ,

[0098] ,

[0099] ,

[0100] ,

[0101] in, Represents the color-lane encoder, Represents feature concatenation operation;

[0102] S4-2. Use 1×1 convolution kernel to stitch features of the past n frames of ordered video sequence Fusion and dimensionality reduction to obtain features , , the formula is as follows:

[0103] ,

[0104] in, Represents the convolution kernel as Convolution operation;

[0105] S4-3. Characteristics The color-lane line features are obtained by the spatial attention module in the color-lane line branch module , the formula is as follows:

[0106] ,

[0107] in, Represents the spatial attention module in the color-lane branch module.

[0108] In this embodiment, step S5 includes the following specific steps:

[0109] S5-1. Phase-lane detection features , color-defogging-lane line features and color-lane features As input features of the masked self-attention module;

[0110] S5-2. The input features are dynamically and adaptively generated by the masked self-attention module to generate S tokens (in the present invention, from =3600 to S =32), denoted as [ ];

[0111] S5-3. Based on color-lane line features Intrinsic self-similarity, and as well as and The similarity between the two modalities is set as: , calculate the similarity, and then get the features composed of S tokens. The formula is as follows:

[0112] ,

[0113] in, = ;

[0114] S5-4. Input the features composed of the above S tokens into the modified TokenFuser module for restoration. arrive , to obtain the restored features, the present invention removes the layer normalization operations before and after the linear layer to maintain the diversity of the restored features;

[0115] S5-5. Send the restored features to the decoder to predict the lane line results.

[0116] Example 2

[0117] In this embodiment, based on the constructed VIL-hazing67 dataset, the existing method TGC-Net was tested for lane line detection on fog-free images and foggy images of the same scene, and then the video lane line detection method based on multi-source feature fusion of the present invention was compared with other existing methods.

[0118] In this embodiment, Figure 4 The figure shows the lane line detection test results of the video lane line detection method TGC-Net on fog-free images and foggy images of the same scene. It can be seen that there is an obvious error between the lane lines detected by TGC-Net on foggy images and those on fog-free images.

[0119] In this embodiment, Figure 5As shown in the figure, the lane line detection results obtained by the video lane line detection method based on multi-source feature fusion of the present invention and the lane line detection methods such as SCNN, ENet-SAD and TGC-Net on the VIL-hazing67 dataset are compared. It can be seen from the figure that the lane line detected by the method proposed in the present invention is more accurate than that by other methods, and the lane line detection effect is better.

[0120] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A video lane detection method based on multi-source feature fusion, characterized in that: The following steps are involved: S1. Construct the VIL-hazing67 lane detection dataset in foggy weather; S2. Construct a phase branch module. The phase branch extracts the current frame of the video in the dataset through the phase encoder and obtains the phase map feature through fast Fourier transform. The spatial attention module is used to refine the phase map feature into phase-defogging feature and phase-lane line detection feature. The specific steps are as follows: S2-1. The phase encoder in the phase branch module extracts the current frame The phase map is obtained by fast Fourier transform Phase characteristics , the formula is as follows: ， ， in, represents the fast Fourier transform, represents a phase encoder; S2-2. Phase characteristics The two spatial attention modules in the phase branch module respectively obtain the phase-dehazing features. and phase-lane detection features , the formula is as follows: ， ， in, Represents the phase branch module used to obtain the phase-defog feature The spatial attention module, Represents the phase branch module used to obtain phase-lane detection features The spatial attention module; S3. Construct a color-defogging branch module. The color-defogging branch extracts the color-defogging feature of the current frame, and concatenates and reduces the dimension with the phase-defogging feature, and decomposes the reduced-dimensional phase-defogging feature into color-defogging-defogging feature and color-defogging-lane line feature. The specific steps are as follows: S3-1. The encoder in the color-defogging branch module extracts the current frame Original color-dehazing features , the original color-desmog feature The color-dehazing features obtained by the spatial attention module in the color-dehazing branch module , the formula is as follows: , , in, Represents the color-defog encoder, Represents the spatial attention module in the color-dehazing branch module; S3-2. The phase-defog feature and color-dehazing features Splicing is performed to obtain splicing features, and the splicing features are reduced in dimension using a 1×1 convolution kernel to obtain features after dimensionality reduction. , the formula is as follows: in, represents the feature concatenation operation, The convolution kernel is Convolution operation; S3-3. Reduce the dimension of the features Decompose into color-dehaze-dehaze features and color-dehazing-lane line features ; S4. Construct a color-lane branch module. The color-lane branch extracts and concatenates the features of the past n frames of ordered video sequences, uses 1×1 convolution pairs to fuse and reduce the dimension, and finally applies the spatial attention module to the fused and reduced features to obtain the color-lane features. S5. Construct a mask self-attention module, and use the phase-lane line detection feature and the color-desmog-lane line feature to enhance the color-lane line feature through the mask self-attention module to obtain a lane line detection result.

2. The video lane detection method based on multi-source feature fusion according to claim 1 is characterized in that: Step S1 specifically includes: S1-1. According to the existing public dataset VIL-100, there are 100 videos in VIL-100, including different scenes such as sunny days, cloudy days, nights, heavy fog and congested traffic. The scenes with obvious shadows in VIL-100 are removed, and then 3 foggy scenes without fog-free scene labels are removed, and 67 videos that meet the conditions are obtained; S1-2. Construct an atmospheric scattering model, obtain the depth map of the 67 video scenes through the robust_cvd method, and generate foggy videos of different concentrations for each clean video in the depth map by adjusting the ambient light A and the scattering factor β in the atmospheric scattering model to obtain the final VIL-hazing67 dataset. The ambient light A and the scattering factor β are set between [1.0, 1.5] and [1.8, 2.4] respectively.

3. The video lane detection method based on multi-source feature fusion according to claim 2 is characterized in that: Step S4 includes the following specific steps: S4-1. The encoder in the color-lane branch module extracts adjacent features of the past n frames of ordered video sequence and sequentially splices the adjacent features together to obtain the splicing features of the past n frames of ordered video sequence , the formula is as follows: ， ， ， ， in, Represents the color-lane encoder, Represents feature concatenation operation; S4-2. Use 1×1 convolution kernel to stitch features of the past n frames of ordered video sequence Fusion and dimensionality reduction to obtain features , the formula is as follows: ， in, The convolution kernel is Convolution operation; S4-3. The characteristics The color-lane line features are obtained through the spatial attention module in the color-lane line branch module , the formula is as follows: , in, Represents the spatial attention module in the color-lane branch module.

4. The video lane detection method based on multi-source feature fusion according to claim 3 is characterized in that: Step S5 includes the following specific steps: S5-1. The phase-lane detection feature , color-defogging-lane line features and color-lane features As input features of the masked self-attention module; S5-2. The input features are dynamically and adaptively generated into S tokens by the masked self-attention module, denoted as [ ]; S5-3. Based on color-lane line features Intrinsic self-similarity, and as well as and The similarity between the two modalities is set as: , calculate the similarity, and then get the features composed of S tokens. The formula is as follows: ， in, = ; S5-4. The features composed of the S tokens are input into the modified TokenFuser module for restoration to obtain the restored features; S5-5. Send the restored features to the decoder to predict the lane line results.

5. The video lane detection method based on multi-source feature fusion according to claim 4 is characterized in that: The modified TokenFuser module described in step S5-4 removes the layer normalization operations before and after the linear layer in the TokenFuser module.

Images