Lane line detection method, system, equipment and medium

Abstract

The invention discloses a lane line detection method, system and device and a medium. The method comprises the following steps: inputting a to-be-detected lane line image into a trained generative adversarial network model to obtain a lane line detection result; the adversarial generative network model comprises a generator network and a discriminator network, and the generator network serves as a lane line detection model and comprises a trunk feature extraction network based on Swin Transform, a bidirectional feature pyramid and a bilateral up-sampling decoder; the Swin Transform-based trunk feature extraction network is used for extracting multi-scale global features of a lane line through a window-based self-attention method, the bidirectional feature pyramid is used for carrying out top-down and bottom-up weighted fusion on the multi-scale global features, the bilateral up-sampling decoder is used for carrying out up-sampling on the fused features, and the Swin Transform-based trunk feature extraction network is used for extracting the multi-scale global features of the lane line through a window-based self-attention method. And obtaining a detection result of the lane line. According to the method, the detection performance in a scene with good lane line visual information can be ensured, and meanwhile, the detection performance in complex scenes such as lane line abrasion and lack can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to the field of visual lane line detection of autonomous driving vehicles, and specifically to a lane line detection method, system, device and medium. Background Art

[0002] Lane detection is a challenging task in the field of autonomous driving technology because it is affected by many factors, such as the complexity and diversity of road traffic scenes, the lack of lane line visual information such as lane line wear and loss, and the inherent narrowness of lane lines. As the basis of intelligent advanced driver assistance system (ADAS) functions such as path planning, lane keeping and adaptive cruise control, lane detection must have good accuracy and robustness.

[0003] At present, lane detection technology is mainly divided into two types: traditional lane detection method and lane detection method based on deep learning. Traditional lane detection method requires manual design of features and uses original image information such as edges, colors, and textures to solve the lane detection problem. It has low computational complexity and limited feature extraction ability. It is only applicable to scenes with little illumination change and obvious lane features. It cannot maintain robustness in complex real scenes. The method based on deep learning automatically learns richer and more abstract feature representations by using convolutional neural networks. It has stronger feature extraction ability and higher detection accuracy and robustness than traditional methods. Therefore, this method has become mainstream. At present, the network framework of lane detection based on deep learning usually adopts the form of encoder-decoder. The encoder of most algorithms uses stacked convolutional neural networks to extract features from local areas of the image while downsampling. However, due to the limitations of convolution operations, only local pixels are operated when extracting image features, while ignoring the global information of the image. At the same time, the importance of features at various scales to the lane detection task is not fully considered. When the visual feature information of the lane is missing, the detection performance is greatly reduced. In practical applications, due to the complexity and variability of real road scenes, and the lack of visual feature information caused by wear and tear and missing lane lines due to long-term use, the existing lane line detection technology performs poorly. Therefore, it is very challenging to perform accurate lane line detection in a large number of complex scenes in real situations.

[0004] Zequn Qin et al. disclosed a lane detection method in Ultra Fast Structure-aware Deep Lane Detection (Qin, Z., Wang, H., Li, X. Ultra Fast Structure-Aware Deep Lane Detection [C]. The European Conference on Computer Vision (ECCV). ECCV, 2020.). This method uses a ResNet stacked convolutional neural network to downsample the lane image multiple times, and performs a fully connected (FC) calculation on the last layer of feature maps extracted by the network to obtain the global feature information of the image, which is used to solve the problem of no lane line visual clues, such as lane line occlusion, wear and loss. However, the stacked convolutional neural network downsamples the image multiple times, reducing the resolution of the feature map, resulting in the loss of tiny lane line target information. At the same time, only the last layer of feature maps extracted by the network is used for prediction, and the lane line feature information of different scales is not fully considered. The global information obtained by fully connecting (FC) the feature map will cause the destruction of the spatial structure information of the lane line. This technology fails to effectively improve the detection performance in the scene without lane line visual clues. Summary of the invention

[0005] In order to solve at least one of the problems existing in the prior art, the present invention provides a lane line detection method, system, device and medium based on Swin Transformer and Generative Adversarial Network (GAN), uses image processing technology to process a public data set on the Internet to construct a data set without obvious lane line visual clues, and uses a self-attention mechanism to overcome the inability of traditional convolutional neural networks to obtain the global context information of the image. At the same time, the influence of different scale features on lane line feature extraction is considered, and adversarial game training of Generative Adversarial Network (GAN) is introduced to provide more gradient information for the lane line detection model, thereby ensuring the detection performance of the model in scenes with good lane line visual information, and improving the detection accuracy in scenes without obvious lane line visual features.

[0006] To achieve the purpose of the present invention, the present invention provides a lane line detection method, which obtains a lane line detection result by inputting a lane line image to be detected into a trained adversarial generative network model;

[0007] The adversarial generative network model includes a generator network and a discriminator network. During training, the discriminator network and the generator network play an adversarial game. The generator network, as a lane line detection model, includes a backbone feature extraction network based on SwinTransformer, a bidirectional feature pyramid and a bilateral upsampling decoder. The backbone feature extraction network based on SwinTransformer is used to extract multi-scale global features of lane lines through a window-based self-attention method. The bidirectional feature pyramid is used to perform top-down and bottom-up weighted fusion of multi-scale global features. The bilateral upsampling decoder is used to upsample the fused features to obtain lane line detection results.

[0008] Among them, image processing technology is used to erase the lane line visual information to construct a road traffic dataset without obvious lane line visual information, and the adversarial generative network model is trained by the road traffic dataset without obvious lane line visual information.

[0009] The present invention uses image processing technology to erase the visual information of lane lines and constructs a data set without obvious visual information of lane lines. The multi-scale features of lane lines are extracted through Swin Transformer, and the narrow and long structure of lane lines is locally and globally associated. The multi-scale features are weightedly fused using a bidirectional weighted feature pyramid to construct a generator network. The discriminator network enhances the feature extraction ability of the model by introducing GAN adversarial game training, making the model suitable for lane line detection tasks in complex traffic environments.

[0010] Furthermore, a road traffic scene diagram containing lane line information is obtained by collecting video data from a car driving recorder, and lane line labels are annotated to construct an original data set, or a road data set for lane line detection tasks publicly available on the Internet is downloaded as the original data set, and the original data set contains lane lines and their labels; the lane line area of ​​the image in the original data set is erased using content-filling-based image processing technology to construct a new data set without lane line visual information, the training set of the new data set is a composition set of the training set of the original data set after the lane lines are erased, the test set of the new data set includes the original test set and the composition set after the lane line visual information of the original test set is erased, the training set of the new data set is used to train the generative adversarial network model, and the test set of the new data set is used to test the generative adversarial network model.

[0011] Among them, during testing, the trained adversarial generative network model is tested using the test set, where the discriminator network does not work during testing.

[0012] During training, the newly constructed dataset is used to train the lane detection model through an adversarial game against the generative network model, so that the lane detection model converges to obtain the parameters of the lane detection model, that is, the parameters of the generator network.

[0013] Furthermore, the Swin Transformer-based backbone feature extraction network includes multiple feature extraction layers for capturing local and global information and gradually extracting multi-scale features. Each feature extraction layer includes a Swin TransformerBlock module, and each Swin Transformer Block module includes a window-based multi-head self-attention and a moving window-based multi-head self-attention. The window-based multi-head self-attention divides the input feature map into multiple windows of equal size and performs self-attention calculation in each window. It can focus on the local structure and detail features of the image, so that the lane line detection model pays more attention to the lane features. The moving window-based multi-head self-attention shifts the windows so that the originally independent windows can interact with each other, thereby promoting the integration of local and global context information.

[0014] Furthermore, the backbone feature extraction network based on Swin Transformer enables each layer to obtain the global features of the image through window-based multi-head self-attention (W-MSA) and moving window-based multi-head self-attention (SW-MSA), including:

[0015] Divide each dimension of the feature map F into multiple windows of size window_size×window_size, each window contains a part of local information;

[0016] After linear transformation of the features in each window, we get the query vector Q, keyword K and feature value V. At the same time, in order to capture the relative relationship between different positions in the window, we introduce the relative position bias B and calculate the attention value Attention through the dot product.

[0017] Multiply the attention value Attention by the feature value V to get the output;

[0018] Recombining the processed window features to restore the original feature map shape;

[0019] Then SW-MSA is calculated. Based on W-MSA, the windows are shifted to enable information interaction between originally independent windows, thus promoting the interaction between local information and global context information.

[0020] Finally, multi-scale lane line features are obtained through downsampling operations.

[0021] Furthermore, the bidirectional feature pyramid performs bidirectional weighted fusion on the feature maps extracted based on the Swin Transformer backbone network, and constrains the weight range by fast standardized weight fusion. The fast standardized weight fusion formula is:

[0022]

[0023] The output after bidirectional weighted fusion is:

[0024] O = conv(w io ·F i )

[0025] Among them, ω i is the initial weight of the i-th input feature map, ∈ is a preset very small value, w j is the weight of the jth input, ω io is the weight of the i-th input after fast normalization weight fusion, F i is the i-th input feature map, conv is convolution, and O is the fused output.

[0026] Furthermore, a decoder is constructed by combining linear interpolation upsampling and deconvolution upsampling operations to upsample the fused features and realize lane line detection based on instance segmentation. The fused features are input into the bilateral upsampling decoder and the output F O for:

[0027] F O =Bilinear(F * )+Upconv(F * )

[0028] In the formula, Bilinear(·) is the linear interpolation upsampling function, Upconv(·) is the deconvolution upsampling, and F * The fused features.

[0029] Furthermore, PatchGAN is constructed as a discriminator network, including multiple convolution modules, each of which is used to downsample the input image pair, and finally outputs a probability prediction map. The value of each point on the probability prediction map represents the probability value prob output by the discriminator network. ij .

[0030] Furthermore, the overall objective function of the adversarial generation network model is expressed as:

[0031]

[0032] Where G is the generator network, D is the discriminator network, x is the input lane line image to be detected, y is the label corresponding to the lane line image to be detected, G(·) is the output of the generator network G, D(·) is the output of the discriminator network D, E x,y [·] represents the expected loss function of the discriminator network on the real image pair (x, y), E x,G(x) [·] represents the expected value of the loss function of the discriminator network on the generated image pair (x, G(x)), and V(G,D) is the overall objective function of the adversarial generative network model.

[0033] The goal of the discriminator network is to maximize the objective function as much as possible, and the goal of the generator network is to minimize the objective function as much as possible, thus forming an adversarial game between the two, providing more information for the lane detection model, enhancing the model's ability to extract key features, and making the lane detection model pay more attention to lane features.

[0034] The invention also provides a lane line detection system.

[0035] The invention also provides a computer device.

[0036] The present invention also provides a computer-readable storage medium.

[0037] Compared with the prior art, the present invention has at least the following beneficial effects:

[0038] The present invention constructs a data set without lane line visual information by using image processing technology, uses SwinTransformer to build a backbone feature extraction network to extract multi-scale features of lane lines, uses window-based multi-head self-attention and moving window-based multi-head self-attention in each layer to make the model pay more attention to key lane line features, strengthens feature fusion through bidirectional weighted feature pyramid, and introduces adversarial game training of adversarial generative network to provide more useful gradient information for lane line detection model, thereby ensuring the detection performance of lane line detection model in scenarios with good lane line visual information, and improving the detection accuracy of the model in scenarios without obvious lane line visual information, so that the model is suitable for a variety of complex road traffic scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flowchart of a lane line detection method provided by an embodiment of the present invention.

[0040] Figure 2 This is an example diagram of a road traffic image after the lane line visual information is erased using image processing technology in an embodiment of the present invention.

[0041] Figure 3It is a schematic diagram of the overall structure of the lane line detection method model based on Swin Transformer and GAN in an embodiment of the present invention.

[0042] Figure 4 It is a schematic diagram of the generator network structure in an embodiment of the present invention.

[0043] Figure 5 It is a schematic diagram of the backbone feature extraction network structure based on Swin-T in the generator network of an embodiment of the present invention.

[0044] Figure 6 Schematic diagram of the PatchGAN discriminator network structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention, through the description of the embodiments, the specific implementation of the present invention will be further described in detail. The specific embodiments described here are only used to explain the present invention, and do not limit the present invention. In order to make the technical solution of the present invention clearer, the present invention is explained by the following embodiments:

[0046] See also Figure 1 The present invention provides a lane line detection method based on Swin Transformer and Generative Adversarial Network (GAN), comprising the following steps:

[0047] Step S1: Use image processing technology to erase lane line visual information, construct a road traffic data set without obvious lane line visual information, and pre-process the data set.

[0048] In some embodiments of the present invention, a public road dataset Tusimple is downloaded from the Internet as an original dataset, which is one of the datasets dedicated to lane line detection tasks, including lane line images and labels, wherein the training set includes 3626 road traffic map images, and the test set includes 2782 road traffic images; the lane line label image is an instance segmentation map, and the lane line area is the non-0 area in the map, and the lane line area is obtained by reading the non-0 area in the lane line label image, and then the lane line visual information is erased from the lane line area using a content-filling-based image processing technique to construct a new dataset without lane line visual information, wherein the training set of the new dataset is a set of components after the lane lines are erased from the training set of the original dataset, totaling 3626 images, and the test set of the new dataset includes the original test set and the set of components after the lane line visual information of the original test set is erased, totaling 5564 images, and since the process of erasing the lane lines does not change the road structure and traffic environment in the image, the labels of the original lane line images can be used directly; such as Figure 2The figure shows an example of a road traffic image after the visual information of the lane line is erased; then the new data set is preprocessed, and the preprocessing includes adjusting the size of the road traffic image to 256×512×3, and performing data enhancement by random rotation and random flipping to enrich the data set. It can be understood that the use of a public data set can save research time and improve research efficiency. In other embodiments, the public data set can be used as the original data set instead of the original data set. Instead, the data can be collected by itself to construct the data set. The video data of the car driving recorder is collected, and then the video data is cut into frames to obtain a road traffic scene image containing lane line information, and the lane line labels are labeled using annotation software to construct the original data set.

[0049] In this step, you can download a data set for lane line detection tasks that is publicly available on the Internet, and use image processing technology to erase the lane line visual information to construct a new data set without lane line visual information; you can also collect data yourself to construct the data set.

[0050] Step S2, constructing a generative adversarial network model, wherein the generative adversarial network model includes a generator network and a discriminator network, the generator network is a lane line detection model, which is used to detect lane lines, and the discriminator network is used to distinguish whether the input image pair is real or generated by the generator network.

[0051] like Figure 3 As shown in the figure, real_A is the input road traffic image to be detected containing lane line information, the label of the road traffic image is real_B, the prediction image generated by the generator network is fake_B, the combination of real_A and real_B is defined as a real image pair, and the combination of real_A and fake_B is defined as a generated image pair; for the corresponding discriminator network, when the input is a real image pair, the goal of the discriminator network is to discriminate these real image pairs as True; when the input is a generated image pair, the goal of the discriminator network is to discriminate these generated image pairs as False; for the generator network, the generator network hopes that the generated image pairs can pass the test of the discriminator network, that is, the discriminator network discriminates the generated image pairs as True, thereby forming an adversarial game between the discriminator network and the generator network, enhancing the feature extraction capability of the generator network.

[0052] Construct a generator network such as Figure 4The figure shows the model structure of the generator network of the present invention, wherein the generator network includes a backbone feature extraction network based on Swin Transformer (Swin-T), a bidirectional feature pyramid and a bilateral upsampling decoder, wherein the backbone feature extraction network is used to extract multi-scale features of lane lines, the bidirectional feature pyramid is used to perform top-down and bottom-up weighted fusion of multi-scale features, and the bilateral upsampling decoder is used to input the fused features into the bilateral upsampling decoder to realize lane line detection based on the instance segmentation method.

[0053] In some embodiments of the present invention, Swin Transformer (Swin-T) is used to construct a backbone feature extraction network. The backbone feature extraction network based on Swin Transformer is used to extract multi-scale features of lane lines, and each layer is able to obtain global features of the image through window-based multi-head self-attention (W-MSA) and moving window-based multi-head self-attention (SW-MSA).

[0054] The network structure of the backbone feature extraction network based on Swin Transformer is as follows Figure 5 As shown. Figure 5(a) shows the structural diagram of Swin-T. The backbone feature extraction network (Swin-T) based on Swin Transformer includes four feature extraction layers, namely stage1, stage2, stage3 and stage4. Stage1 is the starting stage, which is used to process the input image through image patch partitioning and linear embedding and calculate Swin Transformer Blocks for preliminary processing. The feature map obtained in this stage has a high resolution and a small number of channels, and mainly extracts local detail features of the image; stage2 reduces the resolution of the feature map and increases the number of channels through downsampling operations and calculation of Swin Transformer Blocks, further extracting more abstract features; stage3 further downsampling operations, this stage contains more Swin Transformer Block calculations, can process features more deeply, and capture more global feature information; stage4 is the final stage, which generates the most abstract feature map through further downsampling operations, can capture the most global features, and finally obtain lane line features of different scales. In some embodiments of the present invention, stage1, stage2 and stage4 each include 2 Swin Transformer Block modules, and stage3 includes 6 Swin Transformer Block modules. The number of Swin Transformer Block modules in stage1 and stage2 is relatively small, set to 2, mainly to quickly extract the feature information of the image and perform preliminary downsampling. A smaller number of Blocks can reduce the computational complexity while maintaining the richness of the features. Stage3 includes 6 Swin Transformer Block modules, which is to extract rich features at a deeper level and capture more complex content. A larger number of Blocks can better process features and improve the expressive power of the backbone feature extraction network based on Swin Transformer. Stage4 includes 2 Swin Transformer Block modules, which are mainly used to generate the final feature representation. A smaller number of Blocks can reduce the computational complexity while maintaining the abstractness of the features. The backbone feature extraction network based on Swin-T gradually extracts multi-scale features through 4 feature extraction layers. The number of Swin Transformer Block modules in each stage is different, which can balance computational efficiency and feature expression capabilities at different levels. This design makes the SwinTransformer-based backbone feature extraction network more efficient in processing high-resolution images while capturing rich local and global information.The number of feature extraction layers is not strictly limited to 4. In other embodiments, the number of feature extraction layers may be other values. The number of Transformer Block modules in each stage may not necessarily be 2 or 6.

[0055] like Figure 5 (b) shows the structure diagram of the Swin Transformer Block module, where LN is the normalization layer (Layer Normalization) and MLP is the multi-layer perceptron (Multi-Layer Perceptron). In each layer of Swin-T, the feature map z l-1 Input to the Swin Transformer Block module, and the output z is obtained through LN, window-based multi-head self-attention W-MSA, moving window-based multi-head self-attention SW-MSA and MLP calculations l+1 After completing the calculation of the SwinTransformer Block module, the lane line features of different scales are obtained through the image block merging (Patch Merging) downsampling operation. The core parts of the Swin Transformer Block module are the window-based multi-head self-attention (W-MSA) and the moving window-based multi-head self-attention (SW-MSA). The window-based multi-head self-attention divides the input feature map into multiple windows of equal size and performs self-attention calculation in each window. It can focus on the local structure and detail features of the image, so that the lane line detection model pays more attention to the lane features. At the same time, this window-based localized calculation method greatly reduces the amount of calculation; the moving window-based multi-head self-attention shifts the window to enable information interaction between the originally independent windows, promotes the integration of local and global context information, and improves the accuracy of feature extraction.

[0056] The key to Swin-T is the Swin Transformer Block module. In some embodiments of the present invention, the calculation process of the Swin Transformer Block module is as follows:

[0057] Set the window size window_size = 7. The window-based multi-head self-attention W-MSA divides each dimension of the input feature map F into multiple 7×7 windows, each of which contains a part of the local information.

[0058] Perform linear transformation on the features in each window to generate query vector Q, keyword K and feature value V. In order to capture the relative relationship between different positions in the window, introduce relative position bias B, and calculate the attention value Attention through dot product, that is, the strength of the association between pixels. The calculation formula of the attention value Attention is:

[0059]

[0060] Among them, d k is the dimension of keyword K, T represents the transpose operation of the matrix;

[0061] Multiply the attention value Attention by the eigenvalue V to get the output z of the window-based multi-head self-attention W-MSA l :

[0062] z l =V·Attention

[0063] Recombining the processed window features to restore the original feature map shape;

[0064] Then the moving window-based multi-head self-attention SW-MSA is calculated. SW-MSA is based on the window-based multi-head self-attention W-MSA. By shifting the window, the originally independent windows can interact with each other, which promotes the interaction of local and global context information.

[0065] Finally, multi-scale lane line features are obtained through downsampling operations.

[0066] Swin-T stacks different numbers of Swin Transformer Block modules in each stage so that each layer of feature maps can obtain local and global information of the feature map, thereby improving the ability to extract lane features.

[0067] The bidirectional feature pyramid is used to perform top-down and bottom-up weighted fusion of the multi-scale features of the extracted lane lines. The feature fusion process is as follows: Figure 4 As shown in (b), four feature maps F of different scales are obtained through four feature extraction layers based on the Swin Transformer backbone feature extraction network. 1 、F 2 、F 3 and F 4 , the three feature maps F 2 、F 3 、F 4 Input bidirectional feature pyramid, in the top-down fusion process, feature map F 3 With the feature map F 4Fusion is performed to obtain the feature map F 5 , feature map F 2 With the feature map F 5 Fusion is performed to obtain the feature map F 6 ; In the bottom-up fusion process, the feature map F 6 With the feature map F 3 、F 5 Fusion is performed to obtain the feature map F 7 , feature map F 7 With the feature map F 4 Fusion is performed to obtain the feature map F 8 , and multiply the corresponding weight coefficient during the fusion process to obtain the fused feature F 9 , completing the entire process of feature fusion.

[0068] In some embodiments of the present invention, the weight range is constrained by fast normalized weight fusion, so that the fused weight value ω io Between 0 and 1, because feature maps of different scales contain different information, they have different impacts on the final detection effect of the lane detection model. Through learning, the network automatically adjusts the size of the weight. This method can prevent training instability caused by excessive weight values.

[0069] Among them, the fast normalized weight fusion formula is:

[0070]

[0071] The output after bidirectional weighted fusion is:

[0072] O = conv(w io ·F i )

[0073] Among them, ω i is the initial weight of the i-th input feature map, ∈ is a preset very small value to prevent the denominator from being 0, w j is the weight of the jth input, ω io is the weight of the i-th input after fast normalization weight fusion, F i is the i-th input feature map, conv is a 3×3 convolution, and O is the fused output.

[0074] In some embodiments of the present invention, since feature maps of different scales are obtained by downsampling in the backbone feature extraction network based on Swin Transformer, the bidirectional feature pyramid needs to perform linear interpolation expansion or pooling reduction on the feature map to obtain the feature map of different scales. Figure 4 The feature map F in (b) 5 and F 7Take the example, the output after bidirectional weighted fusion is:

[0075]

[0076] w i are the weights of the i-th input feature map; in particular, w 3 It is the feature map F used in the top-down fusion process 3 The weight used when performing feature fusion, w′ 3 It is the feature map F used in the bottom-up fusion process 3 The weight coefficient used when performing feature fusion; R(·) is the function used to adjust the size of the feature map; conv is a 3×3 convolution.

[0077] In some embodiments of the present invention, ∈=0.0001 is set to prevent the numerical value from being unstable;

[0078] The constructed bilateral upsampling decoder structure is as follows Figure 4 As shown in (c), a bilateral upsampling decoder is constructed by combining linear interpolation upsampling and deconvolution upsampling operations to upsample the fused feature map output by the bidirectional feature pyramid to achieve lane line detection based on the instance segmentation method. 9 Input bilateral upsampling decoder, output F O for:

[0079] F O =Bilinear(F * )+Upconv(F * )

[0080] Among them, Bilinear(·) is the linear interpolation upsampling function, Upconv(·) is the deconvolution upsampling, and F * The fused features.

[0081] In some embodiments of the present invention, Figure 4 As shown, the fused feature F * F 9 .

[0082] PatchGAN is constructed as a discriminator network to enhance feature extraction capabilities. Its network structure is as follows Figure 6 shown.

[0083] PatchGAN consists of multiple convolutional modules, each of which is used to downsample the input image pair and finally output a probability prediction map. The value at each point on the probability prediction map represents the probability value prob output by the discriminator network. ij. conv+BN+ReLU constitutes a convolution module. PatchGAN performs convolution downsampling operation on the input image by setting the convolution module, and finally obtains a prediction probability map. The probability value of each location in the probability map reflects the discrimination probability of the corresponding convolution kernel receptive field size area in the input image. In some embodiments of the present invention, PatchGAN includes 5 conv+BN+ReLU convolution modules, and the convolution kernel receptive field size is 70×70. In other embodiments, the number of convolution modules can also be other values.

[0084] Design the loss function of the adversarial generative network;

[0085] The loss function L of the discriminator network D D for:

[0086] L D =(D real +D fake )×0.5

[0087] Where D real is the loss of the discriminator network for real image pairs. When the input of the discriminator network is a real image pair, the goal of the discriminator network is to discriminate these real image pairs as True. Therefore, D real The loss measures the performance of the discriminator network on real image pairs; D fake is the loss of the discriminator network for the generated image pairs. When the input of the discriminator network is the generated image pairs, the goal of the discriminator network is to discriminate these generated image pairs as False, so D fake The loss measures the performance of the discriminator network in generating image pairs; by minimizing the total loss L D , the discriminator network is able to better learn to distinguish between real and generated image pairs.

[0088] Where D real The calculation process is:

[0089] pred_real=D(real_A,real_B)

[0090] pred_real is the output probability prediction graph obtained by simultaneously inputting real_A and real_B into the discriminator network D; similarly, the output probability prediction graph pred_fake obtained by simultaneously inputting real_A and fake_B into the discriminator network D is:

[0091] pred_fake=D(real_A,fake_B)

[0092] The role of the discriminator network is to distinguish between real image pairs and generated image pairs as much as possible, so D real and Dfake The calculation formula is:

[0093] D real =BCE(pred_real,True)

[0094] D fake =BCE(pred_fake,False)

[0095] Where BCE represents the binary cross entropy loss function.

[0096] The loss function L of the generator network G G for:

[0097] L G =G_GAN+βL seg

[0098] Among them, G_GAN measures the difference between the image generated by the generator network and the real image, and is the BCE loss between the discriminator network's discrimination result of the generated image pair and the real label; L seg It represents the instance segmentation loss, which is calculated by the crossentropy loss function, and β is the weight coefficient of the instance segmentation loss.

[0099] For the generator network, the prediction results it generates should be close to the label to deceive the discriminator network as much as possible, so the calculation formula of G_GAN is:

[0100] G_GAN=BCE(pred_fake,True)

[0101] The overall objective function of the adversarial generation network model is expressed as:

[0102]

[0103] Where G is the generator network, D is the discriminator network, x is the input lane line image to be detected, y is the label corresponding to the lane line image to be detected, G(·) is the output of the generator network G, D(·) is the output of the discriminator network D, E x,y [·] represents the expected loss function of the discriminator network on the real image pair (x, y), E x,G(x) [·] represents the expected value of the loss function of the discriminator network on the generated image pair (x, G(x)), and V(G,D) is the overall objective function of the adversarial generative network model.

[0104] The goal of the discriminator network is to maximize the objective function as much as possible, while the goal of the generator network is to minimize the objective function as much as possible, thus forming an adversarial game between the two, providing more information for the lane detection model, enhancing the model's ability to extract key features, and making the lane detection network pay more attention to lane features.

[0105] Step S3, using the newly constructed data set with the lane line visual information erased to train the constructed adversarial generative network model, so that the model converges to obtain the lane line detection network parameters, that is, the parameters of the generator network model; using the test set to test the trained adversarial generative network model, wherein the discriminator network does not work during the test, to obtain the lane line detection result.

[0106] In some embodiments of the present invention, in this step, the preprocessed new data set is input into the adversarial generative network model for training, and the network is optimized using the AdamW optimizer, and the learning rate is set to 1e -4 , the momentum is set to 0.9, the weight decay rate is 0.05, the batch size for each training is 8, and the number of training rounds is 50. Training is done on a server equipped with an NVIDIA GeForce RTX 4090 (24G video memory).

[0107] Step S4: input the lane line image to be detected into the trained generative adversarial network model to obtain the lane line detection result.

[0108] It is understandable that the discriminator network in the adversarial generative network model does not participate in the work in this step. The discriminator network only works during the training process of the adversarial generative network model. When the adversarial generative network model training is completed, the discriminator network does not participate in the work in actual application.

[0109] In some embodiments of the present invention, a lane line detection system is further provided, which is used to implement the method provided in the above embodiments, and includes the following modules:

[0110] A training module, which trains an anti-generation network model by constructing a road traffic data set without obvious lane line visual information. The anti-generation network model includes a generator network and a discriminator network. The discriminator network and the generator network play an adversarial game. The generator network is used as a lane line detection model, including a backbone feature extraction network based on Swin Transformer, a bidirectional feature pyramid and a bilateral upsampling decoder. The backbone feature extraction network based on Swin Transformer is used to extract multi-scale global features of lane lines through a window-based self-attention method. The bidirectional feature pyramid is used to perform top-down and bottom-up weighted fusion of multi-scale global features. The bilateral upsampling decoder is used to upsample the fused features to obtain the detection results of the lane lines, wherein the image processing technology is used to erase the lane line visual information to construct a road traffic data set without obvious lane line visual information;

[0111] The detection module is used to input the lane line image to be detected into the trained adversarial generative network model to obtain the lane line detection result.

[0112] In some embodiments of the present invention, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a lane line detection method provided in the aforementioned embodiment when executing the computer program.

[0113] In some embodiments of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, a lane line detection method provided in the aforementioned embodiment is implemented.

[0114] The present invention uses the aforementioned embodiments, specifically, by using image processing technology to erase the visual feature information of lane lines in road traffic images, construct a data set without obvious lane line visual information, and based on the data set, SwinTransformer can be used to obtain multi-scale lane line feature maps, and through window-based self-attention, each layer of feature map extracted contains the characteristics of the global context information of the image. Based on the deep learning algorithm, the road traffic image without obvious lane line visual information is used as the input of the lane line detection model, and the extracted features are weighted fused using a bidirectional feature pyramid, and the adversarial game training of the generative adversarial network is introduced to enhance the feature extraction ability of the lane line detection model, thereby realizing the detection of lane lines. The present invention realizes the guarantee of the detection performance of the lane line detection model in the scene with good lane line visual information, and at the same time improves the detection accuracy of the model in the scene without obvious lane visual information, so that the model is suitable for a variety of complex road traffic scenes.

[0115] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A lane line detection method, characterized in that: Input the lane line image to be detected into the trained adversarial generative network model to obtain the lane line detection result; The adversarial generative network model includes a generator network and a discriminator network. During training, the discriminator network and the generator network play an adversarial game. The generator network, as a lane line detection model, includes a backbone feature extraction network based on Swin Transformer, a bidirectional feature pyramid and a bilateral upsampling decoder. The backbone feature extraction network based on Swin Transformer is used to extract multi-scale global features of lane lines through a window-based self-attention method. The bidirectional feature pyramid is used to perform top-down and bottom-up weighted fusion of multi-scale global features. The bilateral upsampling decoder is used to upsample the fused features to obtain lane line detection results. Among them, image processing technology is used to erase the lane line visual information to construct a road traffic dataset without obvious lane line visual information, and the adversarial generative network model is trained by the road traffic dataset without obvious lane line visual information.

2. A lane line detection method according to claim 1, characterized in that: By collecting video data from a car's driving recorder to obtain a road traffic scene graph containing lane line information, the lane line labels are annotated to construct an original dataset, or a road dataset for lane line detection tasks publicly available on the Internet is downloaded as the original dataset, which contains lane lines and their labels; The lane line area of ​​the image in the original data set is erased using content-filling-based image processing technology to construct a new data set without lane line visual information. The training set of the new data set is the component set of the training set of the original data set after the lane lines are erased. The test set of the new data set includes the original test set and the component set after the lane line visual information of the original test set is erased. The training set of the new data set is used to train the generative adversarial network model, and the test set of the new data set is used to test the generative adversarial network model.

3. A lane line detection method according to claim 1, characterized in that: The backbone feature extraction network based on Swin Transformer includes multiple feature extraction layers, which are used to capture local and global information and gradually extract multi-scale features. Each feature extraction layer includes a Swin Transformer Block module. Each Swin Transformer Block module includes a window-based multi-head self-attention and a moving window-based multi-head self-attention. The window-based multi-head self-attention divides the input feature map into multiple windows of equal size and performs self-attention calculation in each window. It can focus on the local structure and detail features of the image, so that the lane line detection model pays more attention to the lane features. The moving window-based multi-head self-attention shifts the window so that the originally independent windows can interact with each other, thereby promoting the integration of local and global contextual information.

4. A lane line detection method according to claim 1, characterized in that: The bidirectional feature pyramid performs bidirectional weighted fusion on the feature maps extracted based on the Swin Transformer backbone network, and constrains the weight range through fast standardized weight fusion. The fast standardized weight fusion formula is: The output after bidirectional weighted fusion is: O=conv(w io ·F i ) Among them, ω i is the initial weight of the i-th input feature map, ∈ is a preset very small value, w j is the weight of the jth input, ω io is the weight of the i-th input after fast normalization weight fusion, F i is the i-th input feature map, conv is convolution, and O is the fused output.

5. A lane line detection method according to claim 1, characterized in that: The bilateral upsampling decoder combines linear interpolation upsampling and deconvolution upsampling operations to upsample the fused features, where the fused features are input into the bilateral upsampling decoder and the output F O for: F O =Bilinear(F * )+Upcov(F * ) In the formula, Bilinear(·) is the linear interpolation upsampling function, Upconv(·) is the deconvolution upsampling, and F * The fused features.

6. A lane line detection method according to claim 1, characterized in that: Construct PatchGAN as a discriminator network, including multiple convolution modules, each of which is used to downsample the input image pair, and finally output a probability prediction map. The value of each point on the probability prediction map represents the probability value prob output by the discriminator network. ij .

7. A lane line detection method according to claim 1, characterized in that: The overall objective function of the adversarial generation network model is expressed as: Where G is the generator network, D is the discriminator network, x is the input lane line image to be detected, y is the label corresponding to the lane line image to be detected, G(·) is the output of the generator network G, D(·) is the output of the discriminator network D, E x，y [·] represents the expected loss function of the discriminator network on the real image pair (x, y), E x，G(x) [·] represents the expected value of the loss function of the discriminator network on the generated image pair (x, G(x)), and V(G, D) is the overall objective function of the adversarial generative network model.

8. A lane detection system, characterized in that: The method for implementing any one of claims 1 to 7 comprises the following modules: A training module, which trains a generative adversarial network model by constructing a road traffic data set without obvious lane line visual information. The generative adversarial network model includes a generator network and a discriminator network. The discriminator network and the generator network play an adversarial game. The generator network is used as a lane line detection model, including a backbone feature extraction network based on Swin Transformer, a bidirectional feature pyramid and a bilateral upsampling decoder. The backbone feature extraction network based on Swin Transformer is used to extract multi-scale global features of lane lines through a window-based self-attention method. The bidirectional feature pyramid is used to perform top-down and bottom-up weighted fusion of multi-scale global features. The bilateral upsampling decoder is used to upsample the fused features to obtain the detection results of the lane lines, wherein the image processing technology is used to erase the lane line visual information to construct a road traffic data set without obvious lane line visual information; The detection module is used to input the lane line image to be detected into the trained adversarial generative network model to obtain the lane line detection result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, a lane line detection method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a lane line detection method according to any one of claims 1 to 7 is implemented.

Images