Night scene lane line detection method

Abstract

The invention relates to the technical field of image enhancement, and particularly provides a night scene lane line detection method, which comprises the following steps: performing enhancement processing on a night image by using a light enhancement curve mapping generator to obtain an enhanced night image, optimizing the brightness of the enhanced night image by using a dual-scale discriminator to obtain an optimized night image, a lane line in an optimized night image is detected by utilizing a lane line detection network, the lane line detection network comprises an encoder backbone network, a space Transform module and a decoder, a feature map of the optimized night image is extracted by utilizing the encoder backbone network, the space information of the feature map is enhanced by utilizing the space Transform module, and the space information of the feature map is extracted by utilizing the decoder. And recovering the original size of the feature map of the enhanced spatial information by using a decoder to obtain a segmented map, and analyzing the existence probability and the number of the lane lines to obtain a lane line detection result. According to the invention, the structure information in the night image can be better captured, the night lane line detection capability is improved, and reliable technical guarantee is provided for night intelligent driving.

Description

Technical Field

[0001] This invention relates to the field of image enhancement technology, specifically providing a method for lane line detection in nighttime scenes. Background Technology

[0002] Lane detection, a core technology in Advanced Driver Assistance Systems (ADAS) and autonomous driving, is crucial for improving vehicle safety. It provides powerful support to drivers by offering precise driving decisions, path planning, and lane keeping functions. Traditional lane detection methods primarily rely on manually extracted low-level features, combined with post-processing algorithms such as Hough transform for lane fitting. However, these methods heavily depend on manual parameter tuning, a process that is not only tedious and time-consuming but also exhibits poor generalization ability. Especially under nighttime lighting conditions, where road scenarios are complex and varied, traditional methods often struggle to cope effectively, leading to a significant drop in lane detection accuracy.

[0003] With the development of deep learning technology, lane detection methods based on convolutional neural networks (CNNs) have gradually become mainstream. These methods can automatically learn the features and shapes of lane lines from large amounts of data, exhibiting stronger adaptability and generalization capabilities. Deep learning-based lane detection methods are mainly divided into two categories: segmentation-based methods and regression-based methods. Segmentation-based methods predict each pixel in an image individually, classifying it as either a lane line or background, thus accurately identifying the shape and location of lane lines. Among these, semantic segmentation methods can achieve high-precision lane detection in complex environments, and therefore have received widespread research and application in recent years. Regression-based methods mainly focus on regressing the overall shape or geometric parameters of lane lines. Lane shape regression methods detect lane lines by regressing their overall shape, typically relying on preset anchor points or keypoints to fit the position and shape of the lane line. These methods are suitable for handling complex lane structures and can effectively identify different lane shapes. Lane parameter regression methods, on the other hand, model lane lines by regressing their geometric parameters, typically using mathematical models such as polynomials to represent lane line curves and extracting global features through geometric modeling. These methods simplify the calculation process and improve inference speed, making them suitable for scenarios where high accuracy is not required but high inference speed is needed.

[0004] In recent years, with the successful application of Transformer models in natural language processing, scholars have begun to introduce them into computer vision, resulting in several Transformer-based lane detection methods. For example, the LSTR method uses a Transformer network to learn complex structural and contextual features, outputting lane shape model parameters through an end-to-end network. The Laneformer method enhances the Transformer model to capture lane shape and semantic features with minimal latency overhead. The HW-Transformer method combines Transformer models, limiting attention to individual rows and columns, and utilizing cross-features to exchange information, forming a lane detection network based on row and column multi-head self-attention.

[0005] However, most existing research focuses on network optimization under normal lighting conditions, neglecting the degradation of road features in nighttime lighting environments. In low-light and uneven lighting conditions, road images often suffer from low lane line visibility, local overexposure, and shadow areas, making feature extraction difficult and impacting the safety and reliability of advanced driver assistance systems (ADAS) or autonomous driving systems. While some studies, such as SIM-CycleGAN, use style transfer data augmentation algorithms to improve the adaptability of lane detection models in nighttime scenes, style transfer techniques struggle to accurately simulate the visual features of roads in real-world nighttime scenarios and cannot effectively address practical challenges such as lighting variations, noise, and complex backgrounds. Although methods like FLLENet improve lane line detection accuracy through image augmentation, the augmentation process introduces noise, causing the original network model to become overly reliant on image augmentation, resulting in unstable detection performance. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides a lane line detection method for nighttime scenes. It employs an encoding-decoding semantic segmentation architecture, integrates a spatial Transformer module with a self-attention mechanism, and utilizes a row and column token encoding strategy to enhance spatial dimension perception capabilities, thereby improving the accuracy of lane line detection in complex nighttime scenes.

[0007] The lane line detection method for nighttime scenes provided by this invention includes: S1: A light enhancement curve mapping generator is established using convolutional layers and iterative mapping layers. The light enhancement curve mapping generator is then used to enhance the nighttime image to obtain an enhanced nighttime image. S2: Utilize a dual-scale discriminator to optimize and enhance the brightness of nighttime images, thereby obtaining optimized nighttime images; S3: Optimize lane line detection in nighttime images using a lane line detection network: The lane line detection network consists of an encoder backbone network, a spatial Transformer module, and a decoder; The encoder backbone network is used to extract and optimize feature maps for nighttime images; Enhance the spatial information of feature maps using the Spatial Transformer module; The original size of the feature map with enhanced spatial information is recovered using the decoder to obtain the segmentation map, and the probability and number of lane lines are analyzed to obtain the lane line detection results.

[0008] Preferably, the light enhancement curve mapping generator includes 7 convolutional layers and 8 iterative mapping layers.

[0009] Preferably, the iterative expression for the iterative mapping layer is: ; in, Represents the x-coordinate of the pixel position. Indicates the number of iterations in the mapping. Indicates the first The output result of the next iteration. This represents the nighttime image to be enhanced, serving as the initial input for the iteration. Represents a group of parameter matrices. .

[0010] Preferably, the dual-scale discriminator includes a global discriminator and a local discriminator, and the method for enhancing the brightness optimization of nighttime images is as follows: By utilizing global and local discriminators, global and local information is used to optimize the parameters of the light enhancement curve.

[0011] Preferably, the encoder backbone network is a ResNet34 pre-trained network.

[0012] Preferably, the spatial Transformer module is a row-column encoding mechanism spatial Transformer network, including a row encoding spatial Transformer module and a column encoding spatial Transformer module.

[0013] Preferably, the decoder includes a bilateral upsampling decoder module and a confidence evaluation module, wherein the bilateral upsampling decoder module comprises... It consists of two bilateral upsampling blocks and one 1×1 convolutional layer. Indicates the number of times the encoder upsamples; The expression for the confidence assessment loss function in the confidence assessment module is as follows: , in, Indicates the number of pixels. Indicates the first The true labels of each sample, Indicates the pixel index. This represents the sigmoid activation function. Indicates the first The network output of each sample.

[0014] Preferably, the bilateral upsampling block is composed of a coarse-grained upsampling branch and a fine-grained upsampling branch connected in parallel, which are used to recover coarse-grained features and fine-grained features, respectively.

[0015] Preferably, the coarse-grained upsampling branch reduces the channel dimension of the input feature map by half through a 1×1 convolutional layer, and expands the resolution to twice the original value using bilinear upsampling; the fine-grained upsampling branch first expands the size of the output feature map of the coarse-grained upsampling branch to twice the original value through a 3×3 transposed convolutional layer, and then reduces its channel dimension by half through a 1×1 convolutional layer.

[0016] Preferably, the results of the coarse-grained upsampling branch and the fine-grained upsampling branch are added pixel by pixel to obtain an upsampled output with half the number of channels and double the resolution.

[0017] Compared with the prior art, the present invention can achieve the following beneficial effects: This invention employs an encoder-decoder semantic segmentation architecture, integrating a spatial Transformer module with a self-attention mechanism. It utilizes a row-column token encoding strategy to enhance spatial dimension perception, improve the computational efficiency of the Transformer module, and increase the accuracy of lane line detection in complex nighttime scenes. The designed nighttime enhancement network can significantly reduce noise generated during the enhancement process while maintaining low computational cost.

[0018] This invention improves the sensitivity of Transformer networks to changes in spatial structure by introducing a weight sharing mechanism. Attached Figure Description

[0019] Figure 1 This is a structural diagram of a spectral enhancement curve mapping generator provided according to an embodiment of the present invention; Figure 2 This is a structural diagram of a dual-scale discriminator provided according to an embodiment of the present invention; Figure 3 This is a diagram of the lane detection network structure provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a bilateral upsampling decoder module provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of a confidence assessment module provided according to an embodiment of the present invention; Figure 6These are local SBR histograms for different algorithms provided in various scenarios according to embodiments of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and do not constitute a limitation thereof. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of the invention. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, some operations related to the invention are not shown or described in the specification. This is to avoid obscuring the core parts of the invention with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined to form various implementations. Furthermore, the order of the steps or actions in the method description can be changed or adjusted in a manner readily apparent to those skilled in the art. Therefore, the various orders in the specification and drawings are merely for the clear description of a particular embodiment and do not imply a mandatory order, unless otherwise stated that a particular order must be followed.

[0022] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0023] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0024] The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] This invention provides a lane line detection method for nighttime scenes, as detailed below: S1: A light enhancement curve mapping generator is established using convolutional layers and iterative mapping layers. The light enhancement curve mapping generator is then used to enhance the nighttime image to obtain an enhanced nighttime image.

[0026] like Figure 1 As shown, the light enhancement curve mapping generator is a module that can improve the quality of nighttime images, using nighttime images as input. In this embodiment of the invention, the architecture of the light enhancement curve mapping generator includes 7 convolutional layers and 8 iterative mapping layers, which work together to achieve efficient image enhancement. In terms of processing flow, the nighttime image first enters a feature extraction stage consisting of 6 3×3 convolutional layers to extract 32-channel parametric feature maps. The convolutional layers directly pass the feature information from shallow layers to deeper layers through a skip connection mechanism, avoiding the feature information loss and gradient vanishing problems caused by increasing the number of layers in traditional deep networks, thus ensuring efficient feature transfer.

[0027] After feature extraction, the nighttime image features enter the channel adjustment stage. A 1×1 point convolution operation is used to reduce the number of channels from 32 to 24, reducing computation and the number of parameters, thus improving the network's efficiency. Subsequently, the Tanh activation function normalizes the adjusted nighttime image features, limiting the output values ​​to the range of -1 to 1, generating a 24-dimensional curve parameter matrix. This generated curve parameter matrix is ​​then split into eight 3-dimensional parameter matrix groups, represented as follows: , The nighttime images are optimized based on a specific iterative formula. In this embodiment of the invention, the iterative formula is expressed as: ; in, Represents the x-coordinate of the pixel position. Indicates the number of iterations in the mapping. Indicates the first The output result of the next iteration. This represents the nighttime image to be enhanced, serving as the initial input for the iteration. Represents a group of parameter matrices. . Figure 1 The formulas involved are respectively: In Obtained by assignment.

[0028] Through this iterative computation method, the light enhancement curve mapping generator can flexibly adjust the brightness value of each pixel, performing fine-tuning according to the brightness requirements of different regions. In low-light areas, brightness can be appropriately increased to enhance detail visibility; in relatively bright areas, over-enhancement is avoided to prevent information loss or noise amplification, resulting in enhanced nighttime images. The optimization strategy of this invention enables the light enhancement curve mapping generator to effectively handle complex lighting changes and noise problems in nighttime scenes, significantly improving image brightness, contrast, and color reproduction, providing high-quality input images for subsequent computer vision tasks.

[0029] S2: Utilize a dual-scale discriminator to optimize and enhance the brightness of nighttime images, thereby obtaining optimized nighttime images.

[0030] like Figure 2 As shown, the enhanced nighttime image is input into a dual-scale discriminator for optimization. In this embodiment of the invention, the dual-scale discriminator includes a global discriminator and a local discriminator. The dual-scale discriminator guides the optimization of the light enhancement curve parameters using global and local information, making the brightness distribution of the enhanced image closer to the natural lighting state, avoiding noise, overexposure, or underexposure in the enhanced image, and maintaining the rationality of brightness changes. Figure 2 In this context, True / False indicates whether the condition is true or false.

[0031] The specific process for enhancing the brightness of nighttime images is as follows: A relative discriminator is used in the global discriminator to estimate the realism of real samples relative to fake samples, thus simplifying the training process of the discriminator and making fake samples closer to real samples. In this embodiment of the invention, real samples are daytime images, and fake samples are enhanced nighttime images. The expression for the relative discriminator is: , , in, Represents the discriminator network. This refers to samples collected from the set of true samples. This refers to samples collected from the fake sample set. This represents the Sigmoid function. This represents the probability that a real sample looks more realistic than a generated sample generated by the light enhancement curve mapping generator. This indicates the probability that the generated samples from the light enhancement curve mapping generator look more realistic than the real samples. express The image is obtained by sampling from the fake sample set. This represents the mean. express The image is sampled from a set of true samples (well-lit).

[0032] Substituting the expression for the relative discriminator into the loss function of the Least Squares Generative Adversarial Networks (LSGAN) yields the loss functions for the global discriminator and the light enhancement curve mapping generator. The loss function expression for the global discriminator is: , in, This represents the global discriminator loss function. This represents the mean.

[0033] The global generator loss function is expressed as follows: , in, This represents the global generator loss function.

[0034] Local discriminator loss function The expression is: , in, express It is the mean value sampled from real sample image patches. express It is the mean value sampled from fake sample image patches. This represents the mean.

[0035] Local generator loss function The expression is: , The generator loss is obtained by adding the loss function of the global generator to the loss function of the local generator. : .

[0036] The discriminator loss is obtained by adding the loss function of the global discriminator to the loss function of the local discriminator. : .

[0037] During training, based on the expression of the generator loss... and discriminator loss The expression is used to alternately update network parameters, thereby improving the performance of the network model.

[0038] The above process optimizes the visual effects of nighttime images. The nighttime images output by the generator can be considered as optimized nighttime images.

[0039] S3: Detect and optimize lane lines in nighttime images using a lane line detection network: The lane line detection network includes an encoder backbone network, a spatial Transformer module, and a decoder; the encoder backbone network is used to extract and optimize feature maps of the nighttime images; the spatial Transformer module is used to enhance the spatial information of the feature maps; the decoder is used to restore the original size of the feature maps with enhanced spatial information, and the probability and number of lane lines are analyzed to obtain the lane line detection results.

[0040] like Figure 3 As shown, the optimized image output in S2 is input into the lane detection network to detect lane lines in the optimized image. In this embodiment of the invention, the lane detection network includes an encoder backbone network, a spatial Transformer module, and a decoder. The encoder backbone network is a ResNet34 pre-trained network. After the optimized network is input into the lane detection network, it first passes through the encoder backbone network for feature extraction, and then is input into the spatial Transformer module for information augmentation. It should be noted that in visual models, images are typically segmented into multiple small blocks or regions, each called a "token." These tokens are the basic processing units of the image. In this embodiment of the invention, the "token" is a pixel.

[0041] The Transformer module is a self-attention mechanism that first multiplies the input vector sequence by a weight matrix to obtain three sub-vector sequences, which are the query vector and the input vector. Index vector and content vector Then, the three sub-vectors are input into the multi-head self-attention module for dimensionality splitting, and attention operations are performed separately. The formula is as follows: , in, Represents the query vector Index vector and content vector Attention computation, Represents the normalized exponential function, Represents an index vector The dimension is used as a scaling factor. Finally, the calculation result is concatenated and input into a multilayer perceptron to obtain the output of the Transformer module.

[0042] Visual Transformer networks, composed of Transformer modules, exhibit a significant advantage over the limited receptive field of convolutional neural networks when processing slender objects with strong spatial continuity, such as lane lines. However, the image patch encoding mechanism used by visual Transformer networks ignores the structural relationships between rows and columns in the image, leading to decreased sensitivity to changes in the row and column structure of slender, continuous targets like lane lines, resulting in the loss of detailed information. In long-distance imaging or nighttime scenes, lane line details are not obvious, the pixel ratio is low, features are sparse, and visual cues become incomplete or even completely disappear, further increasing the difficulty of detection.

[0043] In view of the above problems, the spatial Transformer module provided in this embodiment of the invention is a row-column encoding mechanism spatial Transformer network, including a row encoding spatial Transformer module and a column encoding spatial Transformer module. It uses a row and column token encoding mechanism to replace the image block encoding mechanism, and combines a weight sharing mechanism and a self-attention mechanism to construct a spatial Transformer module that is sensitive to changes in spatial structure. It makes full use of the prior information between lane line rows and columns, improves the integration capability of global features and local detail features of the target, enhances the adaptability to slender targets, effectively captures spatial information at a distance, and reduces irrelevant attention operations between background information.

[0044] The feature map of the optimized nighttime image extracted by the encoder backbone network is used as input to the spatial Transformer module. This feature map is represented as... ,in, These represent the number of channels, height, and width of the feature map, respectively. To reduce computational complexity, the feature map is first... The feature map is mapped using a 1×1 convolutional kernel. Based on the odd and even rows, it is divided into two sub-feature maps: odd rows and even rows. and Expand these two feature maps row by row according to each channel to form two feature vector matrices. and The feature vector is obtained by performing self-attention computation with shared odd and even row weights through two identical Transformer modules. and Then, odd-even row feature reconstruction is performed to obtain the feature map. and The feature map is reassembled by piecing together the odd and even rows. This weight-sharing mechanism allows the spatial Transformer module to possess the inherent spatial positional bias of CNNs, compensating for the Transformer's deficiency in perceiving spatial positional information. The above calculation process is shown in the following equation: , , , , , , in, Represents a 1×1 convolution. This indicates a row-by-row feature expansion operation. and These represent the odd and even row partitioning operations, The Transformer module represents shared weights. Indicates the reshaping of odd and even row features. This indicates channel cascading.

[0045] feature map Repeat the above process according to the odd and even columns to obtain the feature map. To prevent spatial information loss during the attention processing, finally, and The layers are cascaded and passed through a 1×1 convolution kernel to obtain the output of the spatial Transformer module, resulting in a feature map with enhanced spatial information.

[0046] The feature map with enhanced spatial information from the output of the spatial Transformer module is input into the decoder. The decoder then upsamples the feature map to restore its original size, resulting in a segmentation map. For example... Figure 4 and Figure 5 As shown, in this embodiment of the invention, the decoder includes a Bilateral Up-Sampling Decoder (BUSD) module and a confidence assessment module.

[0047] Commonly used bilinear upsampling can lead to insufficiently refined output results when scaling. To address this, this invention constructs a bilateral upsampling decoder module, using feature maps with strong spatial information as input. The bilateral upsampling decoder module consists of... It consists of two bilateral upsampling blocks and one 1×1 convolutional layer. This indicates the number of upsampling operations performed by the encoder. After each upsampling, the number of feature map channels is reduced to half, and the resolution is doubled. After upsampling, the output is a feature map with the same resolution as the night image. Then, a 1×1 convolutional layer is used to adjust the number of channels to l+1. In the former, l represents the number of predictable lane lines, and in the latter, 1 represents the background.

[0048] The bilateral upsampling block consists of a coarse-grained upsampling branch and a fine-grained upsampling branch connected in parallel. These branches are used to recover coarse-grained and fine-grained features, respectively. The coarse-grained upsampling branch reduces the channel dimension of the input feature map by half using a 1×1 convolutional layer, and then expands the resolution to twice its original value using bilinear upsampling. The fine-grained upsampling branch first expands the size of the output feature map from the coarse-grained upsampling branch to twice its original value using a 3×3 transposed convolutional layer, and then reduces its channel dimension by half using a 1×1 convolutional layer. Finally, the results from the coarse-grained and fine-grained upsampling branches are added pixel-by-pixel to obtain an upsampled output with half the number of channels and doubled resolution, i.e., the segmentation map.

[0049] Furthermore, the total loss function of the lane detection network in this embodiment of the invention. It includes two loss functions: a segmentation loss function and a splitting loss function. and confidence assessment loss function Then we have: , in, This represents the weighting coefficient between the balanced segmentation loss function and the existence prediction loss. In this embodiment of the invention, , .

[0050] Specifically, the segmentation loss function is the Dice loss of the semantic segmentation map and mask label of the lane detection network, and its expression is: , in, This represents the probability map output by the decoder. Indicates the true mask. This is used to avoid the denominator being zero.

[0051] The confidence evaluation loss is the binary cross-entropy loss between the lane detection network segmentation map and the ground truth label, and its expression is: , in, Indicates the number of pixels. Indicates the first The true labels of each sample, Indicates the pixel index. This represents the sigmoid activation function. This represents the result output by the confidence assessment module, i.e., the first... The network output of each sample.

[0052] To more clearly illustrate the method of this invention and to verify the effectiveness of the proposed lane detection method for nighttime scenes combining light enhancement curves and spatial Transformer networks, this invention also used the publicly available datasets Tusimple and CULane for training and experimentation. The Tusimple dataset, developed by TuSimple, provides stable lighting conditions and clearly visible lane lines. The dataset contains 3626 training images and 2782 test images. Each image set contains 20 frames, with only the 20th frame providing lane line coordinate labels and ground truth labels. This invention uses only labeled images for training and testing. Segmentation labels are generated from the coordinate labels to train the segmentation network, while the ground truth labels are used to train the lane line confidence evaluation module.

[0053] The CULane dataset contains complex road traffic scenes in a city in China, comprising 133,235 images, of which 88,880 are the training set, 9,675 are the validation set, and 34,680 are the test set. The dataset samples include various traffic scenarios such as normal, congested, nighttime, no lane markings, shadows, arrows, and glare.

[0054] In training the night enhancement network, this invention selected 4000 nighttime road scene images from the CULane dataset and divided them into training and test sets in a 7:3 ratio. Simultaneously, 1016 images taken under good lighting conditions were selected from existing literature; these images served as real samples (images under good lighting conditions) and fake samples (nighttime images), and were used in an unpaired manner to train the night enhancement algorithm.

[0055] The experimental environment configuration for this invention is shown in Table 1.

[0056] Table 1 Experimental Parameter Configuration

[0057] The evaluation metrics for the Tusimple dataset, including accuracy (ACC), false positive rate (FP), and false negative rate (FN), are calculated as follows: , , , in, This indicates the number of correctly predicted lane pixels. This represents the total number of pixels in the lane. Indicates the pixel index. Represents the image index. This indicates the number of lane markings that were incorrectly detected. This indicates the number of lane markings detected. This indicates the number of lane markings that were not detected. This indicates the total number of lane markings.

[0058] The CULane dataset is passed through The value is used to evaluate lane detection performance. The value is the weighted harmonic mean of precision and recall. (Calculation) Value, precision and recall rate The formula is as follows: , , , in, This represents the number of actual lane markings and the number predicted as lane markings. This represents the number of actual background elements that were predicted as lane markings. This represents the number of pixels that are actually lane markings but are predicted as background.

[0059] Used to evaluate whether an image possesses the statistical characteristics that a natural, realistic image should have. A lower score indicates a more natural visual effect. The expression is as follows: , , in, Represents the image to be tested The mean, Represents reference image The mean, Indicates the dynamic range of an image. and Indicates the regulating factor. Represents the image to be tested and reference image covariance, Structural similarity is a metric used to evaluate image quality. Represents the image to be tested variance Represents reference image The variance.

[0060] To evaluate the enhancement effects of different nighttime enhancement algorithms on local lane markings, this invention uses local signal-to-background ratio. (Signal to Background Ratio, SBR) is used to measure the difference between lane lines and the background. The calculation formula is as follows: , in, This represents the average grayscale value of the background within a local area of ​​the lane lines. This represents the average grayscale value of the signal within a local area of ​​the lane markings.

[0061] To verify the effectiveness of the night-time enhancement network LEC-GAN of this invention, it was compared with typical enhancement networks LIME, BIMEF, EnlightenGAN, and Zero-dce. Objective evaluation data of the experimental results are shown in Table 2.

[0062] As shown in Table 2, the LEC-GAN network proposed in this invention outperforms current mainstream nighttime image enhancement networks in all evaluation metrics. The LEC-GAN network achieved the lowest NIQE score, indicating optimal naturalness of the enhancement result. In the SSIM metric, the LEC-GAN network improved by 104.97% compared to the Zero-dce network, demonstrating that introducing a global discriminator to optimize the light enhancement curve parameters can improve the overall image quality. The inference time of the LEC-GAN network is 8.08% of that of the EnlightenGAN network, indicating a significant advantage in reducing inference time. Since the convolutional structure used to generate the light enhancement curve parameters in the generator part of the LEC-GAN network designed in this invention is consistent with the structure of the zero-reference loss Zero-dce network, it retains the low latency advantage of Zero-dce. However, the global-local adversarial structure introduced in this invention can deeply mine image features, reduce noise in the enhanced image, and improve the naturalness of the image.

[0063] Table 2 Objective evaluation data of nighttime enhanced network

[0064] In the visualization results, Scene 1 and Scene 2 represent nighttime road scenes with uniform illumination; Scene 3 and Scene 4 represent nighttime road scenes with uneven illumination; and Scene 5 represents a nighttime road scene with insufficient illumination. The visualization results show that the images enhanced by LIME and Zero-dce networks have higher overall brightness, but introduce a large amount of noise, severely affecting image clarity and causing the loss of many details. Zero-dce exhibits local overexposure when processing scenes with uneven illumination. BIMEF, EnlightenGAN, and the enhancement algorithm proposed in this invention can effectively maintain low noise and have good local strong light suppression effects, but the image resulting from the enhancement network of this invention has higher brightness and a more natural visual experience. This indicates that the introduction of a generative adversarial network structure and an iterative optimization algorithm for light enhancement curve parameters in the nighttime enhancement network of this invention can sharpen key details such as lane line edges and road surface textures while preserving the overall brightness uniformity of the image, avoiding feature distortion and noise amplification.

[0065] To more accurately evaluate the performance of various nighttime enhancement algorithms in distinguishing lane markings from the background in local areas, this invention introduces the signal-to-background ratio (SBR) as an evaluation metric. This invention selects four different nighttime road scene images: Scenes 1 and 2 are well-lit road scene images, while Scenes 3 and 4 are poorly lit road scene images. These scene images contain SBR regions. Figure 6 As shown, the bar charts represent the local SBR (Signal-to-Background Ratio) of different algorithms in various scenarios, with the vertical axis representing the signal-to-background ratio.

[0066] from Figure 6 As can be seen, the enhancement network proposed in this invention achieved optimal values ​​in all test scenarios. Other algorithms, due to factors such as noise interference and local overexposure, did not significantly improve the SBR value compared to the original image in some scenarios, and even decreased it. This indicates that the introduction of a local discriminator in the algorithm of this invention to optimize the light enhancement curve parameters can eliminate the influence of noise and improve the difference between the lane lines and the background area.

[0067] To analyze the performance of the lane detection network of this invention, the lane detection network STLane of this invention was compared with the segmentation-based methods SAD, SCNN, RESA, LNet and HWlane, the shape regression-based methods PointLaneNet, LaneATT, E2E, UFast, CLRNet and CondLaneNet, and the parametric regression-based methods LSTR, BézierLaneNet, FastDraw and PolyLaneNet in the Tusimple and CULane datasets.

[0068] The detection results of the Tusimple dataset are shown in Table 3. Since the Tusimple dataset has a relatively simple scene and clear and standardized lane lines, all methods have achieved good detection results. The difference in detection accuracy is not significant and tends to be saturated. The accuracy of the STLane method of this invention is the highest among all the comparison methods.

[0069] Table 3 Detection results of the Tusimple dataset

[0070] In scenarios 1, 2, and 3, vehicle occlusion exists. The RESA-ResNet34 method, lacking sufficient contextual reasoning ability, cannot completely predict lane lines within the occluded area. HWlane, while able to predict lane lines in the occluded area as a whole, exhibits varying degrees of interruption in its prediction results due to its inability to effectively utilize the continuity features of lane lines. In contrast, the STLane proposed in this invention provides continuous lane lines and accurate boundary prediction. In scenarios 4 and 5, lane-line interference exists, which RESA-ResNet34 and HWlane incorrectly identify as lane lines. However, the STLane detection results of this invention do not show any misdetected lane lines. This is because the spatial Transformer network designed in this invention, with its row and column token encoding mechanism and weight sharing mechanism, enhances the sensitivity to spatial structure changes and extracts distant visual cues, eliminating misjudgments and completely predicting lane lines.

[0071] The experimental results on the CULane dataset are shown in Table 4, demonstrating the nighttime performance of the lane detection network of this invention. The value was 75.4%, an improvement of 9.9% compared to the backbone network; overall scenario The value was 77.2%, an improvement of 7.3% compared to the backbone network.

[0072] In nighttime and shadowy scenes, the STLane network of this invention achieved the highest performance compared to all comparable methods. Value. In the overall scenario, the overall STLane method of this invention... The value is higher than that of the semantic segmentation method and the lane line parameter regression method, and is close to the existing best typical lane line detection method CLRNet (shape regression method).

[0073] Table 4 Objective evaluation index results of the CULane dataset

[0074] To verify the performance of different night-time enhancement algorithms combined with different lane detection networks, this invention selected two night-time scene images for a visual comparison experiment. The LIME and Zero-dce night-time enhancement algorithms showed high brightness but contained significant noise, leading to severe missed detections in all lane detection networks. This indicates that even with high brightness, substantial noise can severely impact the accuracy of lane detection networks. In contrast, the BIMEF and EnlightenGAN night-time enhancement algorithms had less noise but lower brightness, resulting in varying degrees of lane line interruptions in the detected lanes. The detection network of this invention showed almost no lane line interruptions, while the RESA-ResNet34 results showed missed detections.

[0075] In the results of the LEC-GAN night enhancement algorithm of this invention, no lane line was missed or interrupted in the results of all lane line detection networks; due to the interference of roadside trees, the lane line ends in the results of RESA-ResNet34 and HWlane methods showed large errors; in the results of STLane of this invention, the lane lines were complete and continuous.

[0076] Due to the simple road conditions, the detection results of different lane detection networks are not significantly different under the same night enhancement algorithm. Therefore, only the impact of different night enhancement algorithms on the detection network is analyzed. In the LIME and Zero-DCE enhancement algorithms, a large amount of noise still appears in the enhancement results, leading to a relatively serious problem of missed detections. Due to insufficient lighting in the original image, the brightness of the enhancement results of EnlightenGAN and BIMEF is not high. Although the lane lines in the detection results are relatively complete, missed detections still occur. This indicates that noise has a greater impact on the performance of lane detection networks than insufficient light. However, under the LEC-GAN enhancement network of this invention, all lane detection networks can detect all lane lines relatively completely and accurately.

[0077] To further explore the impact of different nighttime enhancement network algorithms on the lane detection network STLane of this invention, STLane was used to perform detection inference on images processed by different enhancement algorithms, and the results were visualized by combining attention heatmaps. Table 5 shows the objective evaluation data of the CULane nighttime test set.

[0078] The attention heatmap reveals significant missed detections in the unenhanced images. The LIME+STLane, BIMEF+STLane, Zero-DCE+STLane, and EnlightenGAN+STLane algorithms introduce noise or exhibit localized overexposure, leading to discontinuities and deviations in lane lines. Analysis of the attention heatmap shows that the proposed STLane network exhibits lower attention to lane line regions processed by other enhancement algorithms, while showing higher attention and accuracy to the proposed LEC-GAN. This demonstrates that the enhancement algorithm proposed in this invention can improve the attention given to lane line detection algorithms.

[0079] As can be seen from Table 5, the combination of the night enhancement network and the lane detection network of the present invention, LEC-GAN+STLane, achieved the highest [performance / optimization]. In terms of accuracy and recall, it boasts the second-highest recall rate (only 0.04% lower than EnlightenGAN+STLane). LEC-GAN+STLane's... The value is 11.09% higher than ResNet34; and 1.16% higher than STLane without the nighttime enhancement network.

[0080] Table 5. Objective evaluation metrics of nighttime images in the CULane dataset after combining STLane with various nighttime enhancement networks.

[0081] Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0082] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for detecting lane lines in nighttime scenes, characterized in that, include: S1: A light enhancement curve mapping generator is established using convolutional layers and iterative mapping layers. The light enhancement curve mapping generator is used to enhance the night image to obtain an enhanced night image. S2: Optimize the brightness of the enhanced night image using a dual-scale discriminator to obtain an optimized night image; S3: Detect lane lines in the optimized nighttime image using a lane line detection network: The lane line detection network includes an encoder backbone network, a spatial Transformer module, and a decoder; The encoder backbone network is used to extract feature maps from the optimized nighttime image; The spatial information of the feature map is enhanced using the spatial Transformer module. The original size of the feature map that enhances spatial information is recovered using the decoder to obtain a segmentation map, and the probability and number of lane lines are analyzed to obtain lane line detection results.

2. The lane line detection method for nighttime scenes as described in claim 1, characterized in that, The light enhancement curve mapping generator includes 7 convolutional layers and 8 iterative mapping layers.

3. The lane line detection method for nighttime scenes as described in claim 1, characterized in that, The iterative expression for the iterative mapping layer is: ； in, Represents the x-coordinate of the pixel position. Indicates the number of iterations in the mapping. Indicates the first The output result of the next iteration. This represents the nighttime image to be enhanced, serving as the initial input for the iteration. Represents a group of parameter matrices. .

4. The lane line detection method for nighttime scenes as described in claim 1, characterized in that, The dual-scale discriminator includes a global discriminator and a local discriminator, and the brightness optimization method for enhancing nighttime images is as follows: The global discriminator and local discriminator are used to guide the optimization of light enhancement curve parameters using global and local information.

5. The lane line detection method for nighttime scenes as described in claim 1, characterized in that, The encoder backbone network is a ResNet34 pre-trained network.

6. The lane line detection method for nighttime scenes as described in claim 1, characterized in that, The spatial Transformer module is a row-column encoding mechanism spatial Transformer network, including a row encoding spatial Transformer module and a column encoding spatial Transformer module.

7. The lane line detection method for nighttime scenes as described in claim 1, characterized in that, The decoder includes a bilateral upsampling decoder module and a confidence evaluation module. The bilateral upsampling decoder module consists of... It consists of two bilateral upsampling blocks and one 1×1 convolutional layer. Indicates the number of times the encoder upsamples; The expression for the confidence assessment loss function in the confidence assessment module is as follows: ， in, Indicates the number of pixels. Indicates the first The true labels of each sample, Indicates the pixel index. This represents the sigmoid activation function. Indicates the first The network output of each sample.

8. The lane line detection method for nighttime scenes as described in claim 7, characterized in that, The bilateral upsampling block is composed of a coarse-grained upsampling branch and a fine-grained upsampling branch connected in parallel, which are used to recover coarse-grained features and fine-grained features, respectively.

9. The lane line detection method for nighttime scenes as described in claim 8, characterized in that, The coarse-grained upsampling branch reduces the channel dimension of the input feature map by half through a 1×1 convolutional layer and expands the resolution to twice the original value using bilinear upsampling. The fine-grained upsampling branch first expands the size of the output feature map of the coarse-grained upsampling branch to twice the original value through a 3×3 transposed convolutional layer, and then reduces its channel dimension by half through a 1×1 convolutional layer.

10. The lane line detection method for nighttime scenes as described in claim 9, characterized in that, The results of the coarse-grained upsampling branch and the fine-grained upsampling branch are added pixel by pixel to obtain an upsampling output with half the number of channels and double the resolution.

Images