A lane line detection method, device and storage medium in real-time infrared scene

Abstract

A method, device, and storage medium for lane line detection in real-time infrared scenes utilizes an infrared camera to capture real-time infrared scene images. The method uses a CSCE-Resnet18 backbone network optimized by a Cross-Scale Channel Enhancement (CSCE) module to extract local and global features from the images, spatially and collaboratively fuse multi-scale contextual information, and utilizes the fused, refined information to enhance lane line detection capabilities. Ultimately, lane line detection is treated as a holistic regression of the prior lane lines using the pre-set lane line knowledge in the feature map, yielding the specific location information for each lane line. While maximizing the accuracy of lane line detection in infrared scenes, the present invention also ensures the real-time nature of the algorithm, enabling deployment in practical applications.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent video image processing, relates to 2D lane line detection on roads, and is a lane line detection method, device and storage medium in real-time infrared scenes. Background Art

[0002] With the rapid advancement of artificial intelligence and automotive technologies, lane detection technology, a key component of intelligent transportation systems, has expanded beyond autonomous driving to encompass intelligent traffic monitoring, driver assistance systems, and road maintenance. Lane detection helps vehicles and traffic management systems accurately identify lane markings, ensuring safe lane travel and providing critical navigation information for various systems.

[0003] In recent years, numerous researchers have devoted significant effort to the research of lane detection technology, proposing numerous advanced detection methods based on visible light. Typical detection models include SCNN, PolyLaneNet, UFSD, and LineCNN. While these methods perform well in visible light scenarios, they are less applicable under infrared and other non-standard lighting conditions. In reality, driving and traffic management environments are often complex and ever-changing. Factors such as fluctuating sunlight intensity, low light conditions at night, shadows cast by vehicles and trees, and worn lane markings all pose challenges to the accuracy and stability of lane detection.

[0004] In various traffic applications, systems must be able to accurately detect lane markings under varying lighting conditions. However, traditional visible light detection systems face significant challenges at night or in adverse weather conditions such as fog, rain, and snow. In these situations, the performance of visible light scene models degrades significantly, resulting in reduced reliability of lane detection. To address the limitations of traditional lane detection in complex environments, infrared cameras, with their advantages of illumination independence and strong penetration, have gradually become one of the key technologies in traffic systems. Infrared cameras can provide clear images in almost all lighting conditions, including complete darkness, thus overcoming the limitations of nighttime and low-visibility environments. This allows both vehicles and traffic management systems to reliably detect lane markings when operating in low light or at night.

[0005] Research on lane detection based on infrared images aims to address the shortcomings of visible light image detection, thereby improving the perception and reliability of various transportation systems in complex environments. For example, infrared imaging can clearly capture lane markings at night, even in low light conditions or with interference from other light sources. By integrating infrared imagery with advanced image processing algorithms, intelligent transportation systems can reliably identify lane markings in a wider range of conditions, improving overall driving safety. Summary of the Invention

[0006] The problem to be solved by the present invention is: using infrared cameras to solve the technical difficulties of lane line detection in poor lighting conditions at night, low visibility, bad weather, various shadows, etc., and proposes a lane line detection method in real-time infrared scenes based on a cross-scale channel information enhancement mechanism. While maximizing the accuracy and stability of lane line detection, the real-time nature of the method is guaranteed, and it can be used for practical application deployment.

[0007] The technical solution of the present invention is: a real-time lane detection method in infrared scenes, which constructs a ResNet-based lane detection model to detect lane position in infrared scenes. The lane detection model uses an improved ResNet backbone network to extract a feature map containing local and global features from infrared images. Based on the lane prior knowledge pre-set in the feature map, lane detection is regarded as a holistic regression of the prior lane. The feature map is input into two parallel fully connected layers, one for classification and the other for regression, to generate a final prediction and obtain lane position information in the infrared scene.

[0008] The improved ResNet backbone network introduces a cross-scale channel collaborative fusion module in the residual block, fuses the original feature X and the residual feature Y of each residual block of ResNet to obtain the initial feature U, and then enhances the CSCE module based on cross-scale channel information to obtain a refined feature map U′ with local and global features. The original feature X and the residual feature Y are then combined with the refined feature map U′ through the Sigmoid activation function to achieve cross-scale channel collaborative fusion. Multi-scale context information is spatially collaboratively fused as the final residual output to obtain a feature map containing local and global features.

[0009] The cross-scale channel information enhancement module CSCE is divided into two branches. One branch obtains the channel information of the global feature through global average pooling, and the other branch obtains the channel information of the local feature. The two sets of channel information features are spliced ​​and processed by the Sigmund function nonlinear activation to output the refined feature map. Specifically, the input feature is U, U' is the feature after processing by the CSCE module, and the channel information of the local feature is L(U). The expression is as follows:

[0010] L(U)=B(Conv2(δ(B(Conv1(U)))))

[0011] Among them, Conv1 represents a 1*1 point convolution that reduces the number of channels of the input feature map U to the input B represents the BN batch normalization layer, δ represents the ReLU activation function, and the number of channels is restored to the same as the original input channels through Conv21*1 point convolution. r is the channel scaling ratio;

[0012] The channel information of the global feature is G(U), which is expressed as follows:

[0013] G(U)=B(Conv2(δ(B(Conv1(Gpool(U))))))

[0014] Among them, Gpool represents global average pooling;

[0015] The obtained local feature channel information L(U) and global feature channel information G(U) are subjected to a nonlinear activation function to obtain feature channel weights, and the output U′ is obtained after performing an attention operation on the input feature U:

[0016]

[0017] Among them, σ represents the Sigmoid nonlinear activation function, that is, the weight normalization process, Indicates the addition of the corresponding elements of the two feature maps, Represents the multiplication of corresponding elements of two feature maps.

[0018] Furthermore, the lane line detection model obtains the position information of the lane line in the real-time infrared scene, including the following steps:

[0019] Step 1: Collect real-time monitoring video from the infrared camera to obtain real-time infrared road images. Use OpenCV to read the video images as input for the lane detection model.

[0020] Step 2: Preprocess the input image by performing a random affine transformation on the image while performing a random horizontal flip. The random affine transformation includes translation, rotation, and scaling. During preprocessing, the lane lines in the image are extended to the image boundaries.

[0021] Step 3: The preprocessed infrared lane marking image is input into the improved ResNet backbone network. The input image passes through the convolution layer, batch normalization layer, and maximum pooling layer, and then enters the convolution module composed of four improved residual blocks to extract features. Finally, the features are output through global average pooling and a fully connected layer to obtain a feature map containing local and global features.

[0022] The improved residual block introduces a cross-scale channel collaborative fusion module. The output feature of the ResNet residual block after two 3*3 convolutions is X, and the feature of the residual connection is Y. Y is a shallow feature. The original feature X and the residual feature Y are first fused. After the CSCE module, a refined feature map U' with local and global features is obtained. U' is activated by the Sigmoid function, and the output value is between 0 and 1. The original feature X and the residual feature Y are fused across scale channels to obtain the feature Z. The cross-scale channel collaborative fusion expression is as follows:

[0023]

[0024] Among them, M represents the cross-scale channel information enhancement module CSCE,

[0025] The feature Z is then output through global average pooling and a fully connected layer to obtain a feature map containing local and global features;

[0026] Step 4: The feature map containing local and global features is input into two parallel fully connected layers, one for classification and one for regression, which produce the final prediction to determine the lane line position.

[0027] The present invention also provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the lane line detection model of the above-mentioned lane line detection method in real-time infrared scenes, and is used for lane line detection in real-time infrared images.

[0028] The present invention also provides a computer-readable storage medium, which stores at least one instruction or at least one program. When the at least one instruction or at least one program is executed, the lane line detection model of the lane line detection method in the above-mentioned real-time infrared scene is implemented.

[0029] The present invention achieves real-time automatic detection of road lane markings using infrared cameras. In real-world scenarios, conditions such as inclement weather and insufficient nighttime illumination pose significant challenges to lane marking detection. While some existing infrared image-based lane marking detection schemes can improve detection accuracy and real-time performance by establishing prior knowledge of lane markings and incorporating global information, existing infrared lane marking detection methods still face challenges when handling complex scenes. For example, they lack the ability to extract refined features from low-contrast infrared images, particularly when lane markings are obscured or subject to ambient noise, which can reduce detection effectiveness. Furthermore, existing methods lack the ability to fully exploit contextual information in infrared images by integrating feature fusion and multi-scale information processing. This method addresses challenges such as low contrast and the lack of single image features in infrared images by introducing a cross-scale channel collaborative fusion and cross-scale channel information enhancement module. This module extracts local and global features from the image and spatially integrates multi-scale contextual information, improving the model's ability to detect lane markings in infrared scenes. The improved ResNet18 network maximizes the accuracy of lane detection within an acceptable time cost increase, with a detection speed of up to 105 FPS. The model meets real-time requirements and can be deployed in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of a lane line detection method in a real-time infrared scene based on a cross-scale channel information enhancement mechanism of the present invention.

[0031] Figure 2 This is a flow chart of the improved ResNet18 network of the present invention.

[0032] Figure 3 Schematic diagram of the residual block of the improved ResNet18 network in the present invention.

[0033] Figure 4 This is a flowchart of the cross-scale channel collaborative fusion process in the present invention.

[0034] Figure 5 This is a flowchart of the cross-scale channel information enhancement module CSCE in the present invention.

[0035] Figure 6 This is Example 1 of the present invention applied to lane line detection in an infrared camera scenario.

[0036] Figure 7 This is the second embodiment of the present invention applied to lane line detection in an infrared camera scenario. DETAILED DESCRIPTION

[0037] The present invention combines image enhancement, cross-scale channel collaborative fusion and deep learning methods to significantly improve the accuracy of lane line detection and recognition in infrared camera scenarios while maintaining the real-time performance of the model. Considering that the most advanced methods for lane line detection are currently designed based on visible light scenarios, they are not suitable for real-time detection of lane lines in infrared scenarios. The present invention uses an improved lightweight ResNet18 model to extract feature information of lane lines in images, and adopts a cross-scale channel information enhancement module (CSCE, Cross-Scale Channel Enhancement) to improve the model's attention to different channels. Refined features are obtained through cross-scale channel collaborative fusion, which can improve the accuracy and stability of the model's lane line detection in infrared scenarios. Classification and regression are used to finally obtain the position information of the lane line target, which makes up for the shortcomings of the visible light lane line detection algorithm, thereby improving the perception ability of the intelligent transportation system in complex environments.

[0038] The schematic flow diagram of the present invention is as follows Figure 1 shown.

[0039] The specific implementation steps of the infrared camera view lane line detection are as follows:

[0040] Step 1: Read in an infrared road image containing lane markings. Capture real-time monitoring video from an infrared camera to obtain a real-time infrared road image. Use OpenCV to read the video image as input for the lane detection model. The lane marking length in the video must occupy at least 30% of the image height, and the lane marking pixel resolution must be at least 90*90.

[0041] Step 2: Preprocessing of the read-in image. Since lane lines are long and thin structures and account for a small proportion of pixels in the image, deep learning-based networks can easily overfit the training set, resulting in poor performance on the validation set. In order to prevent this phenomenon and improve the generalization ability of the model, the present invention adopts an image enhancement method that performs random affine transformation while performing random horizontal flipping. Affine transformation includes translation, rotation, and scaling. In addition, in order to maintain the complete lane line structure in the road image captured by the camera, the lane lines in the image are extended to the image boundary during preprocessing.

[0042] Step 3: The preprocessed image is input into the improved ResNet backbone network. This embodiment is based on the ResNet18 backbone network and is called the CSCE-ResNet18 network. The improved ResNet18 network includes a convolution layer, a batch normalization layer, and a maximum pooling layer. It then enters a convolution module composed of four improved residual blocks to extract features. Finally, the features are output through global average pooling and a fully connected layer to obtain a feature map. In this embodiment, through a convolution of size 7*7, a step size of 2 and a maximum pooling layer of size 3*3, a step size of 2, the input image will become a feature map of size 56×56, which greatly reduces the amount of calculation. The four improved residual blocks connected in series are used to extract the lane line features of the image, and finally pass through the output layer, including the global average pooling layer, the fully connected layer and the output layer Softmax activation function, to output the recognition result.

[0043] The present invention improves the residual block of ResNet18. The improved residual block consists of two 3*3 convolutional layers and a skip connection. The lane feature information is extracted by stacking the intermediate residual blocks and a cross-scale channel collaborative fusion module is introduced. The improved residual block is as follows Figure 3 shown.

[0044] The cross-scale channel collaborative fusion module performs cross-scale channel collaborative fusion on the original features and residual features of each residual block of ResNet. The output feature of the residual block after two 3*3 convolutions is X, and the feature of the residual connection is Y. Y is a shallow feature with a larger receptive field. The expression of cross-scale channel collaborative fusion is as follows:

[0045]

[0046] Among them, M represents the cross-scale channel information enhancement CSCE module, which first performs initial feature fusion on the two input features X and Y, and then passes the obtained initial feature U through the CSCE module to obtain a refined feature map U' with local and global properties. U' passes through the Sigmoid activation function, and the output value is between 0 and 1. Then, a weighted average is performed on X and Y, and the network is trained to determine their respective weights. The present invention adopts a cross-scale channel collaborative fusion mechanism for the X and Y features of the residual block, rather than the commonly used simple addition or concat fusion method. It pays more attention to those channels with obvious lane line features, and fully explores the contextual information in the infrared image to improve the model detection performance. The cross-scale channel collaborative fusion flow chart is as follows: Figure 4 shown.

[0047] The cross-scale channel information enhancement CSCE module is as follows Figure 5 As shown, specifically:

[0048] The original feature X and residual feature Y of each residual block of ResNet are fused to obtain the initial feature U as the input feature, and U' is the feature after the enhancement module, that is, the refined feature map U' with local and global features.

[0049] The channel information of the local feature is L(U), which is expressed as follows:

[0050] L(U)=B(Conv2(δ(B(Conv1(U)))))

[0051] Among them, Conv1 represents 1*1 point convolution to reduce the number of channels of input feature U to input B represents the BN batch normalization layer, δ represents the ReLU activation function, and the number of channels is restored to the same as the original input channels through Conv21*1 point convolution, and r is the channel scaling ratio.

[0052] The channel information calculation formula of the global feature G(U) is different from L(U) in that a global average pooling operation is first performed on the input U. The expression is as follows:

[0053] G(U)=B(Conv2(δ(B(Conv1(Gpool(U))))))

[0054] Among them, Gpool represents global average pooling;

[0055] The obtained local feature channel information L(U) and global feature channel information G(U) are subjected to a nonlinear activation function to obtain feature channel weights, which are used to perform an attention operation on the input feature U and obtain the output U′. The formula is as follows:

[0056]

[0057] Among them, σ represents the Sigmoid nonlinear activation function, that is, the weight normalization process, Indicates the addition of the corresponding elements of the two feature maps, Represents the multiplication of the corresponding elements of two feature maps. The flow chart of the cross-scale channel information enhancement module is as follows Figure 5 shown.

[0058] Cross-scale channel collaborative fusion The feature Z is then output through global average pooling and a fully connected layer to obtain a feature map containing local and global features.

[0059] Step 4: The captured infrared lane image is fed into the improved CSCE-ResNet18 network. The resulting refined features serve as input to two parallel fully connected layers. One fully connected layer is used for classification, outputting probabilities that are used to predict the probability of the lane line and background corresponding to the a priori lane line. The other fully connected layer is used for regression, outputting the lane line length and the lateral offset between the a priori lane line and the predicted lane line point. The outputs of these two fully connected layers are then used to determine the lane line position.

[0060] Based on the above method, the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the lane line detection model of the lane line detection method in the real-time infrared scene described in the above embodiment, which is used for lane line detection in real-time infrared images.

[0061] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer processor, implements the lane detection model of the lane detection method in a real-time infrared scene described in the above embodiment. The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0062] Most existing lane detection methods are designed based on visible light images and work best during the day and in scenes with good lighting conditions. However, there are significant differences in the physical properties of visible light images and infrared images, which makes existing image recognition technology unable to be directly applied to infrared scenes. First, visible light images rely on external light sources and are affected by weather, time of day, and ambient light. This leads to a significant decrease in lane recognition accuracy at night or in inclement weather such as rain, fog, and snow. In contrast, infrared images detect the thermal radiation of the object itself and have the advantage of being insensitive to lighting conditions. They can provide clear scene images in almost all lighting conditions, including completely dark environments.

[0063] However, the imaging characteristics of infrared images also pose challenges to traditional visible light-based detection algorithms. Lane markings in infrared images typically have low contrast and complex background noise, which is different from the characteristics of lane markings in visible light images. As a result, algorithms designed for visible light do not perform well when processing infrared images. For example, many lane marking detection algorithms in visible light scenes rely on obvious color, brightness, and edge features for recognition, which are often not obvious or completely absent in infrared images. In addition, the physical characteristics of infrared images may also lead to more image noise and interference, such as thermal radiation from background objects and the single characteristics of infrared images, which makes feature extraction more complicated. Therefore, existing visible light image recognition technology cannot be directly transferred to infrared scenes for use.

[0064] To address these technical difficulties, this paper proposes an optimized lane detection method for infrared images. By combining the Cross-Scale Channel Information Enhancement (CSCE) module with a cross-scale channel collaborative fusion mechanism, this method more effectively extracts both local and global features of lane lines in infrared images, overcoming the low contrast and monotonous image features found in infrared images. This method not only adapts to complex infrared scenes but also accurately predicts and locates the specific locations of lane lines through lane classification and regression. This significantly improves lane line accuracy while maintaining real-time performance of 105 FPS.

[0065] Figure 6 and Figure 7 This is an example of the present invention being applied to lane detection in an infrared camera scenario. It overcomes the challenges of lane detection at night and in low-visibility environments. It can further enhance the intelligent transportation system's road perception and indirectly ensure driver safety.

Claims

1. A lane line detection method in a real-time infrared scene, characterized by A ResNet-based lane detection model is constructed to detect lane positions in infrared scenes. This model uses a modified ResNet backbone network to extract feature maps containing local and global features from infrared images. Using the feature map's pre-set lane prior knowledge, lane detection is considered a holistic regression of the prior lanes. The feature map is fed into two parallel fully connected layers, one for classification and one for regression, to generate a final prediction and obtain lane position information in the infrared scene. The improved ResNet backbone network introduces a cross-scale channel collaborative fusion module in the residual block, fuses the original feature X and the residual feature Y of each residual block of ResNet to obtain the initial feature U, and then enhances the CSCE module based on cross-scale channel information to obtain a refined feature map U′ with local and global features. The original feature X and the residual feature Y are then combined with the refined feature map U′ through the Sigmoid activation function to achieve cross-scale channel collaborative fusion. Multi-scale context information is spatially collaboratively fused as the final residual output to obtain a feature map containing local and global features. The cross-scale channel information enhancement module CSCE is divided into two branches. One branch obtains the channel information of the global feature through global average pooling, and the other branch obtains the channel information of the local feature. The two sets of channel information features are spliced ​​and processed by the Sigmund function nonlinear activation to output the refined feature map. Specifically, the input feature is U, U' is the feature after processing by the CSCE module, and the channel information of the local feature is L(U). The expression is as follows: L(U)=B(Conv2(δ(B(Conv1(U))))) Among them, Conv1 represents a 1*1 point convolution that reduces the number of channels of the input feature map U to the input B represents the BN batch normalization layer, δ represents the ReLU activation function, and the number of channels is restored to the same as the original input channels through Conv21*1 point convolution. r is the channel scaling ratio; The channel information of the global feature is G(U), which is expressed as follows: G(U)=B(Conv2(δ(B(Conv1(Gpool(U)))))) Among them, Gpool represents global average pooling; The obtained local feature channel information L(U) and global feature channel information G(U) are subjected to a nonlinear activation function to obtain feature channel weights, and the output U′ is obtained after performing an attention operation on the input feature U: Among them, σ represents the Sigmoid nonlinear activation function, that is, the weight normalization process, Indicates the addition of the corresponding elements of the two feature maps, Represents the multiplication of corresponding elements of two feature maps.

2. The lane line detection method in a real-time infrared scene according to claim 1 is characterized by The lane detection model obtains the position information of lane lines in real-time infrared scenes by the following steps: Step 1: Collect real-time monitoring video from the infrared camera to obtain real-time infrared road images. Use OpenCV to read the video images as input for the lane detection model. Step 2: Preprocess the input image by performing a random affine transformation on the image while performing a random horizontal flip. The random affine transformation includes translation, rotation, and scaling. During preprocessing, the lane lines in the image are extended to the image boundaries. Step 3: The preprocessed infrared lane marking image is input into the improved ResNet backbone network. The input image passes through the convolution layer, batch normalization layer, and maximum pooling layer, and then enters the convolution module composed of four improved residual blocks to extract features. Finally, the features are output through global average pooling and a fully connected layer to obtain a feature map containing local and global features. The improved residual block introduces a cross-scale channel collaborative fusion module. The output feature of the ResNet residual block after two 3*3 convolutions is X, and the feature of the residual connection is Y. Y is a shallow feature. The original feature X and the residual feature Y are first fused. After the CSCE module, a refined feature map U' with local and global features is obtained. U' is activated by the Sigmoid function, and the output value is between 0 and 1. The original feature X and the residual feature Y are fused across scale channels to obtain the feature Z. The cross-scale channel collaborative fusion expression is as follows: Among them, M represents the cross-scale channel information enhancement module CSCE, The feature Z is then output through global average pooling and a fully connected layer to obtain a feature map containing local and global features; Step 4: The feature map containing local and global features is input into two parallel fully connected layers, one for classification and one for regression, which produce the final prediction to determine the lane line position.

3. The lane line detection method in a real-time infrared scene according to claim 2 is characterized in that For the collected surveillance video, in Step 1, after capturing the infrared camera's video of the urban road, images of lane lines in different road sections and scenes in the video are captured to train the lane line detection model. The length of the lane lines in the video screen should account for more than 30% of the image height, and the lane line pixel resolution should be no less than 90*90.

4. The lane line detection method in a real-time infrared scene according to claim 2 is characterized in that In Step 2, the preprocessing method is as follows: the video image is randomly translated by -25 to 25 pixels horizontally, and by -10 to 10 pixels vertically. The image is also randomly rotated by an angle ranging from -6 degrees to 6 degrees. The image is randomly scaled by a factor of 0.85 to 1.

15. For obscured lane lines, they are automatically filled to the image boundary through matrix completion.

5. The method for lane line detection in a real-time infrared scene according to claim 2 is characterized in that in Step 4, two fully connected layers are used, one of which is used for classification and outputs probability values ​​for predicting the probability of the lane line and background corresponding to the prior lane line, and the other is used for regression and outputs the lane line length and the lateral offset between the prior lane line and the predicted lane line point. The lane line position is determined through the outputs of the two fully connected layers.

6. An electronic device, characterized in that The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the lane line detection model of the lane line detection method in a real-time infrared scene according to any one of claims 1 to 5, which is used for lane line detection in real-time infrared images.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction or at least one program. When the at least one instruction or program is executed, the lane line detection model of the lane line detection method in a real-time infrared scene as described in any one of claims 1 to 5 is implemented.

Images