Lane line detection method and device in real-time infrared scene and storage medium

Abstract

A lane line detection method and device in a real-time infrared scene and a storage medium, an infrared camera is used to shoot a real-time infrared scene picture, a CSCE-Resnet18 backbone network optimized by a cross-scale channel information enhancement module (CSCE, Cros-Scale Channel Enhancement Module) is used to extract local features and global features of the picture, and multi-scale context information is cooperatively fused in space, so that the lane line detection method and device in the real-time infrared scene can be realized. And the capability of lane line detection is enhanced by applying fused refined information. Finally, lane line detection is regarded as overall regression of prior lane lines through lane line prior knowledge preset in the feature map, and specific position information of each lane line can be obtained. According to the method, the accuracy of infrared scene lane line detection is guaranteed to the maximum extent, meanwhile, the real-time performance of the algorithm is guaranteed, and practical application deployment can be carried out.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent video image processing, relates to 2D lane line detection on roads, and is a lane line detection method, device and storage medium in a real-time infrared scene. Background Art

[0002] With the rapid advancement of artificial intelligence and automotive industry technology, lane detection technology, as a key component of intelligent transportation systems, has been applied beyond the field of autonomous driving and covers multiple aspects such as intelligent traffic monitoring, assisted driving systems, and road maintenance. Lane detection can help vehicles and traffic management systems accurately identify lane markings on the road, thereby ensuring that vehicles drive safely in their lanes and providing key navigation information for various systems.

[0003] In recent years, many researchers have invested a lot of energy in the research of lane detection technology and proposed many advanced detection methods based on visible light. Typical detection models include: SCNN, PolyLaneNet, UFSD, LineCNN, etc. Although these methods perform well in visible light scenes, they are less applicable under infrared light or other non-standard lighting conditions. In fact, the driving and traffic management environment is often complex and changeable. Various factors such as changes in sunlight intensity, low light conditions at night, shadows of vehicles and trees, and worn lane lines will pose challenges to the accuracy and stability of lane detection.

[0004] In various traffic applications, the system must be able to accurately detect lane lines under different lighting conditions. However, traditional visible light detection systems face great challenges at night or in adverse weather conditions such as fog, rain and snow. In these cases, the performance of visible light scene models drops significantly, resulting in reduced reliability of lane line detection. In order to address the limitations of traditional lane line detection in complex environments, infrared cameras have gradually become one of the key technologies in traffic systems with their advantages of light independence and strong penetration. Infrared cameras can provide clear images in almost all lighting conditions, including in complete darkness, thus overcoming the limitations of nighttime and low-visibility environments. This allows both vehicles and traffic management systems to reliably detect lane lines when operating in low light or at night.

[0005] The research on lane line detection based on infrared images aims to make up for the shortcomings of visible light image detection, thereby improving the perception and reliability of various traffic systems in complex environments. For example, infrared imaging can clearly capture lane lines at night, even in low light conditions or with interference from other light sources. By integrating infrared images and advanced image processing algorithms, intelligent transportation systems can reliably identify lane lines under a wider range of conditions, improving overall driving safety. Summary of the invention

[0006] The problem to be solved by the present invention is: to use infrared cameras to solve the technical difficulties of lane line detection in poor lighting conditions at night, low visibility, bad weather, various shadows, etc., and to propose a lane line detection method in real-time infrared scenes based on a cross-scale channel information enhancement mechanism. While maximizing the accuracy and stability of lane line detection, the real-time nature of the method is ensured, and it can be used for practical application deployment.

[0007] The technical solution of the present invention is: a lane line detection method in a real-time infrared scene, constructing a lane line detection model based on ResNet to detect the position of the lane line in the infrared scene, the lane line detection model uses an improved ResNet backbone network to extract a feature map containing local features and global features from the infrared image, and the lane line detection is regarded as an overall regression of the prior lane line through the lane line prior knowledge pre-set in the feature map, and the feature map is input into two parallel fully connected layers, one for classification and the other for regression, to generate a final prediction, and obtain the position information of the lane line in the infrared scene;

[0008] The improved ResNet backbone network introduces a cross-scale channel collaborative fusion module in the residual block, fuses the original feature X and the residual feature Y of each residual block of ResNet to obtain the initial feature U, and obtains the refined feature map U′ with local and global information through the CSCE module based on the cross-scale channel information enhancement. Then, the original feature X and the residual feature Y are combined with the refined feature map U′ through the Sigmoid activation function to realize cross-scale channel collaborative fusion, and the multi-scale context information is collaboratively fused in space as the final residual output, which is used to obtain a feature map containing local features and global features;

[0009] The cross-scale channel information enhancement module CSCE is divided into two branches. One branch obtains the channel information of the global feature through global average pooling, and the other branch obtains the channel information of the local feature. The two sets of channel information features are concatenated and processed by the Sigmod function nonlinear activation to output a refined feature map. Specifically, the input feature is U, U′ is the feature after processing by the CSCE module, and the channel information of the local feature is L(U), and the expression is as follows:

[0010] L(U)=B(Conv2(δ(B(Conv1(U)))))

[0011] Among them, Conv1 represents a 1*1 point convolution that reduces the number of channels of the input feature map U to the input B represents the BN batch normalization layer, δ represents the ReLU activation function, and the number of channels is restored to the same as the original input channel number through Conv21*1 point convolution, and r is the channel scaling ratio;

[0012] The channel information of the global feature is G(U), which is expressed as follows:

[0013] G(U)=B(Conv2(δ(B(Conv1(Gpool(U))))))

[0014] Among them, Gpool represents global average pooling;

[0015] The local feature channel information L(U) and the global feature channel information G(U) are obtained through nonlinear activation function to obtain the feature channel weights, and the output U′ is obtained after attention operation on the input feature U:

[0016]

[0017] Among them, σ represents the Sigmoid nonlinear activation function, that is, the weight normalization process, It means the addition of the corresponding elements of two feature maps. Represents the multiplication of corresponding elements of two feature maps.

[0018] Furthermore, the lane line detection model obtains the position information of the lane line in the real-time infrared scene, including the following steps:

[0019] Step 1: Collect the real-time monitoring video of the infrared camera to obtain the real-time infrared road picture, and use OpenCV to read the video picture as the input of the lane line detection model;

[0020] Step 2: Preprocessing of the input image. An image enhancement method that performs random affine transformation while randomly flipping horizontally. Random affine transformation includes translation, rotation, and scaling. During preprocessing, the lane lines in the image are extended to the image boundary.

[0021] Step 3: The preprocessed infrared lane line image is input into the improved ResNet backbone network. The input image passes through the convolution layer, batch normalization layer, and maximum pooling layer, and then enters the convolution module composed of 4 improved residual blocks to extract features. Finally, the features are output through the global average pooling and fully connected layer to obtain a feature map containing local features and global features.

[0022] The improved residual block introduces a cross-scale channel collaborative fusion module. The output feature of the ResNet residual block after two 3*3 convolutions is X, and the feature of the residual connection is Y. Y is a shallow feature. The original feature X and the residual feature Y are firstly fused. After the CSCE module, a refined feature map U' with local and global features is obtained. U' is activated by the Sigmoid function, and the output value is between 0 and 1. The original feature X and the residual feature Y are fused across scale channels to obtain the feature Z. The cross-scale channel collaborative fusion expression is as follows:

[0023]

[0024] Among them, M represents the cross-scale channel information enhancement module CSCE,

[0025] The feature Z is then output through global average pooling and a fully connected layer to obtain a feature map containing local and global features;

[0026] Step 4: The feature map containing local features and global features is input into two parallel fully connected layers, one for classification and one for regression, which produce the final prediction to determine the lane line position.

[0027] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement a lane line detection model of the above-mentioned lane line detection method in real-time infrared scenes, which is used for lane line detection in real-time infrared images.

[0028] The present invention also provides a computer-readable storage medium, in which at least one instruction or at least one program is stored. When the at least one instruction or at least one program is executed, the lane line detection model of the lane line detection method in the real-time infrared scene is implemented.

[0029] The present invention realizes real-time automatic detection of road lane lines under infrared cameras. In actual scenes, conditions such as bad weather and insufficient light at night pose great challenges to the detection of lane lines. Although there are some lane line detection schemes based on infrared images in the prior art, these schemes can detect lane lines by setting prior knowledge of lane lines and combining global information to improve the accuracy and real-time performance of detection. However, the existing infrared lane line detection methods still face some difficulties when processing complex scenes, such as insufficient ability to extract refined features in low-contrast infrared images, especially when the lane lines are blocked or interfered by environmental noise, the detection effect is reduced. In addition, the existing methods are still insufficient in feature fusion and collaborative processing of multi-scale information, and cannot fully mine the context information in infrared images. In view of the difficulties of low contrast and single image features of infrared images, the present invention introduces cross-scale channel collaborative fusion and cross-scale channel information enhancement modules. The local and global features of the image are extracted, and the multi-scale context information is collaboratively fused in space, which improves the model's ability to detect lane lines in infrared scenes. The improved ResNet18 network maximizes the accuracy of lane line detection within an acceptable range of increased time cost, and the detection speed can reach 105FPS. The model meets the real-time requirements and can be deployed for practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of a lane line detection method in a real-time infrared scene based on a cross-scale channel information enhancement mechanism of the present invention.

[0031] Figure 2 It is a flow chart of the improved ResNet18 network of the present invention.

[0032] Figure 3 Schematic diagram of the residual block of the improved ResNet18 network in the present invention.

[0033] Figure 4 This is a flowchart of the cross-scale channel collaborative fusion in the present invention.

[0034] Figure 5 It is a flowchart of the cross-scale channel information enhancement module CSCE in the present invention.

[0035] Figure 6 This is Example 1 of lane line detection in an infrared camera scenario according to the present invention.

[0036] Figure 7 This is Embodiment 2 of the present invention applied to lane line detection in an infrared camera scenario. DETAILED DESCRIPTION

[0037] The present invention combines image enhancement, cross-scale channel collaborative fusion and deep learning methods to significantly improve the accuracy of lane line detection and recognition in infrared camera scenes while maintaining the real-time performance of the model. Considering that the most advanced methods for lane line detection are currently designed based on visible light scenes, they are not suitable for real-time detection of lane lines in infrared scenes. The present invention uses an improved lightweight ResNet18 model to extract the feature information of lane lines in the image, and uses a cross-scale channel information enhancement module (CSCE, Cross-Scale Channel Enhancement) to improve the model's different attention to different channels. Refined features are obtained through cross-scale channel collaborative fusion, which can improve the accuracy and stability of the model's lane line detection in infrared scenes. Classification and regression are used to finally obtain the location information of the lane line target, which makes up for the shortcomings of the visible light lane line detection algorithm, thereby improving the perception ability of the intelligent transportation system in complex environments.

[0038] The schematic diagram of the process of the present invention is as follows Figure 1 shown.

[0039] The specific implementation steps of the infrared camera viewing angle lane line detection are as follows:

[0040] Step 1: Read the infrared road image containing the lane line. Collect the real-time monitoring video of the infrared camera to obtain the real-time infrared road picture, and use OpenCV to read the video picture as the input of the lane line detection model. The length of the lane line in the video picture must exceed 30% of the picture height, and the pixel resolution of the lane line cannot be less than 90*90.

[0041] Step 2: Preprocessing of the read-in image. Since lane lines are long and thin structures and account for a small proportion of pixels in the image, the deep learning-based network can easily overfit the training set and perform poorly on the validation set. In order to prevent this phenomenon and improve the generalization ability of the model, the present invention adopts an image enhancement method that performs random affine transformation while performing random horizontal flipping. Affine transformation includes translation, rotation, and scaling. In addition, in order to maintain the complete lane line structure when the camera captures the road picture, the lane lines in the picture are extended to the image boundary during preprocessing.

[0042] Step 3: The preprocessed image is input into the improved ResNet backbone network. This embodiment is based on the ResNet18 backbone network and is called the CSCE-ResNet18 network. The improved ResNet18 network includes a convolution layer, a batch normalization layer, and a maximum pooling layer, and then enters a convolution module composed of 4 improved residual blocks to extract features. Finally, the features are output through global average pooling and a fully connected layer to obtain a feature map. In this embodiment, through a convolution of size 7*7, a step size of 2 and a maximum pooling layer of size 3*3, a step size of 2, the input image will become a feature map of size 56×56, which greatly reduces the amount of calculation. The 4 improved residual blocks connected in series are used to extract the lane line features of the image, and finally pass through the output layer, including the global average pooling layer, the fully connected layer and the output layer Softmax activation function, to output the recognition result.

[0043] The present invention improves the residual block of ResNet18. The improved residual block consists of two 3*3 convolutional layers and a skip connection. The lane feature information is extracted by stacking the intermediate residual blocks. A cross-scale channel collaborative fusion module is introduced. The improved residual block is as follows: Figure 3 shown.

[0044] The cross-scale channel collaborative fusion module performs cross-scale channel collaborative fusion on the original features and residual features of each residual block of ResNet. The output feature of the residual block after two 3*3 convolutions is X, and the feature of the residual connection is Y. Y is a shallow feature with a larger receptive field. The expression of cross-scale channel collaborative fusion is as follows:

[0045]

[0046] Among them, M represents the cross-scale channel information enhancement CSCE module, which first performs initial feature fusion on the two input features X and Y, and then passes the obtained initial feature U through the CSCE module to obtain a refined feature map U' with local and global properties. U' passes through the Sigmoid activation function, and the output value is between 0 and 1. Then, X and Y are weighted averaged, and the network is trained to determine their respective weights. The present invention adopts a cross-scale channel collaborative fusion mechanism for the X and Y features of the residual block, rather than the commonly used simple addition or concat fusion method. It pays more attention to those channels with obvious lane line features, and fully mines the contextual information in the infrared image to improve the model detection performance. The cross-scale channel collaborative fusion flow chart is shown as follows: Figure 4 shown.

[0047] The cross-scale channel information enhancement CSCE module is as follows Figure 5 As shown, specifically:

[0048] The original feature X and residual feature Y of each residual block of ResNet are fused to obtain the initial feature U as the input feature, and U' is the feature after the enhancement module, that is, the refined feature map U' with local and global features.

[0049] The channel information of the local feature is L(U), which is expressed as follows:

[0050] L(U)=B(Conv2(δ(B(Conv1(U)))))

[0051] Among them, Conv1 represents a 1*1 point convolution that reduces the number of channels of the input feature U to the input B represents the BN batch normalization layer, δ represents the ReLU activation function, and the number of channels is restored to the same as the original input channel number through the Conv21*1 point convolution, and r is the channel scaling ratio.

[0052] The channel information calculation formula of the global feature G(U) is different from L(U) in that a global average pooling operation is first performed on the input U. The expression is as follows:

[0053] G(U)=B(Conv2(δ(B(Conv1(Gpool(U))))))

[0054] Among them, Gpool represents global average pooling;

[0055] The local feature channel information L(U) and the global feature channel information G(U) are obtained through nonlinear activation function to obtain the feature channel weights, which are used to perform attention operation on the input feature U to obtain the output U′ formula as follows:

[0056]

[0057] Among them, σ represents the Sigmoid nonlinear activation function, that is, the weight normalization process, It means the addition of the corresponding elements of two feature maps. Represents the multiplication of the corresponding elements of two feature maps. The flow chart of the cross-scale channel information enhancement module is as follows Figure 5 shown.

[0058] Cross-scale channel collaborative fusion The feature Z is then output through global average pooling and a fully connected layer to obtain a feature map containing local and global features.

[0059] Step 4: Input the collected infrared lane line images into the improved CSCE-ResNet18 network to obtain refined features as the input of two parallel fully connected layers, one of which is used for classification and outputs probability values, which are used to predict the probability of lane lines and backgrounds corresponding to the prior lane lines. The other fully connected layer is used for regression and outputs lane line length and lateral offsets between the prior lane lines and the predicted lane line points. The outputs of the two fully connected layers are used to determine the lane line position.

[0060] Based on the above method, the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the lane line detection model of the lane line detection method in the real-time infrared scene described in the above embodiment, which is used for lane line detection in real-time infrared images.

[0061] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, implements the lane line detection model of the lane line detection method in a real-time infrared scene described in the above embodiment. The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0062] Most of the existing lane line detection methods are designed based on visible light images, and they work best mainly in daytime and scenes with good lighting conditions. However, there are significant differences in the physical properties of visible light images and infrared images, which makes the existing image recognition technology cannot be directly used in infrared scenes. First, visible light images rely on the illumination of external light sources and are affected by weather, time, and ambient light, resulting in a significant decrease in the accuracy of lane line recognition at night or in bad weather conditions such as rain, fog, and snow. In contrast, infrared images detect the thermal radiation of the object itself, have the advantage of being insensitive to lighting conditions, and can provide clear scene images in almost all lighting conditions, including completely dark environments.

[0063] However, the imaging characteristics of infrared images also pose challenges to traditional detection algorithms based on visible light. Lane lines in infrared images usually have low contrast and complex background noise, which is different from the characteristics of lane lines in visible light images, resulting in poor performance of algorithms designed based on visible light when processing infrared images. For example, many lane line detection algorithms in visible light scenes rely on obvious color, brightness, and edge features for recognition, which are usually not obvious or completely absent in infrared images. In addition, the physical characteristics of infrared images may also cause more image noise and interference, such as thermal radiation from background objects and single features of infrared images, which makes feature extraction more complicated. Therefore, existing visible light image recognition technology cannot be directly migrated to infrared scenes for use.

[0064] In view of these technical difficulties, the present invention proposes an optimized infrared image lane line detection method, which can more effectively extract the local and global features of lane lines in infrared images by combining the cross-scale channel information enhancement module CSCE and the cross-scale channel collaborative fusion mechanism, and overcome the problems of low contrast and single image features in infrared images. The present invention can not only adapt to complex infrared scenes, but also accurately predict and locate the specific position of lane lines through lane line classification and regression, while greatly improving the lane line accuracy, it can still ensure real-time performance of 105FPS.

[0065] Figure 6 and Figure 7 This is an implementation example of the present invention applied to lane line detection in an infrared camera scenario, which can overcome the challenges of lane line detection at night and in low-visibility environments, further improve the road perception of the intelligent transportation system, and indirectly ensure the safety of the driver.

Claims

1. A lane line detection method in a real-time infrared scene, characterized by A lane line detection model based on ResNet is constructed to detect the position of lane lines in infrared scenes. The lane line detection model uses an improved ResNet backbone network to extract feature maps containing local features and global features from infrared images. The lane line detection is regarded as an overall regression of the prior lane line based on the lane line prior knowledge pre-set in the feature map. The feature map is input into two parallel fully connected layers, one for classification and the other for regression, to generate the final prediction and obtain the position information of the lane line in the infrared scene. The improved ResNet backbone network introduces a cross-scale channel collaborative fusion module in the residual block, fuses the original feature X and the residual feature Y of each residual block of ResNet to obtain the initial feature U, and obtains the refined feature map U′ with local and global information through the CSCE module based on the cross-scale channel information enhancement. Then, the original feature X and the residual feature Y are combined with the refined feature map U′ through the Sigmoid activation function to realize cross-scale channel collaborative fusion, and the multi-scale context information is collaboratively fused in space as the final residual output, which is used to obtain a feature map containing local features and global features; The cross-scale channel information enhancement module CSCE is divided into two branches. One branch obtains the channel information of the global feature through global average pooling, and the other branch obtains the channel information of the local feature. The two sets of channel information features are concatenated and processed by the Sigmod function nonlinear activation to output a refined feature map. Specifically, the input feature is U, U′ is the feature after processing by the CSCE module, and the channel information of the local feature is L(U), and the expression is as follows: L(U)=B(Conv2(δ(B(Conv1(U))))) Among them, Conv1 represents a 1*1 point convolution that reduces the number of channels of the input feature map U to the input B represents the BN batch normalization layer, δ represents the ReLU activation function, and the number of channels is restored to the same as the original input channel number through Conv21*1 point convolution, and r is the channel scaling ratio; The channel information of the global feature is G(U), which is expressed as follows: G(U)=B(Conv2(δ(B(Conv1(Gpool(U)))))) Among them, Gpool represents global average pooling; The local feature channel information L(U) and the global feature channel information G(U) are obtained through nonlinear activation function to obtain the feature channel weights, and the output U′ is obtained after attention operation on the input feature U: Among them, σ represents the Sigmoid nonlinear activation function, that is, the weight normalization process, It means the addition of the corresponding elements of two feature maps. Represents the multiplication of corresponding elements of two feature maps.

2. The lane line detection method in a real-time infrared scene according to claim 1 is characterized in that The lane line detection model obtains the position information of the lane line in the real-time infrared scene, including the following steps: Step 1: Collect the real-time monitoring video of the infrared camera to obtain the real-time infrared road picture, and use OpenCV to read the video picture as the input of the lane line detection model; Step 2: Preprocessing of the input image. An image enhancement method that performs random affine transformation while randomly flipping horizontally. Random affine transformation includes translation, rotation, and scaling. During preprocessing, the lane lines in the image are extended to the image boundary. Step 3: The preprocessed infrared lane line image is input into the improved ResNet backbone network. The input image passes through the convolution layer, batch normalization layer, and maximum pooling layer, and then enters the convolution module composed of 4 improved residual blocks to extract features. Finally, the features are output through the global average pooling and fully connected layer to obtain a feature map containing local features and global features. The improved residual block introduces a cross-scale channel collaborative fusion module. The output feature of the ResNet residual block after two 3*3 convolutions is X, and the feature of the residual connection is Y. Y is a shallow feature. The original feature X and the residual feature Y are firstly fused. After the CSCE module, a refined feature map U' with local and global features is obtained. U' is activated by the Sigmoid function, and the output value is between 0 and 1. The original feature X and the residual feature Y are fused across scale channels to obtain the feature Z. The cross-scale channel collaborative fusion expression is as follows: Among them, M represents the cross-scale channel information enhancement module CSCE, The feature Z is then output through global average pooling and a fully connected layer to obtain a feature map containing local and global features; Step 4: The feature map containing local features and global features is input into two parallel fully connected layers, one for classification and one for regression, which produce the final prediction to determine the lane line position.

3. The lane line detection method in a real-time infrared scene according to claim 2 is characterized in that For the collected surveillance video, in Step 1, after collecting the video of the city road by the infrared camera, the lane line conditions in different sections and scenes in the video are captured and used to train the lane line detection model, where the length of the lane line in the video screen accounts for more than 30% of the picture height, and the lane line pixel resolution is not less than 90*90.

4. The lane line detection method in a real-time infrared scene according to claim 2, characterized in that In Step 2, the preprocessing method is as follows: the video image is randomly translated by -25 to 25 pixels in the horizontal direction, and the image is randomly translated by -10 to 10 pixels in the vertical direction, and the image is randomly rotated by an angle ranging from -6 degrees to 6 degrees; at the same time, the image size is randomly scaled, and the ratio range is between 0.85 times and 1.15 times; for the obscured lane lines, they are automatically filled to the image boundary through matrix completion.

5. According to the lane line detection method in a real-time infrared scene described in claim 2, it is characterized in that in Step 4, there are two fully connected layers, one of which is used for classification, and the output information is a probability value, which is used to predict the probability of the lane line and the background corresponding to the prior lane line, and the other fully connected layer is used for regression, and the output information is the lane line length and the lateral offset of the prior lane line and the predicted lane line point. The lane line position is determined through the output of the two fully connected layers.

6. An electronic device, characterized in that The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the lane line detection model of the lane line detection method in a real-time infrared scene according to any one of claims 1 to 5, which is used for lane line detection in real-time infrared images.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction or at least one program. When the at least one instruction or the program is executed, the lane line detection model of the lane line detection method in a real-time infrared scene as described in any one of claims 1-5 is implemented.

Images