A lane line detection method for autonomous driving in rainy weather in mountainous cities

Abstract

This invention provides a method for lane detection in rainy weather for autonomous driving in mountainous cities. This method improves the PINet network into a lane detection network based on a residual structure. Specifically, three different residual units are designed to address the different roles of three bottlenecks in the lane detection module. The attention mechanism, SENet, is combined with the residual module to enhance the weights of channels containing important lane information and reduce the weights of channels containing useless information. This improvement effectively improves the network's lane detection accuracy in rainy weather. Furthermore, to further enhance detection accuracy in rainy weather, this patent prioritizes raindrop removal prior to lane detection. The image is preprocessed for rain removal prior to lane detection.

Description

Technical Field

[0001] The present invention relates to the technical field of lane line detection, and in particular to a method for detecting lanes in rainy autonomous driving in mountainous cities. Background Art

[0002] In recent years, lane detection research has primarily focused on conventional scenarios. Some existing techniques use various handcrafted features to detect lanes. For example, one technique uses the HSI color model to analyze image color and fuzzy C-means clustering of pixel intensities to detect lane lines. Another technique proposes a vanishing point-based lane detection method, in which vanishing points are estimated using a probabilistic voting function defined by segment intensities. Lane segments are determined by geometric constraints, and a scoring function is used to eliminate incorrect segments. However, detection methods based on handcrafted features are not adaptable to complex scenarios.

[0003] Lane detection methods based on deep learning can autonomously learn feature representations at different levels and have excellent robustness in various scenarios. For example, Reference 1 ("Key Points Estimation and Point Instance Segmentation Approach for Lane Detection," IEEE Transactions on Intelligent Transportation Systems, pp. 1-10, 2021) proposes a lane detection network PINet based on key point estimation and point instance segmentation, which transforms the clustering problem of predicting lanes into an instance segmentation problem. In addition, the lane line detection process can be conceptualized as a line-based selection problem, using global features and structural losses to explicitly model lanes.

[0004] However, lane detection in rainy weather is very challenging, which is an extreme condition for autonomous driving. Reference 2 ("Performance of drivable path detection system of autonomous robots in rain and snow scenario," 2014 International Conference on Signal Processing and Integrated Networks, pp. 679-684, 2014) discloses a filtering algorithm to detect and remove rain and snow in images, which uses a support vector machine to divide rain-free images into drivable and non-drivable paths. Reference 3 ("VPGNet: Vanishing Point Guided Network for Lane and Road Marking Detection and Recognition," in 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 1965-1973) established a dataset containing sunny days, light rain, heavy rain, and night, and also proposed a vanishing point guided network VPGNet for lane and road marking detection and recognition. However, the technologies disclosed in References 2 and 3 are not designed for rainy days in mountainous cities, where mountains and dark clouds can lead to low visibility in the environment. Due to low visibility and rain reflections on the road, lanes are often black, and raindrops attached to the windshield blur part of the scene, making lane detection challenging.

[0005] The above-mentioned document 1 proposes a lane line detection method based on key point estimation and point instance segmentation for conventional scenarios. Its network consists of four hourglass networks connected in series, and all hourglass modules are trained simultaneously using the same loss function. The hourglass module can pass information of various scales to deeper layers, thereby helping the entire network obtain global and local features. Due to this feature, the hourglass network is often used to detect the center or corner of an object in the object detection area. This neural network includes three output branches: a confidence branch, an offset branch, and an embedding branch. The confidence and offset branches predict the precise points of the lane line, and the embedding branch generates an embedded feature for each predicted point. The embedded feature is fed into the clustering process to distinguish each instance. The disadvantage of this method is that when facing rainy scenes in mountainous cities, the influence of raindrops and rain fog may lead to missed detection and inaccurate detection of lane lines.

[0006] Reference 3 above proposes a vanishing point-guided network for lane and road sign detection in rainy weather. Its VPGNet consists of four task modules: grid box regression, object detection, multi-label classification, and vanishing point prediction. Each task performs complementary tasks. A vanishing point prediction module is also designed to improve scene representation. This module includes a global context to predict invisible lanes under occlusion or extreme lighting conditions. To infer global information, a quadrant mask is used to divide the entire image into four parts. The intersection of these four parts is the vanishing point. The output of the vanishing point prediction module defines five channels: each output pixel belongs to one of these five channels. An absent channel indicates the absence of a vanishing point, and the four quadrant channels represent the four parts of the image. However, this technique suffers from the inability to fully detect lane boundaries in the presence of shadows. Furthermore, the lane lines output by the network are imprecise and deviate significantly from the actual lane lines. Furthermore, the network has a large number of parameters and a slow runtime. Furthermore, the network's performance is not fully evaluated, being evaluated only on a self-built dataset. This may lead to incorrect detection of lane vanishing points when used on public datasets.

[0007] In addition, there are already many excellent deraining algorithms in the existing technology to remove rain from videos and images, but most image / video deraining methods focus on removing rain streaks, while only a few methods aim to remove raindrops. Moreover, the practical applications of these methods, such as semantic segmentation and object detection, need further testing and verification. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, this invention provides a method for lane detection in rainy mountainous urban autonomous driving. This method addresses the low accuracy and robustness of existing conventional lane detection algorithms in rainy mountainous urban scenarios. Targeting rainy mountainous urban scenarios, this patent combines rain removal with lane detection based on an improved PINet network. This method preprocesses the image using rain removal to improve lane detection accuracy in rainy conditions.

[0009] The technical solution adopted by the present invention to solve the technical problem is: a method for detecting lane lines of automatic driving in rainy days in mountainous cities, comprising the following steps:

[0010] Step 1: Use the raindrop removal module to perform rain removal preprocessing on the input image to obtain a rain-removed image;

[0011] Step 2: Use the lane detection module to perform lane detection on the derained image. The lane detection module includes a resizing layer and four hourglass modules. After the derained image is reduced in size by the resizing layer, it is input into the four hourglass modules for lane key point prediction and line fitting post-processing. The four hourglass modules have the same structure and loss function. A single hourglass module consists of a downstream bottleneck, a common bottleneck, an upstream bottleneck, and an output branch. The downstream bottleneck, common bottleneck, and upstream bottleneck are all composed of residual units. At the same time, the channel attention mechanism SENet is inserted into the residual units in the downstream bottleneck and the common bottleneck. The output branch contains confidence, offset, and embedded features.

[0012] Furthermore, the residual unit structure in the downstream bottleneck is: a 1×1 convolution layer for reducing the number of channels; a convolution layer with a convolution kernel size of 3, a stride of 2, and a padding of 1 for reducing the feature map size and extracting image features; another 1×1 convolution layer for restoring the initial number of channels; then the channel attention mechanism SENet is used to increase the weight of important channels; a convolution layer with a convolution kernel size of 3, a stride of 2, and a padding of 1 is used in the shortcut branch to reduce the size, retain the feature map details and ensure the same size of the feature maps.

[0013] Furthermore, the residual unit structure in the same bottleneck is: a 1×1 convolution layer for reducing the number of channels; a convolution layer with a convolution kernel size of 3, a stride of 1, and a padding of 1 for extracting key image features; another 1×1 convolution layer for restoring the initial number of channels; then the channel attention mechanism SENet is used to increase the weight of important channels; a convolution layer with a convolution kernel size of 3, a stride of 1, and a padding of 1 is used in the shortcut branch to retain feature map details.

[0014] Furthermore, the residual unit structure in the upstream bottleneck is: a 1×1 convolution layer for reducing the number of channels; a transposed convolution layer with a convolution kernel size of 3, a stride of 2, and a padding of 1 for restoring the feature map size; and another 1×1 convolution layer for restoring the initial number of channels; and a bilinear interpolation upsampling method is used in the shortcut branch to restore the image size.

[0015] Furthermore, the channel attention mechanism SENet consists of a compression layer, an excitation layer and a scaling layer; the channel attention mechanism SENet performs a convolution operation on the input features and then performs global average pooling through the compression layer to compress the feature layer into a real number sequence, and then performs a nonlinear transformation in the excitation layer to obtain a real number sequence of each channel weight, and finally in the scaling layer, the weight of each channel is weighted to the original channel one by one by multiplication.

[0016] Furthermore, the network loss function of the lane line detection model includes a confidence loss function, an offset loss function, an embedding feature loss function and a distillation loss function, wherein the distillation loss function is applied in the same bottleneck, and the confidence loss function, the offset loss function and the embedding feature loss function are applied to each output branch of the hourglass module.

[0017] Furthermore, the confidence loss function includes an existence loss function and an absence loss function; the existence loss is used for cells with key points, and the absence loss is used to reduce the confidence value of cells without key points; if a grid cell contains a key point, its confidence value is close to 1, and if it does not contain a key point, it is 0; when the predicted confidence value of a cell is higher than 0.01, the absence loss is calculated; the definition of the confidence loss function is as follows:

[0018]

[0019]

[0020]

[0021] Among them, L exist Indicates that there is a loss function, L non_exist Indicates that there is no loss function and introduces the huber function To enhance the robustness to error points; N e Indicates the number of cells containing key points, N n Indicates the number of cells without key points, G e Represents a unit set consisting of key points, G n represents the unit set consisting of all points, c c represents the confidence prediction value of each unit, represents the true value, and α is set to 2.5;

[0022] The offset loss function is defined as follows:

[0023]

[0024] Among them, N e Indicates the number of cells containing key points, G e represents a unit set consisting of key points, c x Indicates the true x-axis offset value of the key point, c y Indicates the true y-axis offset value of the key point, Indicates the x-axis predicted offset value of the key point, Indicates the y-axis predicted offset value of the key point;

[0025] The embedding feature loss function is defined as follows:

[0026]

[0027]

[0028] Among them, N e Indicates the number of cells containing key points, F i represents the predicted embedding feature of cell i, F j represents the predicted embedding feature of cell j, K is a constant, K>0, l(i,j) represents the embedding feature difference between cells i and j; if I ij = 1, then cell i and cell j are the same instance, if I ij =0, then cell i and cell j are not the same instance;

[0029] The distillation loss function is defined as follows:

[0030]

[0031] F(J d )=S(G(J d )),S: spatial softmax,

[0032] G:R C×H×W →R H×W ,

[0033] Among them, R C×H×W Represents a feature map with C channels and H×W dimensions, R H×W represents a 2D feature map with a dimension of H×W; h represents the maximum number of hourglass modules, J d represents the distillation output of the same bottleneck of the d-th hourglass module, J di Indicates J d The i-th channel, C represents the total number of channels, G(J d ) represents the square sum of the distilled output feature map of the d-th hourglass module on C channels into a two-dimensional spatial feature map, F(J d ) represents the G(J d ) performs spatial softmax activation, F(J h ) represents the distillation output feature map of the h-th hourglass module, which is squared and converted into a two-dimensional spatial feature map on C channels and then subjected to spatial softmax activation. V(F(J h )-F(J d )) indicates the difference between F(J h ) and F(J d ) is squared and summed to obtain the total distillation loss;

[0034] The weighted combination of the confidence loss function, offset loss function, embedding feature loss function, and distillation loss function constitutes the overall loss function of the lane detection model, which is expressed as follows:

[0035] L total =γ e L exist +γ n L non-exist +γ o L offset +γ f L feature +γ d L distillation

[0036] Among them, γ o , γ f , γ d , γ e , γ n is the weight value of each loss function.

[0037] Furthermore, the raindrop removal model consists of a generative network and a discriminative network, wherein the generative network includes an attention recurrent network and a contextual autoencoder;

[0038] The attention recurrent network identifies raindrops and their surrounding areas in the input image, and then feeds the generated attention map to the contextual autoencoder to generate a clean image with raindrops removed; the discriminant network judges the authenticity of the generated clean image.

[0039] Furthermore, the attention recurrent network consists of four network blocks, each of which consists of a 5-layer ResNet, a convolutional LSTM unit and a convolutional layer; the contextual autoencoder consists of 16 Conv-ReLU blocks.

[0040] Furthermore, the specific process of the discriminant network judging the authenticity of the generated clean image is as follows: the clean image generated by the generative network is used as the input of the discriminant network, and after 5 Conv+Relu layers, the original image features are obtained, and then a Convs operation is performed to obtain an attention mask. The result obtained in the Convs process is multiplied by the original features to guide the discriminator to focus on the dense area of the attention map, and finally a fully connected layer is used to determine the authenticity of the image.

[0041] Positive effects of the invention: This patent proposes a lane line detection method for rainy day scenes in mountainous cities, and improves the PINet network into a lane line detection network based on a residual structure. That is, according to the different effects of the three bottlenecks in the lane line detection module, three residual units with different structures are designed, and the attention mechanism SENet is combined with the residual module to enhance the weight of channels containing important lane line information and reduce the weight of channels containing useless information. This improvement can effectively improve the lane line detection accuracy of the network in rainy environments. At the same time, in order to further improve the detection accuracy in rainy environments, this patent puts the raindrop removal work before the lane line detection processing, first performs rain removal preprocessing on the image, and then performs lane line detection in the image. In addition, the number of parameters of the network model used in this patent is much smaller than that of VPGNet, and the running speed is faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 2. It is a flow chart of the method for detecting lane lines in rainy days for autonomous driving in mountainous cities according to the present invention;

[0043] Figure 2 Schematic diagram of the structure of the raindrop removal module of the present invention;

[0044] Figure 3 Schematic diagram of the structure of the lane detection module of the present invention;

[0045] Figure 4 Schematic diagram of the residual unit structure in the downstream bottleneck of the present invention;

[0046] Figure 5 is a schematic diagram of the residual unit structure in the same bottleneck described in the present invention;

[0047] Figure 6 Schematic diagram of the residual unit structure in the upstream bottleneck of the present invention;

[0048] Figure 7 It is a schematic diagram of the structure of the channel attention mechanism SENet described in the present invention;

[0049] Figure 8 This is a comparison chart of the 8 groups of image processing results described in the present invention. DETAILED DESCRIPTION

[0050] The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0051] Reference Figures 1 to 7 A preferred embodiment of the present invention provides a method for detecting lane lines for autonomous driving in rainy weather in mountainous cities, comprising the following steps:

[0052] Step 1: Use the raindrop removal module to perform rain removal preprocessing on the input image to obtain a rain-removed image;

[0053] Among them, the raindrop removal module is as follows Figure 2 As shown in Figure 1, the network consists of a generator and a discriminator. The generator aims to generate a clean image without raindrops, while the discriminator verifies whether the output of the generator is realistic, that is, whether the image is as close to a rain-free image as possible.

[0054] The generative network includes an attention recurrent network and a contextual autoencoder. The attention recurrent network locates raindrops and surrounding areas potentially affected by them in the image and generates an attention map. The contextual autoencoder uses the attention map to focus on raindrops and surrounding areas potentially affected by them, generating a better local image restoration map (i.e., background restoration after rain removal).

[0055] In the discriminative network, an attention-based discriminator (i.e., a discriminator built using the attention map of the generative network) is used to determine true or false in the area specified by the attention map. (When using a trained model to remove rain from an image, the network has reached its optimal state. Only the generative network is needed to generate rain-free images, and the discriminative network is no longer needed because the discriminative network will only determine whether the image is true.)

[0056] Step 2: Use the lane detection module to perform lane detection on the derained image. The lane detection module includes a resizing layer and four hourglass modules. After the derained image is resized by the resizing layer, it is input to the four hourglass modules for lane key point prediction and line fitting post-processing. The four hourglass modules have the same structure and loss function. Each hourglass module consists of a downstream bottleneck, a shared bottleneck, an upstream bottleneck, and an output branch. The downstream bottleneck, shared bottleneck, and upstream bottleneck are all composed of residual units. The residual units in the downstream and shared bottlenecks are embedded with a channel attention mechanism (SENet). The downstream bottleneck downsamples the input image (to extract image features), reducing the size of the feature map, saving computer computing memory and accelerating processing speed. The shared bottleneck further extracts key lane features from the image, and the upstream bottleneck uses transposed convolution to restore the feature map to the image size after processing the resizing layer.

[0057] The output branch contains confidence, offset and embedded features; the unit with key points (for example, the unit that divides the image into 64×32 grids) has a confidence closer to 1, and the confidence value will be input into the next hourglass module as a reference for the prediction value; the offset determines the specific position of the key point in the unit; the embedded feature divides the key point into a single instance.

[0058] The principle of the method of the present invention is described in more detail below.

[0059] 1. Raindrop Removal Module (Raindrop Removal Network)

[0060] The structure of the raindrop removal network is as follows Figure 2 The rain image is modeled as a combination of image background, raindrop effect, and raindrops, which is defined as:

[0061]

[0062] Where I, B, R and M represent the rain image, background image, the effect of raindrops on the image and binary mask respectively. M(x) = 1 means that pixel x is part of the raindrop area. Denotes element-wise multiplication. The goal of the network is to obtain a background image B from a given rain image I. In the training data, the background image B is subtracted from the rain image I to obtain a binary mask M, and a threshold of 30 is set to determine whether the pixel is part of the raindrop area (the pixel value of the rain image I minus the pixel value of the background image B is the binary mask M. If the pixel value of the region in M is greater than 30, the region is part of the raindrop area).

[0063] This raindrop removal network consists of a generator network and a discriminator network. The generator network is used to generate a clean image with as few raindrops as possible, while the discriminator network determines whether the output image of the generator network is real or fake. The overall loss function of the raindrop removal network is defined as:

[0064]

[0065] Among them, G and D represent the generation network and the discriminant network respectively; I is the sample extracted from the raindrop dataset as the input image, R is the sample extracted from the rain-free image dataset as the real image, D(R) represents the discrimination of the real image, G(I) represents the derained image, D(G(I)) represents the discrimination of the derained image, and E R～pclean [log(D(R))] represents the overall loss function of the discriminant network, E I～praindrop [log(1-D(G(I)))] represents the loss function of the generative network.

[0066] 1.1 Generation Network: The generation network consists of two sub-networks: an attention recurrent network and a contextual autoencoder. On the input image, the attention recurrent network identifies raindrops and their surrounding areas, and then feeds the generated attention map into the contextual autoencoder to generate a more accurate local image restoration map.

[0067] Attention Recurrent Network: Each network block consists of a 5-layer ResNet, a convolutional LSTM unit, and a convolutional layer. The ResNet is designed to extract features from the image and mask; the convolutional layer generates a 2D attention map. The attention distribution map learned in the LSTM is a matrix ranging from 0 to 1, with larger values indicating greater attention, indicating an increase in attention from areas without raindrops to areas with raindrops. Due to the different transparency of raindrop areas, attention also varies within the raindrop area.

[0068] The loss function of the attention loop block is defined by the mean squared error (MSE) between the binary mask M and the output attention map At at time step t. The value corresponding to the attention map increases with the increase of the time step. The loss function is defined as follows:

[0069]

[0070] Among them A t =ATT t (F t-1 ,H t-1 ,C t-1 ), ATT t represents the attention recurrent network at time step t, F t-1 represents the connection between the input image and the attention map generated in the previous step, H t-1 Represents the output characteristics of the LSTM unit in the previous attention cycle network, C t-1 represents the feature encoding of the LSTM in the previous attention recurrent network; when t=1, F t-1 is the input image with an initial attention value of 0.5; set N = 4 and θ = 0.8.

[0071] Contextual Autoencoder: This network is used to generate raindrop-free images. The final attention map generated by the attention recurrent network is combined with the initial input image as input. The entire network consists of 16 Conv-ReLU blocks. To prevent blurring of the output image, skip connections are inserted between convolutional layers. This contextual autoencoder uses two loss functions: a multi-scale loss and a perceptual loss.

[0072] In order to obtain contextual information at different scales, feature outputs are extracted from the last layer, the third to last layer, and the fifth to last layer of the decoder, with output sizes of 1, 1 / 2, and 1 / 4 of the original size, respectively. The multi-scale loss function is defined as:

[0073]

[0074] Among them S i represents the i-th output extracted from the decoder layer, T i Indicates that S iReal clean image features with the same scale. are weights of different scales, and λ is set to 0.6, 0.8, and 1.0 respectively.

[0075] Perceptual loss is used to quantify the global difference between the ground truth clean image features and the corresponding features of the autoencoder output, which can be extracted using the pre-trained VGG16 network. The perceptual loss function is defined as:

[0076] L P (O,T)=L MSE (VGG(O),VGG(T))

[0077] Where O is the output image of the entire generative network, and T is the real rain-free image.

[0078] The overall loss function of the generative network is defined as follows:

[0079] L G =10 -2 log(1-D(O))+L ATT ({A},M)+L M ({S},{T})+L P (O,T)

[0080] 1.2 Discriminant Network

[0081] Because the locations of raindrops in the original input image are unknown, the local discriminator needs to autonomously identify these areas. To address this issue, the attention map is combined with the local discriminator to form the attention discriminator. Specifically, the clean image generated by the generative network is used as the input to the discriminative network. After passing through five Conv+Relu layers to obtain the original image features, a Convs operation is then performed to obtain an attention mask. The result obtained from the Convs process is multiplied with the original features to guide the discriminator to focus on the dense areas of the attention map. Finally, a fully connected layer is used to determine the authenticity of the image.

[0082] The loss function of the discriminant network is expressed as:

[0083] L D (O,R,A N )=-log(D(R))-log(1-D(O))+γL map (O,R,A N )

[0084] Among them, L map is the final attention map A generated from the attention mask map generated by the discriminator and the attention recurrent network N The difference between , is defined as follows:

[0085] Lmap (O,R,A N )=L MSE (D map (O),A N )+L MSE (D map (R),0)

[0086] Among them D map represents the process of the discriminant network generating a two-dimensional map, γ is set to 0.05, R represents a sample image extracted from a real clean dataset, 0 represents an attention map with a value of 0, and L MSE (D map (R),0) means there is no specific area in the real image R that needs attention.

[0087] 2. Lane Prediction Module

[0088] Since the four hourglass modules have the same structure and loss function, only a single hourglass module and three convolutional layers for size modification are introduced. The network details are as follows: Figure 3 shown.

[0089] 2.1 Network Details: First, three convolutional layers resize the output of the rainy image enhancement module to 64×32 to save memory and network inference time. All three convolutional layers use a kernel size of 3, a stride of 2, and padding of 1. Prelu activation and batch normalization (BN) are used after each convolutional layer. The resized output is then fed into an hourglass module for lane keypoint prediction. Each hourglass module consists of a downward bottleneck, a uniform bottleneck, an upward bottleneck, and three output branches. The lower bottleneck is downsampled to reduce the feature map size, the uniform bottleneck produces outputs of the same size, and the upper bottleneck is upsampled to restore the feature map size. To pass feature information of various sizes to deeper layers, skip connections are inserted between each downward bottleneck and the symmetrical upward bottleneck. The three output branches of the hourglass module are confidence, offset, and embedded features. Each branch has three convolutional layers, forming a 64×32 grid. Each grid contains the output predictions for the keypoint's confidence value, offset, and embedded features. The network architecture of four serial hourglass modules is deep, computationally slow, and has strict hardware requirements. In order to improve the performance (lane line detection accuracy) of the detection network containing a smaller number of hourglass modules, knowledge distillation technology is applied to use the deepest hourglass module in the network as the teacher network.

[0090] The residual network is easy to modify and expand. By adjusting the number of channels and blocks in the residual module, the width and depth of the overall network can be modified, and the problems of gradient dissipation and network degradation can be avoided. Therefore, the present invention introduces the residual unit into the hourglass module as the basic unit of the network. SENet is a plug-and-play channel attention network that can improve the accuracy of tasks such as image classification, scene classification and object detection. The details of its network structure are as follows: Figure 7 As shown. SENet consists of compression (Squeeze), excitation (Excitation) and scale layer (Scale). The input feature Q (H'×W'×C') is convolved to obtain U (H×W×C), and then global average pooling is performed through compression (Squeeze), compressing the feature layer into a real number sequence Z (1×1×C), and then a nonlinear transformation is performed in excitation (Excitation) to obtain a real number sequence P of each channel weight. Finally, in the scale layer (Scale), the weight of each channel is weighted one by one to the original channel by multiplication. The main calculation formula of the SENet network is as follows:

[0091]

[0092] P=Excitation(Z,W)=σ[W2δ(W1,Z)]

[0093]

[0094] Where W1 and W2 represent the weights after the first fully connected layer and the second fully connected layer respectively; σ represents the Sigmoid activation function; δ represents the ReLU activation function.

[0095] This paper combines SENet and residual networks to improve lane detection accuracy. Each bottleneck is composed of residual units. Three residual units with different structures are designed to correspond to the different functions of the three bottlenecks. A channel attention mechanism, SENet, is inserted into the downstream bottleneck and the same bottleneck to enhance the weight of important channels and reduce the weight of useless information channels. Figures 4 to 6The details of three types of residual units are described. The residual structure in the downstream bottleneck: a 1×1 convolutional layer is used to reduce the number of channels to improve computational speed; a convolutional layer with a kernel size of 3, a stride of 2, and a padding of 1 is used to reduce the feature map size and extract image features; a 1×1 convolutional layer is used to restore the original number of channels; SENet is then used to increase the weights of important channels; a convolutional layer with a kernel size of 3, a stride of 2, and a padding of 1 is used in the shortcut branch to reduce the size while preserving as much feature map detail as possible and ensuring the same feature map size. The residual unit in the same bottleneck: a 1×1 convolutional layer is used to reduce the number of channels to improve computational speed; a convolutional layer with a kernel size of 3, a stride of 1, and a padding of 1 is used to extract key image features; a 1×1 convolutional layer is used to restore the original number of channels; SENet is then used to increase the weights of important channels; in the shortcut branch, only a convolutional layer with a kernel size of 3, a stride of 1, and a padding of 1 is used to preserve feature map details. Residual unit in the upstream bottleneck: a 1×1 convolution layer is used to reduce the number of channels and improve the operation speed; a transposed convolution layer with a convolution kernel size of 3, a stride of 2, and a padding of 1 is used to restore the size of the feature map; another 1×1 convolution layer is used to restore the initial number of channels; bilinear interpolation upsampling is used in the shortcut branch to restore the image size.

[0096] 2.2 Loss Function: The loss function of this lane detection network consists of four parts: confidence loss, offset loss, embedding feature loss, and distillation loss. The distillation loss is applied in the same bottleneck, while the other three loss functions are applied to each output branch of the hourglass module.

[0097] 1) Confidence loss: This loss is divided into presence loss and absence loss. Presence loss is used for cells with key points, while absence loss is used to reduce the confidence value of cells without key points. If a grid cell contains a key point, its confidence value is close to 1; if not, it is 0. When the predicted confidence value of a cell is higher than 0.01, the absence loss is calculated. The output of the confidence branch is sent to the subsequent hourglass module. At the same time, the Huber function expression is introduced to enhance the robustness to erroneous points (i.e., reduce sensitivity to outliers).

[0098] The confidence loss function is expressed as follows:

[0099]

[0100]

[0101]

[0102] where N e Indicates the number of cells containing key points, N n Indicates the number of cells without key points, Ge Represents a unit set consisting of key points, G n represents the unit set consisting of all points, c c represents the confidence prediction value of each unit, Represents the true value, and α is set to 2.5. If the cell contains a key point, the true value is 1; otherwise, it is 0. The specific confidence thresholds are 0.35 (4h), 0.32 (3h), 0.30 (2h), and 0.52 (1h). When the confidence value of a cell is greater than the predetermined threshold, there is a key point at the cell. Add regularization term To prevent overfitting.

[0103] 2) Offset loss: The predicted value of each cell output from the offset branch includes x-axis and y-axis offset values, which are between 0 and 1 and help to accurately locate the key points within each cell. One cell corresponds to eight pixels of the input image. By multiplying the offset value by the pixel value, the exact location of the key point can be determined. For example, if the predicted offset value is 0.5, the actual location of the key point is 4 pixels away from the edge of the cell. If there is no key point, the offset loss cannot be calculated. The offset loss function is expressed as follows:

[0104]

[0105] Among them, N e Indicates the number of cells containing key points, G e represents a unit set consisting of key points, c x Indicates the true x-axis offset value of the key point, c y Indicates the true y-axis offset value of the key point, Indicates the x-axis predicted offset value of the key point, Indicates the y-axis predicted offset value of the key point;

[0106] 3) Embedding feature loss: If the embedding features of each cell are the same, the embedding feature branch is trained to make the embedding features of each cell closer. The embedding feature loss function is expressed as follows:

[0107]

[0108]

[0109] F i represents the predicted embedding feature of cell i, K is a constant, K>0, and l(i,j) represents the difference in embedding features between cells i and j. ij = 1, then unit i and unit j are the same instance, if I ij= 0, then cells i and j are not the same instance. During training, the loss function forces features to be closer if cells are the same instance. If cells are different instances, features are assigned. A simple distance-based clustering technique is used to distinguish keypoints as individual instances. Instances are considered the same if the keypoint's embedded features are within a specified distance of 0.08.

[0110] 4) Distillation loss: The deepest hourglass module is designated as the teacher network to guide the training of the remaining hourglass modules. The distillation loss function is defined as follows:

[0111]

[0112] F(J h )=S(G(J d )),S:spatial softmax,

[0113]

[0114] where R C×H×W Represents a feature map with a channel dimension of H×W, R H×W represents a 2D feature map with a dimension of H×W; h represents the maximum number of hourglass modules, J d represents the distillation output of the same bottleneck of the d-th hourglass module, J di Indicates J d The i-th channel, C represents the total number of channels, G(J d ) represents the square sum of the distilled output feature map of the d-th hourglass module on C channels into a two-dimensional spatial feature map, F(J d ) represents the G(J d ) performs softmax activation on the space, F(J h ) represents the distillation output feature map of the h-th hourglass module, which is squared and converted into a two-dimensional spatial feature map on C channels and then subjected to spatial softmax activation. V(F(J h )-F(J d )) indicates the difference between F(J h ) and F(J d ) is squared and summed to obtain the total distillation loss;

[0115] The weighted sum of these four loss functions constitutes the overall loss function of the lane line prediction module, which is expressed as follows:

[0116] L total =γ e L exist +γ n L non-exist +γ o Loffset +γ f L feature +γ d L distillation

[0117] During training, γ o Set to 0.2, γ f Set to 0.5, γ d Set to 0.1, γ e and γ n Both are set to 1.0.

[0118] Using the lane line detection method for automatic driving in rainy weather in mountainous cities of the present invention, 8 sets of input images (including 4 sets of images without rain and fog and 4 sets of images with rain and fog) are processed to obtain 8 sets of image processing result comparisons. Figure 8 shown.

[0119] At the same time, the same batch of input image data was processed using PINet (the method described in Reference 1), UfastNet-Res18, and UfastNet-Res34 (referenced in "Ultra Fast Structure-Aware Deep Lane Detection", Cham, 2020, pp. 276-291), as well as the method described in the present invention with confidence thresholds of 0.35 (4 hours), 0.32 (3 hours), 0.30 (2 hours), and 0.52 (1 hour). The obtained image processing results were statistically analyzed, as shown in the following table:

[0120]

[0121] As can be seen, the proposed technology achieves a detection accuracy of 88.96% in mountainous urban rainy days without fog, with a false positive rate of 0.4026 and a false negative rate of 0.3333. This represents a 0.34% improvement in accuracy over PINet, with a 0.0274% reduction in false positive rates and a 0.0024% reduction in false negative rates. In foggy scenes, the detection accuracy reaches 86.59%, with a 0.3537% false positive rate and a 0.3455% false negative rate. This represents a 7.5% improvement in accuracy over PINet, with a 0.122% reduction in false positive rates and a 0.0915 reduction in false negative rates. This demonstrates a significant advantage for lane marking detection in mountainous urban rainy days with fog.

[0122] The above are only preferred embodiments of the present invention. It should be understood that the description of the above embodiments is only used to help understand the method and core ideas of the present invention, and is not used to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, etc. made within the ideas and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting lane lines for autonomous driving in rainy weather in mountainous cities, characterized by: The steps include: Step 1: Use the raindrop removal module to perform rain removal preprocessing on the input image to obtain a rain-removed image; Step 2: Use the lane detection module to perform lane detection on the derained image. The lane detection module includes a resizing layer and four hourglass modules. After the derained image is reduced in size by the resizing layer, it is input into the four hourglass modules for lane key point prediction and line fitting post-processing. The four hourglass modules have the same structure and loss function. A single hourglass module consists of a downstream bottleneck, a common bottleneck, an upstream bottleneck, and an output branch. The downstream bottleneck, common bottleneck, and upstream bottleneck are all composed of residual units. The residual units in the downstream bottleneck and common bottleneck are inserted with the channel attention mechanism SENet. The output branch contains confidence, offset, and embedded features. The residual unit structure in the downstream bottleneck is as follows: a 1×1 convolutional layer to reduce the number of channels; a convolutional layer with a kernel size of 3, a stride of 2, and a padding of 1 to reduce the feature map size and extract image features; another 1×1 convolutional layer to restore the initial number of channels; then the channel attention mechanism SENet is used to increase the weight of important channels; a convolutional layer with a kernel size of 3, a stride of 2, and a padding of 1 is used in the shortcut branch to reduce the size, retain the feature map details, and ensure the same size of the feature maps; The residual unit structure in the same bottleneck is as follows: a 1×1 convolutional layer to reduce the number of channels; a convolutional layer with a kernel size of 3, a stride of 1, and a padding of 1 to extract key image features; another 1×1 convolutional layer to restore the initial number of channels; then the channel attention mechanism SENet is used to increase the weight of important channels; a convolutional layer with a kernel size of 3, a stride of 1, and a padding of 1 is used in the shortcut branch to preserve the feature map details; The residual unit structure in the upstream bottleneck is: a 1×1 convolution layer for reducing the number of channels; a transposed convolution layer with a convolution kernel size of 3, a stride of 2, and a padding of 1 for restoring the feature map size; and a 1×1 convolution layer for restoring the initial number of channels; in the shortcut branch, bilinear interpolation upsampling is used to restore the image size.

2. The method for detecting lane lines for autonomous driving in rainy weather in mountainous cities according to claim 1, characterized in that: The channel attention mechanism SENet consists of a compression layer, an excitation layer and a scaling layer; the channel attention mechanism SENet performs a convolution operation on the input features and then performs global average pooling through the compression layer to compress the feature layer into a real number sequence, and then performs a nonlinear transformation in the excitation layer to obtain a real number sequence of each channel weight, and finally in the scaling layer, the weight of each channel is weighted to the original channel one by one by multiplication.

3. The method for detecting lane lines for autonomous driving in rainy weather in mountainous cities according to claim 1, characterized in that: The network loss function of the lane line detection module includes a confidence loss function, an offset loss function, an embedding feature loss function and a distillation loss function, wherein the distillation loss function is applied in the same bottleneck, and the confidence loss function, the offset loss function and the embedding feature loss function are applied to each output branch of the hourglass module.

4. The method for detecting lane lines for autonomous driving in rainy weather in mountainous cities according to claim 3, characterized in that: The confidence loss function includes an existence loss function and an absence loss function; the existence loss is used for cells with key points, and the absence loss is used to reduce the confidence value of cells without key points; if a grid cell contains a key point, its confidence value is close to 1, and if it does not contain a key point, it is 0; when the predicted confidence value of a cell is higher than 0.01, the absence loss is calculated; the definition of the confidence loss function is as follows: Among them, L exist Indicates that there is a loss function, L non_exist Indicates that there is no loss function and introduces the huber function To enhance the robustness to error points; N e Indicates the number of cells containing key points, N n Indicates the number of cells without key points, G e Represents a unit set consisting of key points, G n Represents the unit set consisting of all points, C c represents the confidence prediction value of each unit, represents the true value, and α is set to 2.5; The offset loss function is defined as follows: Among them, N e Indicates the number of cells containing key points, G e represents a unit set consisting of key points, c x Indicates the true x-axis offset value of the key point, c y Indicates the true y-axis offset value of the key point, Indicates the x-axis predicted offset value of the key point, Indicates the predicted y-axis offset value of the key point; The embedding feature loss function is defined as follows: Among them, N e Indicates the number of cells containing key points, F i represents the predicted embedding feature of cell i, F j represents the predicted embedding feature of cell j, K is a constant, K>0, l(i,j) represents the embedding feature difference between cells i and j; if I ij = 1, then cell i and cell j are the same instance, if I ij =0, then cell i and cell j are not the same instance; The distillation loss function is defined as follows: F(J h )=S(G(J d )),S:spatial softmax, where R C×H×W Represents a feature map with a channel dimension of H×W, R H×W represents a 2D feature map with a dimension of H×W; h represents the maximum number of hourglass modules, J d represents the distillation output of the same bottleneck of the d-th hourglass module, J di Indicates J d The i-th channel, C represents the total number of channels, G(J d ) represents the square sum of the distilled output feature map of the d-th hourglass module on C channels into a two-dimensional spatial feature map, F(J d ) represents the G(J d ) performs softmax activation on the space, F(J h ) represents the distillation output feature map of the h-th hourglass module, which is squared and converted into a two-dimensional spatial feature map on C channels and then subjected to spatial softmax activation. V(F(J h )-F(J d )) indicates the difference between F(J h ) and F(J d ) is squared and summed to obtain the total distillation loss; The weighted sum of the confidence loss function, offset loss function, embedding feature loss function, and distillation loss function constitutes the overall loss function of the lane detection module, which is expressed as follows: L total =c e L exist +g n L non-exist +g o L offset +g f L feature +g d L distillation Among them, γ o , γ f , γ d , γ e , γ n is the weight value of each loss function.

5. The method for detecting lane lines for autonomous driving in rainy weather in mountainous cities according to claim 1, characterized in that: The raindrop removal module consists of a generation network and a discriminant network, wherein the generation network includes an attention recurrent network and a contextual autoencoder; The attention recurrent network identifies raindrops and their surrounding areas in the input image, and then feeds the generated attention map to the contextual autoencoder to generate a clean image with raindrops removed; The discriminant network determines whether the generated clean image is real or fake.

6. The method for detecting lane lines for autonomous driving in rainy weather in mountainous cities according to claim 5, characterized in that: The attention recurrent network consists of four network blocks, each of which consists of a 5-layer ResNet, a convolutional LSTM unit and a convolutional layer; the contextual autoencoder consists of 16 Conv-ReLU blocks.

7. The method for detecting lane lines for autonomous driving in rainy weather in mountainous cities according to claim 5, characterized in that: The specific process of the discriminant network judging the authenticity of the generated clean image is as follows: the clean image generated by the generative network is used as the input of the discriminant network, and after passing through 5 Conv+Relu layers to obtain the original image features, a Convs operation is then performed to obtain an attention mask. The result obtained in the Convs process is multiplied by the original features to guide the discriminator to focus on the dense area of the attention map, and finally a fully connected layer is used to determine the authenticity of the image.

Images