Lightweight image defogging method in intelligent driving scene

Abstract

The invention discloses a lightweight image defogging method in an intelligent driving scene. The invention provides a defogging method suitable for various foggy weather conditions by considering the problems of blurring of images collected by an intelligent automobile, reduction of color saturation, loss of detail information and the like in foggy weather. According to the method, an extraction module fusing multi-scale features is utilized, and the fuzzy problem caused by heavy fog features is reduced. Object information in a scene is highlighted by applying an attention mechanism to high-level features. Bidirectional feature fusion and a pixel shuffling strategy are designed to enrich information in a decoding process, and the authenticity of a subsequent pixel-level defogging effect is improved. The method has the advantages that compared with a frontier image defogging algorithm, the method has a higher processing speed while having a higher defogging effect, and is more suitable for the real-time defogging requirement in an automatic driving scene.

Description

Technical Field

[0001] The present invention belongs to the field of image defogging in intelligent driving scenarios, and relates to an image defogging method using multi-scale feature extraction, attention mechanism, bidirectional feature fusion and pixel shuffling strategy. Background Art

[0002] In the field of autonomous driving, due to the presence of heavy fog, the collected images are blurred. Pedestrians, vehicles, road signs, traffic signs, etc. in traffic scenes will be blurred, color saturation will decrease, and detailed information will be lost due to the influence of fog, which will inhibit the expression of relevant target features, resulting in a decrease in the accuracy of target detection algorithms such as pedestrians, vehicles, and obstacles on the road, and then affect subsequent intelligent driving algorithms such as path planning, which can easily cause traffic safety accidents and endanger the safety of life and property of drivers and others. To this end, in order to reduce or eliminate the impact of fog on intelligent driving, related research can be divided into image defogging methods based on image enhancement, image restoration methods based on priors, and image defogging methods based on deep learning.

[0003] Image defogging algorithms based on image enhancement mainly use technical methods such as histogram equalization and Retinex. The histogram equalization defogging method makes the histogram distribution of the image more uniform through correlation operations, improves the contrast of the image, and achieves the purpose of image defogging. The Retinex method performs color constancy based on the characteristics of human vision's perception of the color and brightness of the environment, decomposes the received image into a reflected object image and an incident light image, estimates the incident light with the help of a mathematical model, and obtains the reflected object property image through the object image, that is, removes the incident light from the object image to obtain the original property of the object. Although these methods can remove fog simply and quickly, they can only be used as conventional defogging methods under simple backgrounds and are not suitable for complex backgrounds.

[0004] The image restoration and defogging method based on prior mainly uses the imaging law to extract the prior information of the image to be processed, and combines the physical model of foggy imaging in the atmospheric medium to infer the fog-free image. Based on this, the dark channel prior defogging algorithm and the color attenuation prior defogging algorithm can be used. The former finds that in most non-sky area images, there is always one channel in the three RGB color channels that can be ignored as 0. For this reason, the dark channel of any foggy image can be calculated, and then the perspective is obtained for defogging. The latter considers the influence of fog concentration changes on image brightness and saturation. In the fog-free area of ​​the image, the pixels have the characteristics of low brightness and high saturation, while the foggy area is the opposite. The fog concentration in the foggy image is proportional to the difference in image brightness and saturation, and the fog-free image is inverted. This type of defogging method based on prior can better retain the original image information, but it is easy to have incomplete defogging and color distortion.

[0005] Image defogging methods based on deep learning are mainly divided into two categories: defogging methods based on learning priors and end-to-end defogging methods. The former is still based on the atmospheric scattering model and uses deep learning to estimate the parameters in the physical model of foggy imaging. The transmittance and atmospheric light value of the foggy image are estimated through a convolutional neural network to obtain the corresponding transmission map, and then the fog-free image is derived based on the foggy image and the transmission map with the help of the physical model. The latter directly outputs the defogging image by learning the mapping relationship between the foggy image and the fog-free image without using the physical model of foggy imaging. Summary of the invention

[0006] In view of this, the technical problem to be solved by the present invention is to provide a lightweight image defogging method in an intelligent driving scenario, which can quickly improve the perception effect of foggy image information in an intelligent driving scenario.

[0007] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0008] A lightweight image defogging method for intelligent driving scenarios, including:

[0009] Step 1) Obtain a foggy data set in an intelligent driving scenario and pre-process and divide the images in the data set;

[0010] Step 2) Input the preprocessed image into the encoder module that integrates multi-scale convolution to increase the model's attention to detail information and reduce the impact of fog occlusion on subsequent machine vision tasks;

[0011] Step 3) Input the last layer of features extracted by the encoder into a feature fusion module based on the channel attention mechanism to improve the expression ability of the high semantic feature map and reduce the impact of fog features on object features;

[0012] Step 4) The features extracted by the encoder and decoder are input into a bidirectional fusion module based on jump connection. The module performs multi-source fusion design and supplements detail information in the process of the decoder generating the dehazed image, making the details of the enhanced foggy image clearer.

[0013] Step 5) The features extracted by the encoder are input into the pixel shuffling based decoder module, which performs upsampling and scale restoration on the feature map, avoiding the chess and card grid problem caused by deconvolution.

[0014] Step 6) Apply the curve adjustment feature map output by the encoder to the foggy image in the autonomous driving scene, and achieve defogging through pixel-level image adjustment.

[0015] Further, the step 1) specifically includes the following steps:

[0016] Step 11) Division of foggy dataset in autonomous driving scenario: The prepared fire image dataset is divided into training set, verification set and test set in a ratio of 8:1:1.

[0017] Further, the step 2) specifically includes the following steps:

[0018] Step 21) Multi-scale processing: Use convolution kernels of different sizes to design different feature extraction branches, obtain image features under different receptive fields, and improve the information sensitivity of the model.

[0019] Step 22) Multi-scale fusion: The extracted feature maps of different scales are weighted fused to obtain a fused feature map with multi-scale expression capability.

[0020] Step 23) Construct an encoder. Using multi-scale convolution blocks as basic units, normalization and activation operations are added to perform step-by-step multi-scale feature extraction of the encoder.

[0021] Further, the step 3) specifically includes the following steps:

[0022] Step 31) Global information extraction: Perform global average pooling on the feature map extracted by the last layer of the encoder to generate a global description vector in the channel dimension, which covers the global information of each channel.

[0023] Step 32) Full connection and non-linear activation. The vector obtained by global average pooling is input into the fully connected layer, and the ReLU activation function is used to introduce non-linear features.

[0024] Step 33) Weight generation and application. Use the Sigmoid activation function to limit the output value between 0 and 1, generate the weight coefficient of each channel, and multiply the generated weight coefficient by the original input feature map element by channel to suppress the response of unimportant feature channels.

[0025] Further, the step 4) specifically includes the following steps:

[0026] Step 41) Skip connection: Connect each stage of the encoder to the corresponding layer of the decoder, input high-resolution features to the decoding stage, and improve the information fidelity of the decoding stage.

[0027] Step 42) Multi-source fusion. Using multi-level image features, two fusions are performed in the network. On the one hand, each layer of feature information is fused twice, and on the other hand, the features provided by the jump connection and the features extracted by the bottleneck are fused.

[0028] Further, the step 5) specifically includes the following steps:

[0029] Step 51) Design a pixel shuffling module. Convert the size of the feature map and adjust the dimensional order of the tensor to rearrange the feature map to a new spatial position to achieve upsampling operation.

[0030] Step 52) Construct a decoder. Taking the pixel shuffling-based feature extraction module as the basic unit, add convolution, normalization and activation operations to realize the decoder to generate the pixel-level curve parameter map.

[0031] Further, the step 6) specifically includes the following steps:

[0032] Step 61) Curve parameter optimization: The curve parameters are obtained using the parameter feature map output by the decoder, and then the curve is adjusted pixel by pixel for the foggy image in the intelligent driving scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0034] Figure 1 This is the flow chart of the method;

[0035] Figure 2 This is the overall model framework diagram of this method;

[0036] Figure 3 This is the structure diagram of the encoder based on multi-scale convolution proposed in this method;

[0037] Figure 4 The bottleneck module structure diagram based on the channel attention mechanism proposed in this method;

[0038] Figure 5 This is the structural diagram of the feature fusion module proposed in this method;

[0039] Figure 6 This is the module structure diagram of the pixel shuffling based decoder proposed in this method. DETAILED DESCRIPTION

[0040] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0041] The present invention provides a lightweight image defogging method in an intelligent driving scenario, and the method flow chart is as follows: Figure 1 As shown in the figure, the overall model framework diagram proposed by this method is as follows Figure 2 As shown, the method comprises the following steps:

[0042] Step 1) Divide the foggy data set under the intelligent driving scenario;

[0043] Step 2) construct an encoder based on multi-scale convolution;

[0044] Step 3) construct a bottleneck module based on the channel attention mechanism;

[0045] Step 4) construct a bidirectional fusion module;

[0046] Step 5) construct a decoder based on pixel shuffling;

[0047] Step 6) constructing a curve parameter optimization module;

[0048] Step 7) Building a lightweight image defogging network for intelligent driving scenarios;

[0049] Step 8) training a lightweight image defogging network for intelligent driving scenarios;

[0050] Step 9) Test the lightweight image dehazing network in the intelligent driving scenario.

[0051] Further, step 1) specifically includes the following steps:

[0052] Step 11) Obtain foggy images in intelligent driving scenarios from open source channels. The dataset contains different road conditions, different foggy conditions, and different lighting conditions, covering a variety of driving environments.

[0053] Step 12) The obtained data set is randomly divided into a training set and a test set in a ratio of 8:2.

[0054] Further, step 2) specifically includes the following steps:

[0055] Step 21) The input foggy image is first batch normalized so that the subsequent convolutional layer can better learn the image features. Then, the normalized image is sent to the multi-scale convolution module.

[0056] Step 22) Use the convolution operation shown in formula (1) to extract feature information representing different viewing ranges of the image from the image, and the convolution kernel sizes used are 3×3, 5×5, and 7×7 respectively.

[0057]

[0058] represents the feature map of the i-th channel of the (l-1)-th layer, represents the jth convolution kernel bias, represents the feature map of the jth channel of the lth layer, p and q represent the number of channels of the input and output feature maps, and σ represents the activation function and convolution operation.

[0059] Step 23) The output features from convolution kernels of different sizes are concatenated in the channel direction and the number of channels is adjusted. The output features after adjusting the channels are fused with the original features using formula (2) to obtain a feature map with multi-scale expression.

[0060] F = BN(Cat(Conv 3×3 (X), Conv 5×5 (X), Conv 7×7 (X), X id ) (2)

[0061] Step 23) Figure 3 As shown in the figure, in the encoder, multi-scale convolution operation is the core, and a downsampling operation is performed after each feature extraction. The spatial scale of the feature map is reduced and the channel dimension is expanded.

[0062] Further, step 3) specifically includes the following steps:

[0063] Step 31) Figure 4 As shown, the channel attention mechanism is used in the bottleneck structure to enhance feature expression, and the feature compression operation is performed using formula (3).

[0064]

[0065] where z c is the global average of the c-th channel.

[0066] Step 32) uses a structure with two fully connected layers to recalibrate the global description z. The first fully connected layer reduces the dimension from C to C / r, and then the second fully connected layer increases the dimension back to C. Where r is the scaling ratio, the process is shown in formula (4)-formula (5).

[0067] s=ReLU(W 1 z+b 1 ) (4)

[0068]

[0069] Step 33) Recalibrate the weights Reuse for each channel of the feature map output by the bottleneck layer, the process is shown in formula (5).

[0070]

[0071] Further, step 4) specifically includes the following steps:

[0072] Step 41) Figure 5As shown, the same convolutional structure is used to adjust the scale of the feature maps of the corresponding layers of the skip connection, encoder, and decoder, and the feature maps from three different sources are summed element by element.

[0073] Step 42) The obtained fused feature map is cascaded with the feature map after the encoder is upsampled to obtain the final fused feature map.

[0074] Further, step 5) specifically includes the following steps:

[0075] Step 51) Use convolution operation to transform the feature map of size H×W×C into H×W×r 2 C, where r is the upsampling factor.

[0076] Step 52) Figure 6 As shown, the pixel shuffle operation is used to upsample the feature map. The pixel shuffle operation with an upsampling factor r of 3 converts the size of H×W×r 2 The feature map of C sequentially rearranges the 2r pixels at the same position of the channel into a single-channel pixel block of size r×r, thereby realizing an upsampling operation on the entire input feature map.

[0077] Further, step 6) specifically includes the following steps:

[0078] Step 61) As shown in formula (7), A is the parameter adjustment map output by the decoder with the same size as the original image, h is the foggy image in the intelligent driving scenario, and C n-1 (x) represents the image after the last dehazing, C n (x) represents the final dehazed image.

[0079] C n (x) = C n-1 (x)+A n (x)h(x)(1-h(x)) (7)

[0080] Further, step 7) specifically includes the following steps:

[0081] Step 71) Use the modules constructed in steps 2, 3, 4, 5 and 6 to build Figure 2 The lightweight image defogging network in the intelligent driving scenario is shown in the figure. The encoder, bottleneck module, decoder and curve parameter optimization are designed to adapt to the scene problem and optimize the final image defogging effect.

[0082] Further, step 8) specifically includes the following steps:

[0083] Step 81) configure the corresponding configuration, set the training parameters, and carry out the training task;

[0084] Step 82) The training environment configured is: Ubuntu20.04LTS, Python3.8 and Pytorch2.0.0. The hardware equipment includes CPU: Intel(R)Xeon(R)CPU E5-2680 v4@2.40GHz, memory is 32GB; GPU: NVIDIA RTX 3090, memory is 24GB.

[0085] Step 83) The training parameters are configured as follows: the input image size is 256×256, the batch size is set to 8, the Adam random optimization is used to train the network, and the update rate is set to 0.001.

[0086] Further, step 9) specifically includes the following steps:

[0087] Load the trained weight file into the image dehazing network for intelligent driving scenarios, select pictures from the test dataset for testing, and return clear images of the scene outside the car.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A lightweight image defogging method in an intelligent driving scenario, characterized in that: The following steps are involved: Step 1: Collect image datasets of foggy weather types in various intelligent scenarios and preprocess them: collect them from the Internet and offline, and cover different foggy scene instances as much as possible, including light fog, dense fog, non-uniform fog, cluster fog and other real foggy scenes under different lighting environments. After the dataset is formed, it is divided into training set and test set in a ratio of 8:

2. Step 2, build the encoder: The core of this part is the introduction of multi-scale convolution blocks, which use convolution kernels of different sizes to obtain feature maps of different field of view ranges to improve the ability to express details. This method is used as the basic module of the encoder, which can effectively reduce the information loss in the encoder's step-by-step feature extraction process and reduce the suppression of fog features on objects in driving scenes. Step 3, build a bottleneck module: This module is based on the channel attention mechanism. By re-weighting the features of each channel, it improves the network's attention to the key features in the scene, while reducing the impact of useless or noisy features. It introduces nonlinear transformation in the channel dimension, improves the network's nonlinear expression ability, and enhances the network's performance in processing different types and concentrations of fog, making the dehazed image clearer and more natural. Step 4: Construct a fusion module: This module fuses the feature map output by the jump connection and the feature map of the same layer of the encoder and decoder, which not only enriches the information guarantee of the encoder in the process of generating the curve adjustment map, but also retains the initial information of the original foggy image. In this way, the curve adjustment map is more consistent with the distribution of image content, and the visual effect of the enhanced image is better. Step 5, build the decoder: This part uses the pixel shuffling strategy for upsampling. By rearranging and upsampling the low-resolution feature map, pixel shuffling can effectively convert the spatial information of the feature map into a higher-resolution output, so that the dehazed image overcomes the problem of checkerboard generated by deconvolution and improves the texture and details of the dehazed image. Step 6: Construct the curve parameter optimization module: This part obtains more detailed information from the original input foggy image, making up for the information missing during the first enhancement. At the same time, it also solves the problem of over-reliance on the previous enhancement results during iteration. Step 7, using the modules constructed in steps 2, 3, 4, 5, and 6, combined with normalization layers, activation functions, and other operations, a lightweight image defogging network for intelligent driving scenarios is built. First, the encoder is designed using multi-scale convolution modules to enhance the rich scene information in foggy days. Secondly, a channel attention mechanism is applied to the bottleneck structure to suppress redundant information of high-level features. Then, the feature maps from different sources are fused to provide more feature information for the decoder. Finally, the pixel shuffling strategy is used to upsample in the decoding stage to further improve the visibility of the enhanced results. Step 8, training a lightweight image dehazing network for intelligent driving scenarios: using various types of foggy images in real driving scenarios as network input, using a decentralized fusion approach to combine spatial consistency loss, color constancy loss, and illumination smoothness loss, and supervise the training of a lightweight image dehazing network for intelligent driving scenarios, in order to obtain fog-free images with high visual perception effects. Step 9, test the lightweight image defogging network in the intelligent driving scenario: load the trained weight file into the defogging network, use the foggy image in the real driving environment as the test case, and return the enhanced image and the corresponding evaluation indicators such as NIQE and PIQE.

2. The lightweight image defogging method in the intelligent driving scenario according to claim 1 is characterized in that: The encoder based on multi-scale convolution constructed in step 2 extracts image features in the following steps: First, the input foggy image is scaled uniformly. Then, convolution kernels of different sizes are used for feature extraction. Finally, features of different granularities are fused to improve the expression of the extracted features.

3. The lightweight image defogging method in the intelligent driving scenario according to claim 1 is characterized in that: The bottleneck module constructed in step 3 performs self-weighting on the highest-level feature map of the encoder to distinguish the importance of different features in the channel dimension, thereby suppressing the expression of irrelevant information.

4. The lightweight image defogging method in an intelligent driving scenario according to claim 1, characterized in that: The fusion module constructed in step 4 integrates the features of three different information sources: skip connection, encoder and decoder, as the feature composition of the decoder. It not only reduces the information loss during the decoding process, but also enhances the fidelity of the original image content.

5. The lightweight image defogging method in the intelligent driving scenario according to claim 1, characterized in that: The encoder built in step 5 uses the pixel shuffling strategy for the upsampling process, effectively avoiding image artifacts caused by the chess and card grid effect.

6. The lightweight image defogging method in the intelligent driving scenario according to claim 1, characterized in that: The parameter optimization module constructed in step 6 uses iterative processing to perform step-by-step enhancement based on the previous round of enhancement, and adjusts the original foggy image pixel by pixel. This not only reduces the problem of over-reliance on the previous round of enhancement results during the iteration process, but also improves the detail expression of the adjusted image.

Images