Detection-driven foggy day image enhancement method based on half-channel Fourier transform

Abstract

The invention discloses an enhancement method for a foggy day image, which is particularly suitable for an automatic driving scene to solve the influence of the foggy weather on the image quality so as to improve the accuracy of target detection. The method comprises the following steps: firstly, carrying out two-dimensional discrete wavelet transform on a foggy day image, and respectively extracting high-frequency and low-frequency information of the image; for low-frequency information, a low-frequency enhancement model based on half-channel Fourier transform is adopted to enhance the global structure and contrast of the image. For high-frequency information, an adaptive direction detail enhancement module is adopted, and guidance information extracted from low-frequency information is combined, so that edge and texture details in different directions of the image are enhanced. And finally, combining the enhanced low-frequency and high-frequency information, and carrying out two-dimensional discrete wavelet inverse transformation to obtain a preliminary enhanced image. The enhanced image is combined with a standard YOLOX target detection model to form an end-to-end detection framework. Compared with an existing enhancement method, the method can effectively solve the problem of image blurring in automatic driving in foggy days, the generated image is more suitable for machine vision application, and the detection precision and robustness are remarkably improved.

Description

Technical Field

[0001] The invention relates to a detection-driven foggy image enhancement method based on half-channel Fourier transform, and belongs to the technical fields of computer vision, image enhancement, digital image processing, etc. Background Art

[0002] With the rapid development of autonomous driving technology, target detection plays a vital role as a key task in autonomous driving systems. However, severe weather conditions, especially haze weather, greatly affect the quality of images, which in turn affects the accuracy of target detection systems. Haze weather usually leads to decreased image contrast, loss of details, and blurred images, making it difficult for traditional target detection algorithms to identify targets, seriously affecting the performance and safety of autonomous driving systems in complex environments. Therefore, how to effectively improve image quality and enhance target detection capabilities in haze weather has become a difficult problem that needs to be solved in the field of autonomous driving.

[0003] Traditional image enhancement methods, such as histogram equalization and Retinex models, can improve the brightness and contrast of images to a certain extent, but these methods usually rely on artificially designed rules and assumptions and lack flexible processing for different image features. Although these methods can improve the overall visual effect of the image, they often cannot restore the lost details in the image. Especially in haze weather, the image blur and low contrast problems still exist, which cannot meet the requirements of the target detection system.

[0004] In recent years, deep learning methods have made significant progress in the field of image enhancement. Through end-to-end learning with convolutional neural networks (CNNs), many low-light image enhancement methods based on deep learning have shown excellent performance. Nevertheless, most of these methods focus on the visual effects perceived by the human eye, visually improving the image quality by enhancing the brightness, contrast and details of the image, and are suitable for general image display and perception. However, this enhancement method does not fully consider the needs of machine vision, especially in target detection tasks, these methods may not always provide enough information for target detection models to use. Images enhanced by deep learning often optimize the visual quality perceived by the human eye, but the improvement of machine vision (especially target detection) is limited. The target detection system requires that the image contains sufficient spatial information, edge information and texture features, while the enhancement method that simply improves the visual effect may enhance some misleading features while enhancing the contrast and details, which in turn affects the accuracy of target detection. Therefore, how to balance the visual quality and the effectiveness of target detection in image enhancement has become a key issue in the field of image processing.

[0005] In view of the above problems, the present invention proposes a detection-driven foggy image enhancement method, which can enhance foggy images in a targeted manner according to the needs of target detection, thereby effectively improving detection accuracy and robustness. Summary of the invention

[0006] In view of this, the technical problem to be solved by the present invention is to provide a detection-driven foggy image enhancement method based on half-channel Fourier transform, which can perform targeted enhancement of images according to target detection requirements in a complex foggy environment.

[0007] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0008] A detection-driven foggy image enhancement method based on half-channel Fourier transform, including:

[0009] Step 1) Collect and organize existing public foggy image datasets on the Internet and preprocess the images in the dataset;

[0010] Step 2) Perform a two-dimensional discrete wavelet transform on the input foggy image data X to obtain wavelet low-frequency information X_L and a set of wavelet high-frequency information {X_V, X_H, X_D};

[0011] Step 3) using a low frequency enhancement model based on half channel Fourier transform (LFEM-HCFT) to process the wavelet low frequency information to enhance the global structure and contrast in the image;

[0012] Step 4) Adopting the adaptive directional detail enhancement module (AODE) and combining the guidance information extracted from the low-frequency information, the edge and texture details of the image in different directions are enhanced;

[0013] Step 5) Use the enhanced low-frequency and high-frequency information to perform two-dimensional discrete wavelet inverse transform to reconstruct the image. And perform joint training with the YOLOX detection model to obtain the final enhanced result image.

[0014] Further, the step 3) specifically includes the following steps:

[0015] Step 31) First, extract the features of low-frequency information through three convolutional layers, and send the 1 / 4 channel feature map to four parallel convolutions to capture multi-scale features and enhance the global and local understanding of the image. The spliced ​​feature map is input into the half-channel Fourier transform enhancement module, and half of the channels are converted to the frequency domain. By adjusting the amplitude and phase, the image contrast is improved to make the global features of the image clearer. After weighting by the channel attention module, focus on the key features, and finally perform feature fusion.

[0016] Further, the step 4) specifically includes the following steps:

[0017] Step 41) For high-frequency information, first locate the relevant positions of the high-frequency and low-frequency information heights based on the spatial attention module, and extract the guidance information. And fuse the guidance information to the high-frequency information through affine transformation. Send the high-frequency information in different directions to the adaptive direction detail enhancement module, and generate the attention weights in the corresponding directions by performing pooling and compression operations on the global information of the input features, and enhance the details in different directions such as horizontal, vertical, and diagonal.

[0018] Further, the step 5) specifically includes the following steps:

[0019] Step 51) The enhanced low-frequency and high-frequency information is reconstructed into an image through a two-dimensional discrete wavelet inverse transform, and the enhancement network and the YOLOX detection model are combined to obtain the total loss through weighted image enhancement loss and detection loss, and finally an enhanced foggy image is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0021] Figure 1 This is the flow chart of the method;

[0022] Figure 2 This is the overall model framework diagram of this method;

[0023] Figure 3 This is the low-frequency enhancement structure diagram based on half-channel Fourier transform proposed in this method;

[0024] Figure 4 Figure 2. The adaptive directional detail enhancement structure proposed in this method. DETAILED DESCRIPTION

[0025] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0026] The present invention provides a detection-driven foggy image enhancement method based on half-channel Fourier transform, and the method flow chart is as follows: Figure 1 As shown in the figure, the overall model framework diagram proposed by this method is as follows Figure 2 As shown, the method comprises the following steps:

[0027] Step 1) construct a foggy target detection dataset;

[0028] Step 2) constructing a two-dimensional discrete wavelet transform reconstructed image model;

[0029] Step 3) constructing a low-frequency enhancement module based on half-channel Fourier transform;

[0030] Step 4) constructing a module with adaptive directional detail enhancement;

[0031] Step 5) construct a detection-driven foggy image enhancement network;

[0032] Step 6) training foggy image enhancement network;

[0033] Step 7) Test the foggy image enhancement network.

[0034] Further, step 1) specifically includes the following steps:

[0035] Step 11) Obtain an open source foggy weather target detection dataset, which contains five categories: bicycle, bus, car, motorcycle, and person. Use annotation software to annotate the target in each picture and generate an annotation file.

[0036] Step 12) Divide the obtained data set into training set, test set and validation set in a ratio of 8:1:1.

[0037] Further, step 2) specifically includes the following steps:

[0038] Step 21) Perform a two-dimensional discrete wavelet transform on the input foggy image data X to obtain wavelet low-frequency information X_L and a set of wavelet high-frequency information {X_V, X_H, X_D}. The image can be reconstructed by a two-dimensional discrete wavelet inverse transform, and its decomposition and reconstruction process can be expressed by formula (1)-formula (2).

[0039]

[0040] Where h(k) represents a high-pass filter and g(k) represents a low-pass filter. By processing the longitudinal and lateral information respectively, a low-frequency component X_L and three high-frequency components {X_V, X_H, X_D} are obtained.

[0041] Further, step 3) specifically includes the following steps:

[0042] Step 31) Figure 3 As shown in the figure, the low-frequency enhancement module based on half-channel Fourier transform (LFEM-HCFT) includes a multi-scale feature extraction module, a half-channel Fourier transform contrast enhancement module, and a channel attention weighting module. For different spatial scales, 1×1, 3×3, 5×5, and 7×7 convolution blocks are introduced to capture the key features of the image at multiple scales, which can well describe the environment-specific knowledge such as the diffusion of fog on the entire image.

[0043] Step 32) Half-channel Fourier transform First, the input features of half of the channels are transformed into global information through the global average pooling operation, and then Fourier transform is performed to the frequency domain space, and the amplitude and phase features are adjusted through different convolution operations. Taking the C×H×W tensor as an example, first split its channels, perform channel Fourier transform on the C / 2×H×W tensor, and further analyze its global statistical characteristics in the frequency domain. Finally, perform inverse Fourier transform and combine the other half of the features. The global average pooling operation is as follows:

[0044]

[0045] Where X s It contains the average value of each channel and performs a one-dimensional Fourier transform on it. The transformation formula is as follows:

[0046]

[0047] Where X k is the transformed frequency domain feature, and k is the corresponding index.

[0048] Further, step 4) specifically includes the following steps:

[0049] Step 41) Figure 4 As shown, the module in step 3 is used to obtain the enhanced low-frequency information and the original low-frequency information as input, and the spatial attention module locates the relevant positions of the high-frequency and low-frequency information heights to extract the guidance information. And the affine transformation is used to fuse the guidance information to the high-frequency information. The affine transformation formula used in the present invention is as follows:

[0050] M(F i |α i , β i )=α i ⊙F i +β i

[0051] where F i represents the extracted high-frequency information, α i and β i are the scaling and shifting parameters of the i-th decomposition level, respectively.

[0052] Step 42) Figure 4 As shown in the figure, the fused features are fed into the Adaptive Directional Detail Enhancement module (AODE), which consists of an average pooling layer, a convolution layer, and an activation function layer. First, the average is taken in the X and Y directions, the two are concatenated, and nonlinear factors are introduced to enable the network to learn more complex information. Finally, the spatial information is fused by weighting the channels. This module is used for feature maps in the horizontal, vertical, and diagonal directions, respectively, to enhance specific directions.

[0053] Further, step 5) specifically includes the following steps:

[0054] Step 51) Construct a joint training model of the dehazing image enhancement network and the target detection network. The image reconstructed by the two-dimensional discrete wavelet inverse transform is used as the input of the target detection YOLOX model to form an end-to-end detection framework. The loss function is optimized in the joint training. The loss of the enhancement network and the loss of the detection are calculated according to certain weights to obtain the total loss. The detection loss includes classification loss, bounding box loss, etc., and the enhancement loss includes absolute error loss, mean square error loss, etc. Some formulas are as follows:

[0055]

[0056] The above formula is the classification loss, where y i is the true label of the sample, p i Is the predicted probability of the sample. It is used to compare the probability distribution of a given class with the true label.

[0057]

[0058] The above formula is the absolute error loss, where y i (h,w) is the pixel value at position (h,w) in the reconstructed image, y j (h,w) is the pixel value at position (h,w) in the original image.

[0059] Further, step 6) specifically includes the following steps:

[0060] Step 61) configure the corresponding configuration, set the training parameters, and carry out the training task;

[0061] Step 62) The training environment configured is: Ubuntu22.04 LTS, Python3.8 and Pytorch2.0.0. The hardware equipment includes CPU: Intel(R) Xeon(R) CPU E5-2680 v4@2.40GHz, memory is 32GB; GPU: NVIDIARTX 3090, memory is 24GB.

[0062] Step 63) The training parameters are configured as follows: the number of target categories is 5, the input image size is 1333x800, the batch size is set to 25, the learning rate is 0.001, and the number of training rounds is 300.

[0063] Further, step 7) specifically includes the following steps:

[0064] Load the trained weight file into the foggy object detection network, randomly select images from the test dataset for detection, and return labeled images with the location and category information of the target to be detected.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A detection-driven foggy image enhancement method based on half-channel Fourier transform, characterized in that: The following steps are involved: Step 1, collect relevant data sets for foggy weather target detection and preprocess them: download the open source foggy weather target detection data set from the public website, and divide it into training set, test set and validation set in the ratio of 8:1:

1. There are five categories of detection targets, namely bicycles, buses, cars, motorcycles and people. Use labeling software to label the target images of the initial data set, mark various targets in the image with specific boxes, generate corresponding annotation files, and use them as training data sets together with the original image files; Step 2, construct a two-dimensional discrete wavelet transform image reconstruction module: First, perform a two-dimensional discrete wavelet transform on the input foggy image data X to obtain the wavelet low-frequency information X_L and a set of wavelet high-frequency information {X_V, X_H, X_D}, which represent the high-frequency information of the image in the vertical, horizontal and diagonal directions respectively. The low-frequency part contains the main structural information of the image, while the high-frequency part captures the details and edge information of the image. And the image can be reconstructed through the two-dimensional discrete wavelet inverse transform; Step 3, construct a low-frequency enhancement module based on half-channel Fourier transform: This module consists of a multi-scale feature extraction module and a half-channel Fourier transform module. First, after three layers of convolution, a 3×3 convolution operation is used to preliminarily extract image features. In order to effectively capture the key information of the image at different spatial scales, the module then introduces 1×1, 3×3, 5×5 and 7×7 convolution kernels, and sends 1 / 4 channel feature maps to four parallel convolutions to capture multi-scale features, and capture multi-scale features to enhance the global and local understanding of the image. The half-channel Fourier transform module first divides the channel equally, performs a global average pooling operation on half of the channel features to convert the feature expression into global information, performs a Fourier transform to the frequency domain space, and adjusts the amplitude and phase features through different convolution operations to make the global features of the image clearer. Then perform an inverse Fourier transform and combine the other half of the channel features. After weighting through the channel attention module, focus on key features, and finally perform feature fusion through three layers of 3×3 convolution; Step 4, construct an adaptive directional detail enhancement module: by fusing the enhanced low-frequency information and the original low-frequency information as input, and locating the highly correlated positions of the high-frequency and low-frequency information through the spatial attention module, the guidance information is extracted. Subsequently, the guidance information is fused into the high-frequency information using an affine transformation to enhance its feature performance. The adaptive directional detail enhancement module includes an adaptive average pooling layer, a convolution layer, and an activation function layer, which are used to process feature maps in the horizontal, vertical, and diagonal directions, respectively, to achieve detail enhancement in a specific direction and further improve the dehazing effect of the image; Step 5: Build a detection-driven foggy image enhancement network: Use the image reconstructed by the two-dimensional discrete wavelet inverse transform as the input of the target detection YOLOX model to form an end-to-end detection framework. Optimize the loss function in joint training. Calculate the total loss by using a certain weight for the loss of the enhancement network and the loss of the detection. Step 6, training the foggy image enhancement network: using the training set to fully train the task-driven foggy image enhancement network consisting of a two-dimensional discrete wavelet transform reconstruction network, a low-frequency enhancement module based on half-channel Fourier transform, and an adaptive directional detail enhancement module, to obtain the trained network model weights; Step 7, test the foggy image enhancement network: load the trained weight file into the foggy image enhancement network, randomly select pictures from the test set for detection, return the labeled pictures with the location and category information of the target to be tested, and obtain the detection accuracy and detection speed of each category in the foggy image dataset, as well as the overall average detection accuracy.

2. The detection-driven foggy image enhancement method based on half-channel Fourier transform according to claim 1 is characterized in that: The wavelet transform and image reconstruction module constructed in step 2 processes foggy images in the following steps: First, perform a two-dimensional discrete wavelet transform on the input foggy image data to obtain wavelet low-frequency information X_L and a set of wavelet high-frequency information {X_V, X_H, X_D}, which represent the high-frequency information of the image in the vertical, horizontal and diagonal directions respectively. The low-frequency part contains the main structure and contour information of the image, while the high-frequency part captures the details, edges and noise of the image. Then, through the two-dimensional discrete wavelet inverse transform, the low-frequency and high-frequency information are fused to reconstruct a clear image. In this process, the wavelet transform can effectively separate the various frequency components of the image, which is convenient for subsequent haze removal and image enhancement processing. In this way, the multi-scale information of the image can be effectively extracted, and higher-quality image reconstruction can be achieved, thereby improving the clarity and detail performance of the foggy image.

3. The detection-driven foggy image enhancement method based on half-channel Fourier transform according to claim 1 is characterized in that: The low-frequency enhancement module based on half-channel Fourier transform constructed in step 3 processes the low-frequency components in the following steps: First, the low-frequency information is subjected to three layers of 3×3 convolution for preliminary feature extraction to capture the local information of the image. In order to capture the key information of the image more comprehensively at different spatial scales, the module introduces multi-scale convolution operations, in which 1×1, 3×3, 5×5 and 7×7 convolution kernels are used, and the feature maps of 1 / 4 channels are respectively sent to four parallel convolution layers, thereby capturing the global and local features of the image at multiple scales, effectively enhancing the multi-dimensional understanding ability of the image. Next, the half-channel Fourier transform module divides the input features into channels, extracts global information from the feature maps of half channels through global average pooling operations, and performs Fourier transform in the frequency domain space to obtain the amplitude and phase features of the image. These amplitude and phase features are adjusted through different convolution operations to make the global information of the image clearer. Subsequently, the inverse Fourier transform is performed and combined with the features of the other half channel. Finally, after the weighted operation of the channel attention module, the key features in the image are highlighted, and the features are further fused through three layers of 3×3 convolution to enhance the low-frequency information of the image, thereby improving the clarity and quality of the image.

4. The detection-driven foggy image enhancement method based on half-channel Fourier transform according to claim 1 is characterized in that: The steps of processing the image by the adaptive directional detail enhancement module constructed in step 4 are as follows: First, the enhanced low-frequency information is fused with the original low-frequency information as input, and the highly correlated areas of high-frequency and low-frequency information are accurately located through the spatial attention module to extract key guidance information. Then, the extracted guidance information is fused into the high-frequency information using affine transformation, thereby enhancing the expressiveness of high-frequency features and improving the dehazing effect of the image. The adaptive directional detail enhancement module consists of multiple layers, including an adaptive average pooling layer, a convolution layer, and an activation function layer, which are used to process the feature maps of the horizontal, vertical, and diagonal directions of the image respectively. In each direction, the module enhances the details of a specific direction through a convolution operation, thereby improving the detail expression of the image in different directions and further improving the visual effect of the image. Ultimately, the module effectively improves the dehazing quality of the image, making the image clearer and more detailed.

Images