Image defogging method, system, electronic device and storage medium

Abstract

The present invention relates to the field of image processing technology, and in particular to an image defogging method, system, electronic device and storage medium. The image defogging method comprises: constructing an image defogging model and training it; wherein the image defogging model comprises: a shallow feature extraction module, a feature fusion module and an image reconstruction module; the shallow feature extraction module is used to extract shallow primary features of a sample foggy image; the feature fusion module is used to encode, decode and fuse shallow primary features to obtain fused image features; the image reconstruction module is used to perform image channel conversion on the fused image features to obtain converted image features, and fuse the converted image features with the sample foggy image to obtain a sample defogging image; the foggy image to be processed is input into a trained image defogging model to obtain a target defogging image. The present invention improves the efficiency and accuracy of defogging, effectively eliminates the influence of fog on the image, and restores clearer and more accurate image details.

Description

Background Art

[0002] In recent years, with the rapid development of domestic industrial production, automobile exhaust and industrial waste gas have increasingly polluted the air, and scenes such as daily traffic and road monitoring are frequently adversely affected by haze weather. In a haze environment, natural light will be absorbed or scattered by fine particles suspended in the air during propagation, resulting in different degrees of degradation of outdoor visual systems, especially images taken in traffic scenes, showing low contrast, loss of details and color distortion, which seriously affects the effective operation of scenes such as road monitoring and daily traffic. Image dehazing algorithms aim to restore potential haze-free images from given haze images. Due to its wide range of applications, single image dehazing has become a hot topic in the field of computer vision and image processing. Traditional image dehazing methods are mainly based on image priors and atmospheric scattering models. Such methods usually require time-consuming iterative optimization and manually designed priors, so they perform poorly in complex haze scenes. In recent years, deep learning has been widely used in image dehazing tasks. In order to achieve good performance, deep learning-based dehazing methods generally design deep and complex neural networks, and the high computational and memory requirements make it difficult to adapt to devices with different computing capabilities.

[0003] Therefore, it is urgent to provide a technical solution to solve the above problems. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides an image defogging method, system, electronic device and storage medium.

[0005] In a first aspect, the present invention provides an image defogging method, and the technical solution of the method is as follows:

[0006] Construct an image defogging model and perform training; wherein the image defogging model includes: the shallow feature extraction module, the feature fusion module and the image reconstruction module; the shallow feature extraction module is used to extract the shallow primary features of the sample foggy image; the feature fusion module is used to encode, decode and fuse the shallow primary features to obtain the fused image features; the image reconstruction module is used to perform image channel conversion on the fused image features to obtain the converted image features, and fuse the converted image features with the sample foggy image to obtain the sample defogging image;

[0007] The foggy image to be processed is input into the trained image defogging model to obtain the target defogging image.

[0008] The beneficial effects of an image defogging method of the present invention are as follows:

[0009] The method of the present invention improves the efficiency and accuracy of defogging, effectively eliminates the influence of fog on the image, and restores clearer and more accurate image details.

[0010] Based on the above solution, an image defogging method of the present invention can also be improved as follows.

[0011] In an optional manner, the shallow feature extraction module includes: the first convolution layer; the shallow feature extraction module is specifically used to: perform convolution processing on the sample foggy image through the first convolution layer to obtain the shallow primary features.

[0012] In the above optional method, through the processing of the first convolutional layer, the number of channels of the sample foggy image is expanded, and the number of feature maps is increased to better capture the feature information of the image, ensuring the effective extraction of shallow primary features of the sample foggy image.

[0013] In an optional manner, the feature fusion module includes: the encoder, the decoder and the feature fusion submodule; the feature fusion module is specifically used to: use the encoder to perform multi-layer encoding processing on the shallow primary features to obtain multiple encoding features, use the decoder and the feature fusion submodule to perform multi-layer decoding on the multiple encoding features to obtain multiple decoding features, and perform feature fusion of the decoding features of each layer with the encoding features in turn to obtain the fused image features.

[0014] In the above optional method, through the combination of encoder, decoder and feature fusion sub-module, multi-layer encoding, decoding and feature fusion of shallow primary features are realized, which improves the ability to handle complex fog in the image dehazing process and enhances the image structure and detail recovery capabilities.

[0015] In an optional manner, the encoder includes: a first dynamic large core defogging unit, a second dynamic large core defogging unit, a third dynamic large core defogging unit, a first downsampling unit, and a second downsampling unit; the decoder includes: a fourth dynamic large core defogging unit, a fifth dynamic large core defogging unit, a sixth dynamic large core defogging unit, a first upsampling unit, and a second upsampling unit; the feature fusion submodule includes: a first feature fusion unit and a second feature fusion unit; the multiple encoding features include: a first encoding feature, a second encoding feature, and a third encoding feature; the multiple decoding features include: a first decoding feature, a second decoding feature, and a third decoding feature;

[0016] The first dynamic large core defogging unit is used to process the shallow primary feature to obtain the first coding feature; the first downsampling unit is used to downsample the first coding feature to obtain the first downsampling feature; the second dynamic large core defogging unit is used to process the first downsampling feature to obtain the second coding feature; the second downsampling unit is used to downsample the second coding feature to obtain the second downsampling feature; the third dynamic large core defogging unit is used to process the second downsampling feature to obtain the third coding feature;

[0017] The fourth dynamic large core defogging unit is used to decode the third encoding feature to obtain a first decoding feature; the first upsampling unit is used to upsample the first decoding feature to obtain a first upsampling feature; the first feature fusion unit is used to fuse the second encoding feature and the first upsampling feature to obtain a first intermediate image feature; the fifth dynamic large core defogging unit is used to decode the first intermediate image feature to obtain a second decoding feature; the second upsampling unit is used to upsample the second decoding feature to obtain a second upsampling feature; the second feature fusion unit is used to fuse the first encoding feature and the second upsampling feature to obtain a second intermediate image feature; the sixth dynamic large core defogging unit is used to decode the second intermediate image feature to obtain the fused image feature.

[0018] In the above optional methods, the use of dynamic large core defogging unit can adaptively adjust the defogging strategy according to the fog density in different areas, improving the pertinence and accuracy of defogging. The combination of downsampling and upsampling units helps to extract and process features at different scales, further enhancing the defogging effect. The feature fusion submodule ensures the effective fusion of encoding features and decoding features, retaining more image information and details.

[0019] In an optional manner, the image reconstruction module includes: a second convolutional layer; using the second convolutional layer, performing image channel conversion on the fused image features to obtain the three-channel converted image features, and fusing the converted image features with the sample foggy image to obtain a sample defogged image.

[0020] In the above optional method, the fused image features are converted into image channels through the second convolutional layer, which ensures that the converted image features match the original sample foggy image in the channel dimension, and then a high-quality defogging image is obtained through fusion.

[0021] In a second aspect, the present invention provides an image defogging system, the technical solution of the system is as follows:

[0022] The image defogging system comprises: a construction system and an operation system;

[0023] The construction system is used to: construct an image defogging model and perform training; wherein the image defogging model includes: a shallow feature extraction module, a feature fusion module and an image reconstruction module; the shallow feature extraction module is used to extract shallow primary features of a sample foggy image; the feature fusion module is used to encode, decode and fuse the shallow primary features to obtain fused image features; the image reconstruction module is used to perform image channel conversion on the fused image features to obtain converted image features, and fuse the converted image features with the sample foggy image to obtain a sample defogging image;

[0024] The operating system is used to: input the foggy image to be processed into a trained image defogging model to obtain a target defogging image.

[0025] The beneficial effects of an image defogging system of the present invention are as follows:

[0026] The system of the present invention improves the efficiency and accuracy of defogging, effectively eliminates the influence of fog on the image, and restores clearer and more accurate image details.

[0027] Based on the above solution, the image defogging system of the present invention can also be improved as follows.

[0028] In an optional manner, the shallow feature extraction module includes: the first convolutional layer; the shallow feature extraction module is specifically used for:

[0029] The sample foggy image is convolved through the first convolution layer to obtain the shallow primary features.

[0030] In the above optional method, through the processing of the first convolutional layer, the number of channels of the sample foggy image is expanded, and the number of feature maps is increased to better capture the feature information of the image, ensuring the effective extraction of shallow primary features of the sample foggy image.

[0031] In an optional manner, the feature fusion module includes: the encoder, the decoder and the feature fusion submodule; the feature fusion module is specifically used for:

[0032] The encoder is used to perform multi-layer encoding processing on the shallow primary features to obtain multiple encoding features, and the decoder and the feature fusion submodule are used to perform multi-layer decoding on the multiple encoding features to obtain multiple decoding features, and the decoding features of each layer are sequentially fused with the encoding features to obtain the fused image features.

[0033] In the above optional method, through the combination of encoder, decoder and feature fusion sub-module, multi-layer encoding, decoding and feature fusion of shallow primary features are realized, which improves the ability to handle complex fog in the image dehazing process and enhances the image structure and detail recovery capabilities.

[0034] In a third aspect, a technical solution of an electronic device of the present invention is as follows:

[0035] The invention comprises a memory, a processor and a program stored in the memory and running on the processor. When the processor executes the program, the steps of the image defogging method of the present invention are implemented.

[0036] In a fourth aspect, the present invention provides a computer-readable storage medium having the following technical solution:

[0037] Instructions are stored in the computer-readable storage medium. When the computer-readable storage medium reads the instructions, the computer-readable storage medium executes the steps of the image defogging method of the present invention.

[0038] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0040] Figure 1 A schematic diagram of a flow chart of an embodiment of an image defogging method of the present invention;

[0041] Figure 2 It is a network structure diagram of the image defogging model of the present invention;

[0042] Figure 3 It is a structural diagram of the dynamic large core defogging unit (DLDB) of the present invention;

[0043] Figure 4 It is a structural diagram of the dynamic large kernel convolution (DLC) of the present invention;

[0044] Figure 5 is a structural diagram of the feature fusion unit (AFF) of the present invention;

[0045] Figure 6 This is a comparison diagram of the defogging effect in the traffic scene of the present invention;

[0046] Figure 7is a structural schematic diagram of an embodiment of an image defogging system of the present invention;

[0047] Figure 8 The figure is a schematic structural diagram of an embodiment of an electronic device of the present invention. DETAILED DESCRIPTION

[0048] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0049] Figure 1 FIG. 1 is a flow chart of an embodiment of an image defogging method provided by the present invention, wherein the method is performed by an image defogging device. Figure 1 As shown, the following steps are included:

[0050] S1. Construct an image defogging model and perform training; wherein the image defogging model includes: a shallow feature extraction module, a feature fusion module and an image reconstruction module; the shallow feature extraction module is used to extract shallow primary features of the sample foggy image; the feature fusion module is used to encode, decode and fuse shallow primary features to obtain fused image features; the image reconstruction module is used to perform image channel conversion on the fused image features to obtain converted image features, and fuse the converted image features with the sample foggy image to obtain a sample defogging image. In S1:

[0051] ① The sample foggy image is a foggy image in the constructed dataset used to train the model, and the clarity of the image needs to be optimized.

[0052] ② Shallow primary features are obtained by convolution processing of the input sample foggy image through the shallow feature extraction module in the image dehazing model, especially the first convolution layer. This feature contains the local details and basic information of the image and is the basis for subsequent processing steps.

[0053] ③The fused image features are the image features obtained after being processed by the encoder, decoder and feature fusion sub-module, which integrates multiple levels of encoding and decoding information.

[0054] ④The converted image features are features that have been converted into image channel format after being processed by the image reconstruction module.

[0055] ⑤ The sample defogging image is an image obtained by fusing the converted image features with the sample foggy image, which is an image sample with fog removed.

[0056] Specifically, the image defogging device first constructs and trains an image defogging model including a shallow feature extraction module, a feature fusion module and an image reconstruction module. Next, the image defogging device performs convolution processing on the input sample foggy image through the first convolution layer in the shallow feature extraction module, thereby extracting the shallow primary features of the image. The shallow primary features contain the local details and basic information of the image and are the basis for subsequent processing steps. Then, the image defogging device uses the encoder and decoder in the feature fusion module to perform multi-layer encoding and decoding processing on the shallow primary features, and fuses the decoding features of each layer with the encoding features through the feature fusion submodule to finally obtain the fused image features. The fused image features fuse multiple levels of encoding and decoding information, providing rich information support for the defogging process. After obtaining the fused image features, the image defogging device performs image channel conversion on them through the image reconstruction module to obtain the converted image features. The converted image features have been converted into image channel format, which is convenient for subsequent fusion with the sample foggy image. Finally, the image defogging device fuses the converted image features with the sample foggy image to obtain the sample defogging image. This image is a sample of an image with fog removed, and its clarity has been significantly improved. Throughout the process, the image defogging device achieves effective defogging of foggy images by building and training an image defogging model.

[0057] S2: Input the foggy image to be processed into the trained image defogging model to obtain the target defogging image. In S2:

[0058] ① The foggy image to be processed is a foggy image affected by fog collected in the target scene. The image is directly collected image data with fog.

[0059] ② The target defogging image is obtained after being processed by the trained image defogging model, and is the target image with the fog removed.

[0060] Specifically, after completing the construction and training of the image defogging model, the image defogging device further inputs the original image with fog directly collected, that is, the foggy image to be processed, into the trained image defogging model. The model can perform defogging on the foggy image to be processed, remove the fog interference in the image, and finally obtain the target image with the fog removed, that is, the target defogging image. Compared with the original foggy image to be processed, the target defogging image has higher clarity and can more truly reflect the scene and content in the image.

[0061] The technical solution of this embodiment improves the efficiency and accuracy of defogging, effectively eliminates the influence of fog on the image, and restores clearer and more accurate image details.

[0062] In an optional manner, the shallow feature extraction module includes: a first convolution layer; the shallow feature extraction module is specifically used to: perform convolution processing on the sample foggy image through the first convolution layer to obtain shallow primary features.

[0063] Among them, for the input sample foggy image I∈R 3×H×W , through the first convolutional layer Conv3×3, the sample foggy image I∈R 3×H×W Expand the number of channels from 3 channels (RGB) to C to obtain the shallow primary features F0∈R of the image C ×H×W R represents a real number set, indicating that each pixel value of the sample foggy image I is a real number; C represents the number of channels of the sample foggy image I; H represents the height of the sample foggy image I, indicating the number of pixels of the sample foggy image I in the vertical direction. W represents the width of the sample foggy image I, indicating the number of pixels of the sample foggy image I in the horizontal direction.

[0064] Specifically, the image defogging device further uses the shallow feature extraction module to perform convolution processing on the input sample foggy image using the first convolution layer, and extracts shallow primary features containing basic texture, edge and other information from the original image. These features can help the device identify the blurred areas and detail information in the image, so as to perform defogging operations more accurately. Through the processing of this module, the image defogging device can obtain richer image information.

[0065] It should be noted that before the image defogging model is trained, the process of constructing a data set for training the image defogging model is: applying the atmospheric scattering model to perform a fogging operation on the sample clear image of the target scene, and obtaining multiple sample foggy images with different concentrations and types. The atmospheric scattering model formula is:

[0066] I(z)=J(z)e -βd(z) +A(1-e -βd(z) )

[0067] Among them, I(z) is the pixel value of the sample foggy image, J(z) is the pixel value of the sample clear image, A represents the atmospheric light value, β represents the scattering coefficient, which controls the concentration of fog, d(z) represents the depth of field information, and e -βd(z) Indicates the ratio of energy attenuation during light propagation, J(z)e -βd(z) Represents the intensity of light when it propagates through the atmosphere and reaches the observation point, A(1-e -βd(z)) represents the scattered light intensity of the atmosphere itself. In this embodiment, by adjusting the scattering coefficient and the parameters of the atmospheric light, fog of different concentrations and different types can be simulated. The sample clear image and the sample foggy image are paired, and the data set is divided into a training set and a test set at a ratio of 9:1. The training set is used to iteratively train the image defogging model, and the test set is used to evaluate the defogging performance of the image defogging model.

[0068] In an optional manner, the feature fusion module includes: an encoder, a decoder and a feature fusion submodule; the feature fusion module is specifically used to: use the encoder to perform multi-layer encoding processing on shallow primary features to obtain multiple encoding features, use the decoder and the feature fusion submodule to perform multi-layer decoding on the multiple encoding features to obtain multiple decoding features, and fuse the decoding features of each layer with the encoding features in turn to obtain fused image features.

[0069] Among them, the encoder includes: a first dynamic large core defogging unit, a second dynamic large core defogging unit, a third dynamic large core defogging unit, a first downsampling unit, and a second downsampling unit; the decoder includes: a fourth dynamic large core defogging unit, a fifth dynamic large core defogging unit, a sixth dynamic large core defogging unit, a first upsampling unit, and a second upsampling unit; the feature fusion submodule includes: a first feature fusion unit and a second feature fusion unit; multiple encoding features include: a first encoding feature, a second encoding feature, and a third encoding feature; multiple decoding features include: a first decoding feature, a second decoding feature, and a third decoding feature.

[0070] Among them, the first dynamic large core defogging unit is used to process the shallow primary features to obtain the first coding features; the first downsampling unit is used to downsample the first coding features to obtain the first downsampling features; the second dynamic large core defogging unit is used to process the first downsampling features to obtain the second coding features; the second downsampling unit is used to downsample the second coding features to obtain the second downsampling features; the third dynamic large core defogging unit is used to process the second downsampling features to obtain the third coding features.

[0071] Among them, the fourth dynamic large core defogging unit is used to decode the third encoding feature to obtain the first decoding feature; the first upsampling unit is used to upsample the first decoding feature to obtain the first upsampling feature; the first feature fusion unit is used to fuse the second encoding feature and the first upsampling feature to obtain the first intermediate image feature; the fifth dynamic large core defogging unit is used to decode the first intermediate image feature to obtain the second decoding feature; the second upsampling unit is used to upsample the second decoding feature to obtain the second upsampling feature; the second feature fusion unit is used to fuse the first encoding feature and the second upsampling feature to obtain the second intermediate image feature; the sixth dynamic large core defogging unit is used to decode the second intermediate image feature to obtain the fused image feature.

[0072] Figure 2 The network structure diagram of the image defogging model in this embodiment is shown. Among them, the encoder includes three layers of dynamic large core defogging units and two downsampling units. The dynamic large core defogging unit is responsible for extracting and processing higher-level coding features from shallow to deep layers, and gradually building up an understanding of fog density and distribution; the downsampling layer is responsible for halving the spatial resolution of the coding features and doubling the number of channels to help the network extract higher-level coding features. The decoder includes three layers of dynamic large core defogging units and two upsampling units. The dynamic large core defogging unit is responsible for re-expanding and refining the coding features extracted and compressed by the encoder, eliminating the impact of fog on the image, reconstructing the structure and details of the image, and obtaining decoding features. The upsampling layer is responsible for doubling the spatial resolution of the decoding features, halving the number of channels, and gradually restoring the spatial resolution and details of the feature map.

[0073] Figure 3 The structural diagram of the dynamic large kernel dehazing unit (DLDB) in this embodiment is shown. Each DLDB is a Transformer-style convolution residual block, and each DLDB includes: a dynamic large kernel convolution block and a convolution feedforward network. The calculation formula of the dynamic large kernel convolution block is as follows: Y = Linear(DLC(ReLU(Linear(X))))+X; X represents the input feature of the dynamic large kernel convolution block, Y represents the output feature of the dynamic large kernel convolution block, Linear represents the linear layer, DLC represents the dynamic large kernel convolution, and ReLU represents the rectified linear unit. DLC provides a dynamic adaptive large receptive field for the network. Its construction process is divided into two steps: (1) In order to achieve a large receptive field in a low-cost manner without increasing the depth of the network, a large convolution kernel is efficiently decomposed. For a K×K large kernel conventional convolution, it can be decomposed into a cascaded small kernel deep convolution (DWConv) and a small kernel deep dilated convolution (DW-D Conv). The size of the decomposed convolution kernel is expressed as: K DWConv =(2d-1)×(2d-1); Where d represents the dilation factor of the deep dilated convolution. (2) Considering that a fixed large receptive field is difficult to adapt to haze degradation features of different sizes, this embodiment constructs two decomposition convolution branches with kernels of different sizes. First, the input feature X is input into two convolution branches with different kernel sizes to obtain features X1 and X2, and then X1 and X2 are fused by element-by-element addition. The fused information is sent to a series of global average pooling-linear layer-Softmax operations to calculate the weight information of the two convolution channel dimensions. Finally, the convolution block feature map is generated by weighted summation. The structure of dynamic large kernel convolution (DLC) is as follows Figure 4 As shown, the calculation formula is: X1, X2 = DWDConv (DWConv (X)); α, β = softmax (Linear (GAP (X1 + X2))); Among them, DWConv represents the deep convolution operation, DWDConv represents the deep dilated convolution operation, GAP represents the global average pooling, Linear represents the linear layer, Softmax represents the normalized exponential function. α and β represent the weights of the two convolution branches.

[0074] In the traditional Transformer structure, the feed-forward network (Feed-Forward Network) consists of two fully connected layers (linear layers) and an activation function (such as ReLU). The feed-forward network ignores the modeling of spatial information. Therefore, this embodiment introduces a 3×3 deep convolution layer between hidden layers to better capture spatial information. Similar to the layered Vision Transformer, this embodiment applies a layer normalization (LN) in each dynamic large kernel convolution block and convolution feed-forward network. Therefore, the l-th layer dynamic large kernel defogging unit can be calculated as: Among them, X l-1 represents the output of the l-1th layer dynamic large kernel dehazing unit, DLC represents dynamic large kernel convolution, CFFN represents convolutional feedforward network, X l Represents the output of the l-th layer dynamic large kernel dehazing unit.

[0075] Figure 5 FIG. 1 shows a structural diagram of any feature fusion unit (AFF) in this embodiment. The feature fusion submodule plays a bridge role in the encoder and decoder structure, and the feature fusion submodule includes: a first feature fusion unit and a second feature fusion unit. Figure 5 As shown, each feature fusion unit is divided into two branches, which use channel information and spatial information to calibrate the fusion information respectively. Specifically:

[0076] 1) In the upper branch, the encoded features F from the encoder are firste The decoded features F at the resolution corresponding to the decoder stage d Splicing along the channel dimension to obtain the splicing features, and then cascading a global average pooling layer (GAP), a convolution layer and a Sigmooid activation function to obtain the channel attention map A ch , then multiply the attention map and the concatenated features, and then restore the number of channels to the original number through a 1×1 convolution layer. In this process, the channel information guides the convolution layer to retain the most critical features while discarding unimportant features: A ch =Sigmoid(Conv(GAP([F e ,F d ])); Sigmoid represents the S-shaped function, Conv represents the 1×1 convolutional layer, Represents the output features of the upper branch.

[0077] 2) The lower branch is to simulate the encoding feature F e and decoding feature F d The spatial dependency between them, spatial information w sp It is captured by a 1×1 convolutional layer and a Sigmoid activation function. In the lower branch, the intermediate image feature F is obtained by element-wise multiplication of the outputs of the two branches: sp =Sigamoid(Conv(F e )+Conv(F d ); denotes a sigmoid function and Conv denotes a 1×1 convolutional layer.

[0078] Specifically, the image defogging device further performs deep processing on the shallow primary features through a feature fusion module. The module consists of an encoder, a decoder and a feature fusion submodule. First, the first dynamic large core defogging unit processes the shallow primary features to obtain a first encoding feature. Next, the first downsampling unit downsamples the first encoding feature to obtain a first downsampled feature. Then, the second dynamic large core defogging unit processes the first downsampled feature to obtain a second encoding feature. Similarly, the second downsampling unit downsamples the second encoding feature to obtain a second downsampled feature. Finally, the third dynamic large core defogging unit processes the second downsampled feature to obtain a third encoding feature. The fourth dynamic large core defogging unit decodes the third encoding feature to obtain a first decoded feature. Subsequently, the first upsampling unit upsamples the first decoded feature to obtain a first upsampled feature. The first feature fusion unit fuses the second encoding feature with the first upsampled feature to obtain a first intermediate image feature. The purpose of this step is to combine lower-level detail information with higher-level context information to enrich image features. Next, the fifth dynamic large core defogging unit decodes the first intermediate image feature to obtain the second decoded feature. Then, the second upsampling unit upsamples the second decoded feature to obtain the second upsampled feature. Finally, the second feature fusion unit fuses the first encoded feature and the second upsampled feature to obtain the second intermediate image feature. This step further integrates features at different levels, making the image features more comprehensive and accurate. Finally, the sixth dynamic large core defogging unit decodes the second intermediate image feature to obtain a fused image feature. This fused image feature contains the key information of the image at different scales, providing strong support for subsequent image defogging processing.

[0079] In this embodiment, through the design of the encoder-decoder structure and the feature fusion submodule, the image defogging device can make full use of the multi-level information in the image, improve the defogging effect, and improve the details and clarity of the processed image.

[0080] In an optional manner, the image reconstruction module includes: a second convolutional layer; using the second convolutional layer, performing image channel conversion on the fused image features to obtain three-channel converted image features, and fusing the converted image features with the sample foggy image to obtain a sample defogged image.

[0081] The image reconstruction layer uses the second convolutional layer Conv3×3 to reduce the dimension of the fused image features output by the decoder. This step mainly reduces the number of channels of the feature map from C channels to 3 channels to match the standard format of RGB images, thereby obtaining the converted image features. The sample defogging image is finally obtained by adding the converted image features to the sample foggy image element by element.

[0082] Specifically, the image defogging device further processes the fused image features through an image reconstruction module to generate a final sample defogging image. In the image reconstruction module, the second convolutional layer is used. The main function of the second convolutional layer is to perform image channel conversion on the fused image features. Through this step, the fused image features are converted into three-channel converted image features, which match the common RGB image channels and facilitate subsequent image display and processing. Subsequently, the image defogging device fuses the converted image features with the original sample foggy image. The purpose of this step is to combine the defogging image features with the original image to retain some details and color information in the original image, while removing the influence of fog to obtain a clearer and more detailed sample defogging image. In the above manner, the image defogging device can effectively combine the defogging image features with the original image to generate a high-quality sample defogging image.

[0083] It should be noted that the specific steps of using the constructed dataset to train the image defogging model, obtain the pre-trained defogging model, and apply it to the device to achieve the defogging effect are as follows:

[0084] Before the sample foggy image is input into the image defogging model for training, random cropping and random flipping are first used to enhance the data, and then the sample foggy image data is converted into a tensor form and input into the image defogging model for training. The L1 loss function is selected to quantify the pixel-level error between the sample defogging image (predicted value) corresponding to the sample foggy image and the real defogging image (real value) corresponding to the sample foggy image, and the contrast perception loss is introduced to further drive the image defogging model training. The overall loss of the image defogging model is: Among them, G i , i=1,2,3…n represents the i-th layer feature extracted from the fixed pre-trained model VGG-19, D(x,y) represents L1 loss, β is a hyperparameter that balances L1 loss and contrast perception loss, the left half of the overall loss represents the difference between the clear image J and the sample defogging image φ(I,w), and the right half represents the contrast perception loss between the foggy image J to be processed, the sample foggy image I, and the sample defogging image φ(I,w). In addition, the network uses the Adam optimizer (β1=0.9, β2=0.999) to optimize the model parameters, and the learning rate is increased from 2e to 1 through the cosine annealing strategy. -4 Reduced to 2e -6. Input the image data into the image defogging model in batches for training, and use the test set to evaluate the performance and generalization ability of the model, and use indicators such as PSNR and SSIM to quantify the quality of the defogging effect. When the performance indicators are met, save the pre-trained model parameters for fine-tuning or direct deployment in the next application. Deploy the pre-trained image defogging model to the device, such as a surveillance camera or a traffic control system, and send the foggy image to be processed obtained by the device to the image defogging model for processing to obtain the target defogging image, and optimize the image defogging model according to the performance in the actual application scenario.

[0085] Figure 6 is a comparison chart of the defogging effect of the method of this embodiment, wherein Figure 6 The left part is the foggy image to be processed. Figure 6 The right part represents the target defogging image after being processed by the method of this embodiment.

[0086] Figure 7 FIG. 2 is a schematic diagram showing a structure of an embodiment of an image defogging system 200 provided by the present invention. Figure 2 As shown, the system 200 includes: a construction system 210 and an operation system 220 .

[0087] The construction system 210 is used to: construct an image defogging model and perform training; wherein the image defogging model includes: a shallow feature extraction module, a feature fusion module and an image reconstruction module; the shallow feature extraction module is used to extract shallow primary features of the sample foggy image; the feature fusion module is used to encode, decode and fuse the shallow primary features to obtain fused image features; the image reconstruction module is used to perform image channel conversion on the fused image features to obtain converted image features, and fuse the converted image features with the sample foggy image to obtain a sample defogging image.

[0088] The operating system 220 is used to: input the foggy image to be processed into a trained image defogging model to obtain a target defogging image.

[0089] In an optional manner, the shallow feature extraction module includes: a first convolution layer; the shallow feature extraction module is specifically used to: perform convolution processing on the sample foggy image through the first convolution layer to obtain shallow primary features.

[0090] In an optional manner, the feature fusion module includes: an encoder, a decoder and a feature fusion submodule; the feature fusion module is specifically used to: use the encoder to perform multi-layer encoding processing on shallow primary features to obtain multiple encoding features, use the decoder and the feature fusion submodule to perform multi-layer decoding on the multiple encoding features to obtain multiple decoding features, and fuse the decoding features of each layer with the encoding features in turn to obtain fused image features.

[0091] The technical solution of this embodiment improves the efficiency and accuracy of defogging, effectively eliminates the influence of fog on the image, and restores clearer and more accurate image details.

[0092] The above parameters and steps for each module to implement corresponding functions in the image defogging system 200 of this embodiment can refer to the parameters and steps in the above embodiment of the image defogging method, which will not be repeated here.

[0093] like Figure 8 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, the processor 320 is coupled to a memory 310, and the memory 310 stores at least one computer program 330, and the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the above-mentioned image defogging methods, specifically:

[0094] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein at least one computer program 330 is stored in the one or more memories 310, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements any image defogging method provided in the above embodiments. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the electronic device 300 may also include other components for implementing device functions, which will not be repeated here.

[0095] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements any of the above-mentioned image defogging methods.

[0096] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0097] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above-mentioned image defogging methods.

[0098] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0099] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0100] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0101] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. An image defogging method, characterized in that: include: An image defogging model is constructed and trained; wherein the image defogging model comprises: a shallow feature extraction module, a feature fusion module and an image reconstruction module; the shallow feature extraction module is used to extract shallow primary features of the sample foggy image; the feature fusion module is used to encode, decode and fuse the shallow primary features to obtain fused image features; the image reconstruction module is used to perform image channel conversion on the fused image features to obtain converted image features, and fuse the converted image features with the sample foggy image to obtain a sample defogging image; Input the foggy image to be processed into the trained image defogging model to obtain the target defogging image; The feature fusion module includes: an encoder, a decoder and a feature fusion submodule; the feature fusion module is specifically used for: Using the encoder to perform multi-layer encoding processing on the shallow primary features to obtain multiple encoding features, using the decoder and the feature fusion submodule to perform multi-layer decoding on the multiple encoding features to obtain multiple decoding features, and sequentially perform feature fusion on the decoding features of each layer with the encoding features to obtain the fused image features; The encoder comprises: a first dynamic large core defogging unit, a second dynamic large core defogging unit, a third dynamic large core defogging unit, a first downsampling unit, and a second downsampling unit; the decoder comprises: a fourth dynamic large core defogging unit, a fifth dynamic large core defogging unit, a sixth dynamic large core defogging unit, a first upsampling unit, and a second upsampling unit; the feature fusion submodule comprises: a first feature fusion unit and a second feature fusion unit; the multiple encoding features comprise: a first encoding feature, a second encoding feature, and a third encoding feature; the multiple decoding features comprise: a first decoding feature, a second decoding feature, and a third decoding feature; The first dynamic large core defogging unit is used to process the shallow primary feature to obtain the first coding feature; the first downsampling unit is used to downsample the first coding feature to obtain the first downsampling feature; the second dynamic large core defogging unit is used to process the first downsampling feature to obtain the second coding feature; the second downsampling unit is used to downsample the second coding feature to obtain the second downsampling feature; the third dynamic large core defogging unit is used to process the second downsampling feature to obtain the third coding feature; The fourth dynamic large core defogging unit is used to decode the third encoding feature to obtain a first decoding feature; the first upsampling unit is used to upsample the first decoding feature to obtain a first upsampling feature; the first feature fusion unit is used to fuse the second encoding feature and the first upsampling feature to obtain a first intermediate image feature; the fifth dynamic large core defogging unit is used to decode the first intermediate image feature to obtain a second decoding feature; the second upsampling unit is used to upsample the second decoding feature to obtain a second upsampling feature; the second feature fusion unit is used to fuse the first encoding feature and the second upsampling feature to obtain a second intermediate image feature; the sixth dynamic large core defogging unit is used to decode the second intermediate image feature to obtain the fused image feature.

2. The image defogging method according to claim 1, characterized in that: The shallow feature extraction module includes: a first convolutional layer; the shallow feature extraction module is specifically used for: The sample foggy image is convolved through the first convolution layer to obtain the shallow primary features.

3. The image defogging method according to claim 1, characterized in that: The image reconstruction module includes: a second convolutional layer; The second convolutional layer is used to perform image channel conversion on the fused image features to obtain the three-channel converted image features, and the converted image features are fused with the sample foggy image to obtain a sample defogged image.

4. An image defogging system, characterized in that: include: Building systems and running systems; The construction system is used to: construct an image defogging model and perform training; wherein the image defogging model includes: a shallow feature extraction module, a feature fusion module and an image reconstruction module; the shallow feature extraction module is used to extract shallow primary features of a sample foggy image; the feature fusion module is used to encode, decode and fuse the shallow primary features to obtain fused image features; the image reconstruction module is used to perform image channel conversion on the fused image features to obtain converted image features, and fuse the converted image features with the sample foggy image to obtain a sample defogging image; The operating system is used to: input the foggy image to be processed into a trained image defogging model to obtain a target defogging image; The feature fusion module includes: an encoder, a decoder and a feature fusion submodule; the feature fusion module is specifically used for: Using the encoder to perform multi-layer encoding processing on the shallow primary features to obtain multiple encoding features, using the decoder and the feature fusion submodule to perform multi-layer decoding on the multiple encoding features to obtain multiple decoding features, and sequentially perform feature fusion on the decoding features of each layer with the encoding features to obtain the fused image features; The encoder comprises: a first dynamic large core defogging unit, a second dynamic large core defogging unit, a third dynamic large core defogging unit, a first downsampling unit, and a second downsampling unit; the decoder comprises: a fourth dynamic large core defogging unit, a fifth dynamic large core defogging unit, a sixth dynamic large core defogging unit, a first upsampling unit, and a second upsampling unit; the feature fusion submodule comprises: a first feature fusion unit and a second feature fusion unit; the multiple encoding features comprise: a first encoding feature, a second encoding feature, and a third encoding feature; the multiple decoding features comprise: a first decoding feature, a second decoding feature, and a third decoding feature; The first dynamic large core defogging unit is used to process the shallow primary feature to obtain the first coding feature; the first downsampling unit is used to downsample the first coding feature to obtain the first downsampling feature; the second dynamic large core defogging unit is used to process the first downsampling feature to obtain the second coding feature; the second downsampling unit is used to downsample the second coding feature to obtain the second downsampling feature; the third dynamic large core defogging unit is used to process the second downsampling feature to obtain the third coding feature; The fourth dynamic large core defogging unit is used to decode the third encoding feature to obtain a first decoding feature; the first upsampling unit is used to upsample the first decoding feature to obtain a first upsampling feature; the first feature fusion unit is used to fuse the second encoding feature and the first upsampling feature to obtain a first intermediate image feature; the fifth dynamic large core defogging unit is used to decode the first intermediate image feature to obtain a second decoding feature; the second upsampling unit is used to upsample the second decoding feature to obtain a second upsampling feature; the second feature fusion unit is used to fuse the first encoding feature and the second upsampling feature to obtain a second intermediate image feature; the sixth dynamic large core defogging unit is used to decode the second intermediate image feature to obtain the fused image feature.

5. The image defogging system according to claim 4, characterized in that: The shallow feature extraction module includes: a first convolutional layer; the shallow feature extraction module is specifically used for: The sample foggy image is convolved through the first convolution layer to obtain the shallow primary features.

6. An electronic device, characterized in that: The electronic device includes a processor, the processor is coupled to a memory, at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the image defogging method as described in any one of claims 1 to 3.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the computer-readable storage medium implements the image defogging method according to any one of claims 1 to 3.

Images