Image defogging method, electronic device, storage medium and computer program product

Unsupervised training through unpaired clear images and haze images, and the multi-scale attention module and discriminator are used to solve the domain difference problem of defog removal model training in the existing technology, improve the defog removal performance, and ensure the visual system normal operation.

CN114004760BActive Publication Date: 2024-11-29BEIJING UNIV OF TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202111234337.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-22
Publication Date
2024-11-29
Estimated Expiration
2041-10-22

AI Technical Summary

Technical Problem

The limitation of paired image training sets on defogging model training in the prior art leads to defogging performance defogging, especially when real haze images and clear images are difficult to collect, supervised training based on synthetic data leads to domain differences problems.

Method used

Unsupervised training using unpaired clear images and haze images, using multi-scale attention modules and discriminators, through fog conversion and defog conversion processing, combined with adversarial loss and other loss functions, Go out the fog model.

Benefits of technology

The performance of the defog removal model is improved, the limitation of paired image training sets is avoided, the model training can be carried out based on real data, the defog removal ability of real haze images is enhanced, and the normal operation of the visual system is ensured.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114004760B_ABST
    Figure CN114004760B_ABST
Patent Text Reader

Abstract

The present invention provides an image defogging method, an electronic device, a storage medium and a computer program product. The method comprises obtaining a target haze image to be defogged; inputting the target haze image into a defogging model, defogging the target haze image, and obtaining a target defogged image output by the defogging model. The defogging model is obtained by performing unsupervised training on a training image set consisting of unpaired clear images and haze images, and the training model comprises a multi-scale attention module for performing fogging conversion processing and defogging conversion processing, and a discriminator for distinguishing between real images of the training image set and generated images of the multi-scale attention module. The defogging model of the present invention is obtained by performing unsupervised training on a training image set consisting of unpaired clear images and haze images, thereby avoiding the limitation of paired image training sets on defogging model training, and further improving the performance of image defogging.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image defogging method, electronic equipment, storage medium and computer program product. Background Art

[0002] Image dehazing has attracted wide attention as an important visual enhancement technology. It can be applied to monitoring systems, UAV visual navigation systems, and autonomous driving systems. Specifically, when images acquired by image or video acquisition devices are affected by bad weather such as rain, snow, and haze, outdoor visual systems are prone to instability. Haze, as a complex image degradation factor, can seriously reduce the visibility of images and cause loss of edge, color, and detail information. Clear, fog-free images are a prerequisite for the normal operation of many vision-based outdoor systems. Therefore, dehazing of single images is essential.

[0003] Currently, the dehazing problem of a single image is solved by using paired images for supervised training. However, in practice, it is difficult and expensive to collect real haze images and their corresponding clear images, so it is not possible to obtain a large number of real haze images and their corresponding clear images for supervised training of neural networks. Based on this, most learning-based methods are trained on synthetic datasets in a supervised manner, and due to the domain difference between synthetic data and real data, the dehazing performance of these methods will drop significantly when applied to real images.

[0004] In summary, how to improve the dehazing performance of haze images is a problem that needs to be solved urgently. Summary of the invention

[0005] The present invention provides an image defogging method, an electronic device, a storage medium and a computer program product, which are used to solve the defect in the prior art that the paired image training set limits the training of the defogging model, resulting in a decrease in the defogging performance, and realize the defogging model obtained by unsupervised training through a training image set consisting of unpaired clear images and haze images.

[0006] The present invention provides an image defogging method, comprising:

[0007] Obtain a target haze image to be defogged;

[0008] The target haze image is input into a defogging model, and the target haze image is defogged to obtain a target defogged image output by the defogging model. The defogging model is obtained by performing unsupervised training on a training image set consisting of unpaired clear images and haze images. The model to be trained includes a multi-scale attention module for performing haze conversion processing and defogging conversion processing, and a discriminator for distinguishing between real images of the training image set and generated images of the multi-scale attention module.

[0009] According to an image defogging method provided by the present invention, the defogging model is trained by the following method:

[0010] Obtain unpaired clear image training samples and haze image training samples;

[0011] Based on the clear image training samples, the haze image training samples and the preset loss function, the constructed model to be trained is trained to obtain a defogging model, wherein the model to be trained includes a multi-scale attention module for performing haze conversion processing and defogging conversion processing, and a discriminator for distinguishing between real images and generated images, wherein the real image is the clear image training sample or the haze image training sample, and the generated image is the generated image output by the multi-scale attention module.

[0012] According to an image defogging method provided by the present invention, based on the clear image training sample, the haze image training sample and a preset loss function, the constructed model to be trained is trained to obtain a defogging model, including:

[0013] Based on the constructed haze encoder of the to-be-trained model, feature extraction is performed on the haze image training sample to obtain haze information;

[0014] Based on the first multi-scale attention module of the model to be trained and the haze information, performing fogging conversion processing on the clear image training sample to obtain a first fogged image;

[0015] Based on the first discriminator of the model to be trained and a first adversarial loss function in a preset loss function, performing adversarial loss calculation on the first fogged image to obtain a first adversarial loss corresponding to the haze image;

[0016] Based on the second multi-scale attention module of the model to be trained, performing dehazing conversion processing on the haze image training sample to obtain a first dehazed image;

[0017] Based on the second discriminator of the model to be trained and the second adversarial loss function in the preset loss function, performing adversarial loss calculation on the first dehazed image to obtain a second adversarial loss corresponding to a clear image;

[0018] Based on the first adversarial loss and the second adversarial loss, model training is performed on the model to be trained to obtain a defogging model.

[0019] According to an image defogging method provided by the present invention, the haze encoder based on the constructed model to be trained performs feature extraction on the haze image training sample to obtain haze information, including:

[0020] Based on the first convolution unit of the haze encoder, performing a convolution operation on the haze image training sample to obtain a first feature map;

[0021] Based on the second convolution unit of the haze encoder, feature extraction is performed on the first feature map to obtain a second feature map, and based on the third convolution unit of the haze encoder, feature extraction is performed on the first feature map to obtain a third feature map, and the second feature map and the third feature map have different resolutions;

[0022] Based on the fourth convolution unit of the haze encoder, performing feature extraction on the second feature map to obtain a fourth feature map, and based on the fifth convolution unit of the haze encoder, performing feature extraction on the third feature map to obtain a fifth feature map;

[0023] The third characteristic map and the fifth characteristic map are added to obtain haze information.

[0024] According to an image defogging method provided by the present invention, the first multi-scale attention module based on the model to be trained and the haze information performs fogging conversion processing on the clear image training sample to obtain a first fogged image, including:

[0025] Based on the first content encoder of the first multi-scale attention module, encoding the clear image training sample to obtain a clear image content feature;

[0026] Inputting the clear image content features and the haze information into the first generator of the first multi-scale attention module, performing fogging conversion processing, and obtaining a first fogged image output by the first generator;

[0027] The second multi-scale attention module based on the model to be trained performs defogging conversion processing on the haze image training sample to obtain a first defogging image, including:

[0028] Based on the second content encoder of the second multi-scale attention module, encoding the haze image training sample to obtain haze image content features;

[0029] The haze image content features are input into the second generator of the second multi-scale attention module to perform dehazing conversion processing to obtain a first dehazed image output by the second generator.

[0030] According to an image defogging method provided by the present invention, the loss function of the model to be trained includes adversarial loss, content adversarial loss, cycle consistency loss, dark channel loss, identity mapping loss, reconstruction loss and semantic consistency loss.

[0031] According to an image defogging method provided by the present invention, the target haze image is input into a defogging model, the target haze image is defogged, and the target defogging image output by the defogging model is obtained, comprising:

[0032] Inputting the target haze image into a content encoder of a dehazing model, encoding the target haze image, and obtaining target content features output by the content encoder;

[0033] The target content features are input into the generator of the defogging model to perform defogging processing, and a target defogging image output by the generator is obtained.

[0034] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned image defogging methods are implemented.

[0035] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned image defogging methods are implemented.

[0036] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned image defogging methods are implemented.

[0037] The image defogging method, electronic device, storage medium and computer program product provided by the present invention obtain a target haze image to be defogged, and then input the target haze image into a defogging model to perform defogging on the target haze image, thereby obtaining a target defogging image output by the defogging model, thereby achieving image defogging, and further ensuring the normal operation of a vision-based outdoor system. At the same time, the model to be trained includes a multi-scale attention module for performing fog conversion processing and defogging conversion processing, and a discriminator for distinguishing between real images of a training image set and images generated by the multi-scale attention module, so that the defogging model can be obtained by unsupervised training based on a training image set consisting of unpaired clear images and haze images, thereby avoiding the limitation of paired image training sets on defogging model training, and the unpaired clear images and haze images can be real images, so that model training can be performed based on real data, thereby improving the defogging performance of the defogging model, and further improving the image defogging performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 One of the flow charts of the image defogging method provided by the present invention;

[0040] Figure 2 The second flowchart of the image defogging method provided by the present invention;

[0041] Figure 3 A schematic diagram of a haze encoder provided by the present invention;

[0042] Figure 4 A schematic diagram of the network architecture provided by the present invention;

[0043] Figure 5 The third flowchart of the image defogging method provided by the present invention;

[0044] Figure 6 An example physical structure diagram of an electronic device is provided. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Figure 1 One of the flow charts of the image defogging method provided by the present invention is as follows: Figure 1 As shown, the image defogging method provided by the present invention comprises:

[0047] Step 110, obtaining a target haze image to be defogged;

[0048] In this embodiment, the image dehazing method can be applied to various fields, for example, unmanned driving of automobiles, visual navigation of drones, security monitoring, etc. Correspondingly, the target haze image can be an image around the car, an image taken by a drone, a monitoring image, etc.

[0049] The target haze image includes haze information, that is, the environment of the corresponding acquisition device is affected by haze weather. The visibility of the target haze image is low, and the edge, color and detail information of the image have certain losses.

[0050] Specifically, a target haze image to be defogged is selected from target images acquired by an image acquisition device or a video acquisition device.

[0051] Step 120, inputting the target haze image into a defogging model, performing defogging processing on the target haze image, and obtaining a target defogging image output by the defogging model, wherein the defogging model is obtained by performing unsupervised training on a training image set consisting of unpaired clear images and haze images, and the training model includes a multi-scale attention module for performing haze conversion processing and defogging conversion processing, and a discriminator for distinguishing between real images of the training image set and generated images of the multi-scale attention module.

[0052] In this embodiment, the target defogging image is a clear image corresponding to the target haze image, that is, the target defogging image is an image reconstructed after defogging the target haze image. The visibility of the target defogging image is higher than that of the target haze image, and the loss of edge, color and detail information of the target defogging image is also compensated accordingly.

[0053] The defogging model is a deep learning model. Specifically, the defogging model is obtained based on an unsupervised training method, and the unsupervised training method can be a generative adversarial network. Based on this, the training image set can be an unpaired clear image and a haze image.

[0054] It should be noted that the unpaired clear image and haze image are two unrelated images, that is, the clear image does not need to be the clear image corresponding to the haze image, that is, the scenes in the haze image and the clear image are not required to be consistent.

[0055] In a specific embodiment, the dehazing model includes a feature extractor and a generator, wherein the feature extractor is used to extract feature information of a target haze image, and the generator is used to reconstruct the target haze image to obtain a corresponding clear image.

[0056] Specifically, the target haze image is input into the feature extractor of the dehazing model, feature extraction is performed on the target haze image, and feature information output by the feature extractor is obtained; the feature information is input into the generator of the dehazing model, dehazing is performed, and the target dehazed image output by the generator is obtained.

[0057] The feature extractor may be a content encoder, which is used to encode the target haze image to obtain corresponding content features. The content encoder and the generator are both obtained through training. The specific training process of the defogging model is referred to the second embodiment below, which will not be described here one by one.

[0058] According to the image defogging method of the embodiment of the present invention, a target haze image to be defogged is obtained, and then the target haze image is input into the defogging model to perform defogging on the target haze image, and the target defogging image output by the defogging model is obtained, thereby achieving image defogging, and further ensuring the normal operation of the vision-based outdoor system. At the same time, the model to be trained includes a multi-scale attention module for performing fog conversion processing and defogging conversion processing, and a discriminator for distinguishing between the real image of the training image set and the generated image of the multi-scale attention module, so that the defogging model can be obtained by unsupervised training based on the training image set composed of unpaired clear images and haze images, thereby avoiding the limitation of paired image training sets on defogging model training, and the unpaired clear images and haze images can be real images, so as to perform model training based on real data, thereby improving the defogging performance of the defogging model, and further improving the image defogging performance.

[0059] Furthermore, based on the above-mentioned first embodiment, a second embodiment of the image defogging method of the present invention is proposed. Figure 2 The second flowchart of the image defogging method provided by the present invention is as follows: Figure 2 As shown, in this embodiment, a training method for the defogging model is also included:

[0060] Step 210, obtaining unpaired clear image training samples and haze image training samples;

[0061] In this embodiment, the unpaired clear image training samples and haze image training samples are two unrelated images, that is, the clear image training samples do not need to be the clear images corresponding to the haze image training samples, that is, the scenes in the haze image training samples and the clear image training samples are not required to be consistent.

[0062] In one embodiment, the step 210 includes:

[0063] An outdoor clear image and a real haze image are obtained; the outdoor clear image is used as a clear image training sample, and the real haze image is used as a haze image training sample.

[0064] Specifically, unpaired training samples are constructed using outdoor clear images in the OTS (Outdoor Training Set) dataset and real haze images in the URHI (Unannotated Real Hazy Images) dataset as training image sets to train the constructed dehazing neural network (model to be trained). During a training process, the network randomly inputs unpaired training samples in the training image set.

[0065] It can be understood that using real haze images for training can enable the trained dehazing model to better dehaze real haze images, thereby further improving the performance of dehazing processing.

[0066] Step 220, based on the clear image training samples, the haze image training samples and the preset loss function, the constructed model to be trained is trained to obtain a defogging model, wherein the model to be trained includes a multi-scale attention module for performing haze conversion processing and defogging conversion processing, and a discriminator for distinguishing between real images and generated images, wherein the real image is the clear image training sample or the haze image training sample, and the generated image is the generated image output by the multi-scale attention module.

[0067] In this embodiment, the model to be trained is constructed by generating an adversarial network, which is constructed by an unsupervised method based on separation representation, that is, an unsupervised training method is used for model training.

[0068] The preset loss function includes an adversarial loss of the adversarial network, and the adversarial loss includes two adversarial loss functions, one for the adversarial loss corresponding to the haze image, and the other for the adversarial loss corresponding to the clear image. In addition, the preset loss function may also include at least one of the following: content adversarial loss, cycle consistency loss, dark channel loss, identity mapping loss, reconstruction loss, and semantic consistency loss.

[0069] Before the above step 220, the image defogging method further includes:

[0070] Based on a bilinear interpolation algorithm, the clear image training samples and the haze image training samples are scaled to obtain scaled clear image training samples and scaled haze image training samples.

[0071] The sizes of the scaled clear image training samples and the scaled haze image training samples may be 128*128.

[0072] Accordingly, the above step 220 includes:

[0073] Based on the scaled clear image training samples, the scaled haze image training samples and the preset loss function, the constructed model to be trained is trained to obtain a defogging model.

[0074] The model to be trained based on separation representation may include a haze encoder, two multi-scale attention modules and two discriminators. Each multi-scale attention module may include a content encoder and a generator. The haze encoder is used to encode the haze image training sample to obtain the haze information in the haze image training sample. The multi-scale attention module is used to generate the corresponding generated image. The discriminator is used to distinguish between the real image and the generated image, that is, to calculate the adversarial loss on the image to obtain the adversarial loss.

[0075] Specifically, the above step 220 includes:

[0076] Step 221, based on the constructed haze encoder of the to-be-trained model, feature extraction is performed on the haze image training sample to obtain haze information;

[0077] The haze encoder is used to extract haze information from haze image training samples. Specifically, based on the haze encoder, the haze image training samples are encoded to obtain haze information.

[0078] Furthermore, the above step 221 includes:

[0079] Step 2211, based on the first convolution unit of the haze encoder, performing a convolution operation on the haze image training sample to obtain a first feature map;

[0080] In this embodiment, the haze encoder is composed of multiple convolution units. The input haze image training sample is first processed by a convolution unit to obtain a first feature map, and then passes through different convolution units to obtain a second feature map and a third feature map with different resolutions. Similarly, the second feature map and the third feature map are processed by different convolution units to obtain a fourth feature map and a fifth feature map. Finally, the fourth feature map and the fifth feature map are added together as the output of the haze encoder.

[0081] The first convolution unit is used to perform a convolution operation on the haze image training sample to obtain a first feature map, that is, to perform feature extraction on the haze image training sample to obtain the first feature map.

[0082] Specifically, the first convolution unit includes a first convolution layer, and based on the first convolution layer of the haze encoder, performs a convolution operation on the haze image training sample to obtain a first convolution feature.

[0083] In one embodiment, the convolution kernel size of the first convolution layer is 3*3, and the convolution step size is 1.

[0084] Step 2212: Based on the second convolution unit of the haze encoder, perform feature extraction on the first feature map to obtain a second feature map, and based on the third convolution unit of the haze encoder, perform feature extraction on the first feature map to obtain a third feature map, wherein the second feature map and the third feature map have different resolutions;

[0085] Among them, the second convolution unit and the third convolution unit are used to obtain feature maps of different resolution sizes.

[0086] Specifically, the second convolution unit includes a second convolution layer, the third convolution unit includes a third convolution layer and a first pooling layer, based on the second convolution layer of the haze encoder, a convolution operation is performed on the first convolution feature to obtain a second convolution feature, and based on the first pooling layer of the haze encoder, a pooling operation is performed on the first convolution feature to obtain a first pooling feature, and based on the third convolution layer of the haze encoder, a convolution operation is performed on the first pooling feature to obtain a third convolution feature.

[0087] In one embodiment, the convolution kernel size of the second convolution layer is 3*3, and the convolution step size is 1. The pooling kernel size of the first pooling layer is 2*2, and the pooling step size is 2, which is used to perform an average pooling operation. The convolution kernel size of the third convolution layer is 3*3, and the convolution step size is 1.

[0088] Step 2213: Based on the fourth convolution unit of the haze encoder, perform feature extraction on the second feature map to obtain a fourth feature map, and based on the fifth convolution unit of the haze encoder, perform feature extraction on the third feature map to obtain a fifth feature map;

[0089] The fourth convolution unit is used to perform a convolution operation on the second feature map to obtain a fourth feature map, that is, to extract features from the second feature map to obtain the fourth feature map. The fifth convolution unit is used to perform a convolution operation on the third feature map to obtain a fifth feature map, that is, to extract features from the third feature map to obtain the fifth feature map.

[0090] Specifically, the fourth convolution unit includes a second pooling layer, a fourth convolution layer, a third pooling layer, a fifth convolution layer and an eighth convolution layer, and the fifth convolution unit includes a sixth convolution layer, a fourth pooling layer, a seventh convolution layer and a ninth convolution layer.

[0091] Based on the second pooling layer of the haze encoder, a pooling operation is performed on the second convolutional feature to obtain a second pooling feature; based on the fourth convolutional layer of the haze encoder, a convolution operation is performed on the second pooling feature to obtain a fourth convolutional feature; based on the third pooling layer of the haze encoder, a pooling operation is performed on the second convolutional feature to obtain a third pooling feature; based on the fifth convolutional layer of the haze encoder, a convolution operation is performed on the third pooling feature to obtain a fifth convolutional feature;

[0092] Based on the sixth convolutional layer of the haze encoder, performing a convolution operation on the third convolutional feature to obtain a sixth convolutional feature, and based on the fourth pooling layer of the haze encoder, performing a pooling operation on the third convolutional feature to obtain a fourth pooling feature, and based on the seventh convolutional layer of the haze encoder, performing a convolution operation on the fourth pooling feature to obtain a seventh convolutional feature;

[0093] Adding the fourth convolution feature and the sixth convolution feature to obtain a first added feature, and adding the fifth convolution feature and the seventh convolution feature to obtain a second added feature;

[0094] Based on the eighth convolutional layer of the haze encoder, a convolution operation is performed on the first added feature to obtain an eighth convolutional feature, and based on the ninth convolutional layer of the haze encoder, a convolution operation is performed on the second added feature to obtain a ninth convolutional feature.

[0095] In one embodiment, the pooling kernel size of the second pooling layer is 2*2, and the pooling step size is 2, which is used to perform an average pooling operation. The convolution kernel size of the fourth convolution layer is 3*3, and the convolution step size is 1. The pooling kernel size of the third pooling layer is 4*4, and the pooling step size is 4, which is used to perform an average pooling operation. The convolution kernel size of the fifth convolution layer is 3*3, and the convolution step size is 1. The convolution kernel size of the sixth convolution layer is 3*3, and the convolution step size is 1. The pooling kernel size of the fourth pooling layer is 2*2, and the pooling step size is 2, which is used to perform an average pooling operation. The convolution kernel size of the seventh convolution layer is 3*3, and the convolution step size is 1. The convolution kernel size of the eighth convolution layer is 3*3, and the convolution step size is 1. The convolution kernel size of the ninth convolution layer is 3*3, and the convolution step size is 1.

[0096] Step 2214, adding the third characteristic map and the fifth characteristic map to obtain haze information.

[0097] Specifically, the eighth convolution feature and the ninth convolution feature are added to obtain haze information.

[0098] For ease of understanding, refer to Figure 3 , Figure 3 Schematic diagram of the haze encoder provided by the present invention. First, the haze image I input to the module is convolved with a kernel size of 3*3 and a convolution step size of 1 to obtain the feature f in . The feature f in Perform a convolution operation with a convolution kernel size of 3*3 and a convolution step size of 1 to obtain the feature f h . The feature f in Perform an average pooling operation with a pooling kernel of 2*2 and a step size of 2, and then perform a convolution operation with a convolution kernel size of 3*3 and a convolution step size of 1 to obtain the feature f l . The feature f h Perform an average pooling operation with a pooling kernel of 2*2 and a step size of 2, and a convolution operation with a convolution kernel size of 3*3 and a convolution step size of 1 to obtain the feature f h2h . The feature f h Perform an average pooling operation with a pooling kernel of 4*4 and a step size of 4, and then perform a convolution operation with a convolution kernel size of 3*3 and a convolution step size of 1 to obtain the feature f h2l . The feature f l Perform a convolution operation with a convolution kernel size of 3*3 and a convolution step size of 1 to obtain the feature f l2h . The feature f l Perform an average pooling operation with a pooling kernel of 2*2 and a step size of 2, and then perform a convolution operation with a convolution kernel size of 3*3 and a convolution step size of 1 to obtain the feature f l2l . The feature f h2h and feature f l2hAdd them together to get the feature f′ h . The feature f h2l and feature f l2l Add them together to get the feature f′ l . The feature f′ h Perform a convolution operation with a convolution kernel size of 3*3 and a convolution step size of 1 to obtain the feature f′ h2 . The feature f′ l Perform a convolution operation with a convolution kernel size of 3*3 and a convolution step size of 1 to obtain the feature f′ l2h . The feature f′ h2h and feature f′ l2h Add them together to get the output feature f of the haze encoder out .

[0099] It can be understood that the haze encoder can capture more global information by performing multi-frequency representation on the extracted features, that is, by increasing the receptive field of the network, it can efficiently extract the haze information in the haze image training samples.

[0100] Step 222, based on the first multi-scale attention module of the model to be trained and the haze information, performing fogging conversion processing on the clear image training sample to obtain a first fogged image;

[0101] In this embodiment, the first multi-scale attention module integrates multi-scale feature representation and attention mechanism to facilitate feature representation.

[0102] Specifically, the above step 222 includes:

[0103] Step 2221: encoding the clear image training sample based on the first content encoder of the first multi-scale attention module to obtain clear image content features;

[0104] Specifically, based on the first content encoder, feature extraction is performed on the clear image training sample to obtain the clear image content feature. That is, the first content encoder is used to extract the content information of the clear image training sample, that is, the first content encoder extracts features from the input clear image training sample through a convolutional neural network to encode the content information of the image.

[0105] The first content encoder includes a convolution unit and a Res2Net (multi-scale backbone network), and the number of convolution units and Res2Net can be set according to actual needs. The convolution unit may include a convolution layer, an instance normalization layer, and an activation function layer, the instance normalization layer may be a BN (batch normalization layer) layer, and the activation function layer may be a ReLU activation function layer.

[0106] In one embodiment, the first content encoder includes three convolution units and four Res2Net modules, wherein the convolution unit includes a convolution layer, an instance normalization layer, and a ReLU activation function layer. The convolution kernel size of the convolution layer in the first convolution unit is 7*7, and the convolution step size is 1; the convolution kernel size of the convolution layer in the second convolution unit is 3*3, and the convolution step size is 2; the convolution kernel size of the convolution layer in the third convolution unit is 3*3, and the convolution step size is 2.

[0107] It can be understood that, through the design of the above content encoder, the multi-scale feature information of the input image can be efficiently extracted, that is, it has a small number of parameters and can efficiently process the real haze image to restore a high-quality clear image.

[0108] Step 2222, inputting the clear image content features and the haze information into the first generator of the first multi-scale attention module, performing fogging conversion processing, and obtaining a first fogged image output by the first generator;

[0109] In this embodiment, the first generator is used to generate a corresponding fogged image, that is, to generate a corresponding haze image.

[0110] The first generator includes a Res2Net module, a convolution unit and an attention module, and the number of convolution units, Res2Net and attention modules can be set according to actual needs. The convolution unit includes a convolution layer, an instance normalization layer and an activation function layer, the instance normalization layer can be a BN (batch normalization layer) layer, and the activation function layer can be a ReLU activation function layer. In addition, the convolution unit can also include an upsampling layer.

[0111] Among them, the attention module includes channel attention and spatial attention, which can improve the flexibility of the network in processing different types of features. The channel attention generates an attention weight for each channel of the input feature; the spatial attention generates an attention weight map for the input feature, and the same spatial position features of different channels share a weight.

[0112] In one embodiment, the first generator includes four Res2Net modules, three convolution units and an attention module. The first convolution unit includes a convolution layer, an upsampling layer, an instance normalization layer and a ReLU activation function layer, wherein the convolution kernel size of the convolution layer is 3*3 and the convolution step size is 1. The second convolution unit includes a convolution layer, an upsampling layer, an instance normalization layer and a ReLU activation function layer, wherein the convolution kernel size of the convolution layer is 3*3 and the convolution step size is 1. The third convolution unit includes a convolution layer, an instance normalization layer and a ReLU activation function layer, wherein the convolution kernel size of the convolution layer is 7*7 and the convolution step size is 1.

[0113] In some embodiments, the attention module includes a channel attention module and a spatial attention module. The input feature is passed through the channel attention module to obtain a channel attention weight, and then the channel attention weight is multiplied by the input feature to obtain the output of the channel attention module. Then the output of the channel attention module is used as the input of the spatial attention module to obtain the spatial attention weight, and then the spatial attention weight is multiplied by the input of the spatial attention module to obtain the output of the attention module.

[0114] In a specific embodiment, the channel attention in the attention module includes a global average pooling layer and two convolution units, the first convolution unit includes a convolution layer and a ReLU activation function layer, wherein the convolution kernel size of the convolution layer is 3*3, and the convolution step size is 1; the second convolution unit includes a convolution layer and a Sigmoid activation function layer, wherein the convolution kernel size of the convolution layer is 3*3, and the convolution step size is 1.

[0115] In a specific embodiment, the spatial attention in the attention module includes two convolution units, the first convolution unit includes a convolution layer and a ReLU activation function layer, wherein the convolution kernel size of the convolution layer is 3*3, and the convolution step size is 1; the second convolution unit includes a convolution layer and a Sigmoid activation function layer, wherein the convolution kernel size of the convolution layer is 3*3, and the convolution step size is 1.

[0116] Step 223, based on the first discriminator of the model to be trained and the first adversarial loss function in the preset loss function, perform adversarial loss calculation on the first fogged image to obtain a first adversarial loss corresponding to the haze image;

[0117] In this embodiment, the first discriminator is used to distinguish between real haze images and generated haze images, that is, to distinguish between haze image training samples and the first haze-added image.

[0118] Among them, the preset loss function includes the adversarial loss of the adversarial network, and the adversarial loss includes two adversarial loss functions, namely, the first adversarial loss function and the second adversarial loss function, one is the adversarial loss corresponding to the haze image, namely, the first adversarial loss, and the other is the adversarial loss corresponding to the clear image, namely, the second adversarial loss.

[0119] Specifically, the first adversarial loss function is:

[0120]

[0121] Among them, D I The discriminator representing the haze image (the first discriminator), G J Generator representing haze image (first generator), EJ and They represent the content encoder (first content encoder) of a clear image and the haze encoder of a haze image, respectively. represents mathematical expectation, ~ represents obedience relation, P represents data distribution probability, I is haze image training sample, J is clear image training sample, is the adversarial loss corresponding to the haze image (the first adversarial loss).

[0122] Step 224, based on the second multi-scale attention module of the model to be trained, performing dehazing conversion processing on the haze image training sample to obtain a first dehazed image;

[0123] In this embodiment, the second multi-scale attention module integrates multi-scale feature representation and attention mechanism to facilitate feature representation.

[0124] Specifically, the above step 224 includes:

[0125] Step 2241, encoding the haze image training sample based on the second content encoder of the second multi-scale attention module to obtain haze image content features;

[0126] Specifically, based on the second content encoder, feature extraction is performed on the haze image training sample to obtain the haze image content feature. That is, the second content encoder is used to extract the content information of the haze image training sample, that is, the second content encoder extracts features from the input haze image training sample through a convolutional neural network to encode the image content information.

[0127] The second content encoder includes a convolution unit and a Res2Net (multi-scale backbone network), and the number of convolution units and Res2Net can be set according to actual needs. The convolution unit may include a convolution layer, an instance normalization layer, and an activation function layer, the instance normalization layer may be a BN (batch normalization layer) layer, and the activation function layer may be a ReLU activation function layer.

[0128] In one embodiment, the second content encoder includes three convolution units and four Res2Net modules, wherein the convolution unit includes a convolution layer, an instance normalization layer, and a ReLU activation function layer. The convolution kernel size of the convolution layer in the first convolution unit is 7*7, and the convolution step size is 1; the convolution kernel size of the convolution layer in the second convolution unit is 3*3, and the convolution step size is 2; the convolution kernel size of the convolution layer in the third convolution unit is 3*3, and the convolution step size is 2.

[0129] It can be understood that, through the design of the above content encoder, the multi-scale feature information of the input image can be efficiently extracted, that is, it has a small number of parameters and can efficiently process the real haze image to restore the high-quality haze image.

[0130] Step 2242: input the content features of the haze image to the second generator of the second multi-scale attention module, perform dehazing conversion processing, and obtain a first dehazed image output by the second generator.

[0131] In this embodiment, the second generator is used to generate a corresponding dehazed image, that is, to generate a corresponding clear image.

[0132] The second generator includes a Res2Net module, a convolution unit and an attention module, and the number of convolution units, Res2Net and attention modules can be set according to actual needs. The convolution unit includes a convolution layer, an instance normalization layer and an activation function layer, the instance normalization layer can be a BN (batch normalization layer) layer, and the activation function layer can be a ReLU activation function layer. In addition, the convolution unit can also include an upsampling layer.

[0133] Among them, the attention module includes channel attention and spatial attention, which can improve the flexibility of the network in processing different types of features. The channel attention generates an attention weight for each channel of the input feature; the spatial attention generates an attention weight map for the input feature, and the same spatial position features of different channels share a weight.

[0134] In one embodiment, the second generator includes four Res2Net modules, three convolution units and an attention module. The first convolution unit includes a convolution layer, an upsampling layer, an instance normalization layer and a ReLU activation function layer, wherein the convolution kernel size of the convolution layer is 3*3 and the convolution step size is 1. The second convolution unit includes a convolution layer, an upsampling layer, an instance normalization layer and a ReLU activation function layer, wherein the convolution kernel size of the convolution layer is 3*3 and the convolution step size is 1. The third convolution unit includes a convolution layer, an instance normalization layer and a ReLU activation function layer, wherein the convolution kernel size of the convolution layer is 7*7 and the convolution step size is 1.

[0135] In some embodiments, the attention module includes a channel attention module and a spatial attention module. The input feature is passed through the channel attention module to obtain a channel attention weight, and then the channel attention weight is multiplied by the input feature to obtain the output of the channel attention module. Then the output of the channel attention module is used as the input of the spatial attention module to obtain the spatial attention weight, and then the spatial attention weight is multiplied by the input of the spatial attention module to obtain the output of the attention module.

[0136] In a specific embodiment, the channel attention in the attention module includes a global average pooling layer and two convolution units, the first convolution unit includes a convolution layer and a ReLU activation function layer, wherein the convolution kernel size of the convolution layer is 3*3, and the convolution step size is 1; the second convolution unit includes a convolution layer and a Sigmoid activation function layer, wherein the convolution kernel size of the convolution layer is 3*3, and the convolution step size is 1.

[0137] In a specific embodiment, the spatial attention in the attention module includes two convolution units, the first convolution unit includes a convolution layer and a ReLU activation function layer, wherein the convolution kernel size of the convolution layer is 3*3, and the convolution step size is 1; the second convolution unit includes a convolution layer and a Sigmoid activation function layer, wherein the convolution kernel size of the convolution layer is 3*3, and the convolution step size is 1.

[0138] Step 225, based on the second discriminator of the model to be trained and the second adversarial loss function in the preset loss function, perform adversarial loss calculation on the first dehazed image to obtain a second adversarial loss corresponding to a clear image;

[0139] In this embodiment, the second discriminator is used to distinguish between a real clear image and a generated clear image, that is, to distinguish between a clear image training sample and a first defogging image.

[0140] Specifically, the second adversarial loss function is:

[0141]

[0142] Among them, D J The discriminator representing the clear image (the second discriminator), G I The generator representing the clear image (the second generator), E I represents the content encoder of the haze image (the second content encoder), E represents the mathematical expectation, ~ represents the obedience relationship, P represents the data distribution probability, I represents the haze image training sample, J represents the clear image training sample, is the adversarial loss corresponding to the clear image (the second adversarial loss).

[0143] Step 226: Based on the first adversarial loss and the second adversarial loss, perform model training on the model to be trained to obtain a defogging model.

[0144] Among them, the total network loss = the first adversarial loss + the second adversarial loss. In addition, the total network loss can also include at least one of the following: content adversarial loss, cycle consistency loss, dark channel loss, identity mapping loss, reconstruction loss and semantic consistency loss.

[0145] In this embodiment, the overall network structure of the model to be trained includes forward conversion. In the forward conversion, the haze image training samples are defogged and converted into clear images, and the clear image training samples are fogged and converted into haze images.

[0146] Furthermore, based on the above forward conversion, the overall network structure of the model to be trained also includes backward reconstruction. After completing the forward conversion, backward reconstruction is implemented, and accordingly, the first fogged image and the first defogged image are converted back to the original clear image training samples and haze image training samples. Specifically, the first fogged image and the first defogged image are encoded using the corresponding second content encoder and the first content encoder to obtain content features. The process is similar to the forward conversion and will not be repeated here. Finally, the first defogged image is fogged to obtain a haze image (second fogged image), and the first fogged image is defogged to obtain a clear image (second defogged image).

[0147] For ease of understanding, refer to Figure 4 , Figure 4 A schematic diagram of the network architecture provided by the present invention. First, a set of unpaired haze images I and clear images J are obtained. In the forward conversion, the haze image I is dehazed and converted into the clear image J. fake , add fog to the clear image J and convert it into the haze image I fake Specifically, the corresponding content encoder E is used respectively. I and E J Encode the haze image I and the clear image J to obtain the content feature Z I and Z J , use the haze encoder to extract the haze information Z in the haze image I H ; Z J and Z H After concatenation according to the channel dimension, it is input to the generator G J In the example above, we get the haze image I after adding fog fake ; Z I Input to the generator G I In the figure, we get a clear image after defogging J fake After completing the forward transformation, the backward reconstruction is performed. Similarly, the obtained haze image I fake and clear image J fake , converted back to the original clear image and haze image. Specifically, the corresponding content encoder E is used respectively I and E J to I fake and J fake Encoding is performed to obtain content features. The process is similar to forward conversion, and finally the haze image I is obtained after fogging. res Clear image after dehazingJ res .

[0148] Furthermore, the loss function of the model to be trained includes adversarial loss, content adversarial loss, cycle consistency loss, dark channel loss, identity mapping loss, reconstruction loss and semantic consistency loss.

[0149] Among them, the adversarial loss includes two adversarial loss functions, namely the first adversarial loss function and the second adversarial loss function. One is the adversarial loss corresponding to the haze image, namely the first adversarial loss, and the other is the adversarial loss corresponding to the clear image, namely the second adversarial loss.

[0150] Specifically, the first adversarial loss function is:

[0151]

[0152] Among them, D I The discriminator representing the haze image (the first discriminator), G J Generator representing haze image (first generator), E J and denote the content encoder of the clear image (first content encoder) and the haze encoder of the haze image, respectively. E denotes mathematical expectation, ~ denotes a compliance relationship, P denotes data distribution probability, I denotes haze image training samples, J denotes clear image training samples, is the adversarial loss corresponding to the haze image (the first adversarial loss).

[0153] Specifically, the second adversarial loss function is:

[0154]

[0155] Among them, D J The discriminator representing the clear image (the second discriminator), G I The generator representing the clear image (the second generator), E I represents the content encoder of the haze image (the second content encoder), E represents the mathematical expectation, ~ represents the obedience relationship, P represents the data distribution probability, I represents the haze image training sample, J represents the clear image training sample, is the adversarial loss corresponding to the clear image (the second adversarial loss).

[0156] Among them, the content adversarial loss is used to promote the content features (second content features) of haze image training samples and the content features (first content features) of clear image training samples to follow similar distributions.

[0157] Specifically, the content adversarial loss function is:

[0158]

[0159] Among them, D c The discriminator representing the content adversarial loss, Z I Represents the content features of the haze image training samples, Z J represents the content features of the clear image training samples, E represents the mathematical expectation, I represents the haze image training samples, J represents the clear image training samples, Fight loss for content.

[0160] Among them, the cycle consistency loss is used to calculate the loss between the original input image and the image generated by backward reconstruction, so that the reconstructed image can contain the information of the original image as much as possible.

[0161] Specifically, the cycle consistency loss function is:

[0162]

[0163] Among them, E represents the mathematical expectation, I is the haze image training sample, J is the clear image training sample, and I res is the haze image obtained by backward reconstruction (the second haze image), J res is the clear image obtained by backward reconstruction (the second dehazed image), L cc is the cycle consistency loss.

[0164] Among them, the dark channel loss is used to constrain the network using the dark channel prior, which can make the value of the dark channel close to zero, thereby improving the dehazing effect.

[0165] Specifically, the dark channel loss function is:

[0166] L dcp =‖DC(J fake )‖1,

[0167] Among them, DC represents the dark channel of the image, J fake is the forward transformed clear image (first dehazed image), L dcp is the dark channel loss.

[0168] Among them, the identity mapping loss is used to preserve features, improve the quality of the converted images, and stabilize the training process.

[0169] Specifically, the identity mapping loss function is:

[0170] L idt =‖G I (E I (J))-J‖1,

[0171] Among them, G I The generator representing the clear image (the second generator), E Irepresents the content encoder of the haze image (the second content encoder), J is the clear image training sample, and L idt is the identity mapping loss.

[0172] Among them, reconstruction loss is used to facilitate network training.

[0173] Specifically, the reconstruction loss function is:

[0174]

[0175] Among them, G J Generator representing haze image (first generator), G I The generator representing the clear image (the second generator), E J and They represent the content encoder (first content encoder) of the clear image and the haze encoder of the haze image, E I represents the content encoder of the haze image (the second content encoder), I is the haze image training sample, J is the clear image training sample, and L SR is the reconstruction loss for backward reconstruction.

[0176] Among them, the semantic consistency loss is used to make the semantics of the haze image training sample and the first dehazed image consistent.

[0177] Specifically, the semantic consistency loss is:

[0178] L SC =‖φ l (I)-φ l (J fake )‖2,

[0179] Among them, φ l represents the extraction of the l-th layer features of the pre-trained VGG-19 model, I is the haze image training sample, and J fake is the first dehazed image, L SC is the semantic consistency loss.

[0180] In this embodiment, adversarial loss, content adversarial loss, cycle consistency loss, dark channel loss, identity mapping loss, reconstruction loss and semantic consistency loss are aggregated to obtain the total network loss.

[0181] The aggregation processing may be weighted aggregation processing, or other aggregation processing methods.

[0182] In one embodiment, the total network loss is:

[0183]

[0184] in, is the adversarial loss corresponding to the haze image, is the adversarial loss corresponding to the clear image (the second adversarial loss), To combat the loss of content, L cc is the cycle consistency loss, L dcp is the dark channel loss, L idt is the identity mapping loss, L SR is the reconstruction loss for backward reconstruction, L SC is the semantic consistency loss. λ cc =λ rec =λ idt =10,λ dcp =0.5.

[0185] In this embodiment, the defogging model is obtained by unsupervised training based on a training image set consisting of unpaired clear images and haze images, thereby avoiding the limitation of paired image training sets on the training of the defogging model, and the unpaired clear images and haze images can be real images, so as to perform model training based on real data, thereby improving the defogging performance of the defogging model, and further improving the image defogging performance. In addition, the multi-scale attention module of the embodiment of the present invention has a small number of parameters, and can efficiently process real haze images and restore high-quality clear images, thereby further improving the image defogging performance. The haze encoder of the embodiment of the present invention efficiently extracts haze information of haze images by increasing the receptive field of the network, thereby further improving the image defogging performance.

[0186] Furthermore, based on the above first embodiment, a third embodiment of the image defogging method of the present invention is proposed. Figure 5 The third flowchart of the image defogging method provided by the present invention is as follows: Figure 5 As shown, in this embodiment, the above step 120 includes:

[0187] Step 121, inputting the target haze image into a content encoder of a dehazing model, encoding the target haze image, and obtaining target content features output by the content encoder;

[0188] Specifically, the content encoder based on the defogging model extracts features of the target haze image to obtain target content features. That is, the content encoder of the defogging model is used to extract content information of the target haze image, that is, the content encoder of the defogging model extracts features of the input target haze image through a convolutional neural network to encode the content information of the image.

[0189] Among them, the content encoder of the dehazing model is the trained second content encoder, and its specific structure can refer to the above description of the second content encoder, which will not be repeated here.

[0190] Step 122: input the target content features into the generator of the defogging model to perform defogging processing to obtain a target defogging image output by the generator.

[0191] In this embodiment, the generator of the defogging model is used to generate a defogging image corresponding to the target haze image, that is, to generate a corresponding clear image.

[0192] Among them, the generator of the dehazing model is the trained second generator, and its specific structure can refer to the above description of the second generator, which will not be repeated here.

[0193] In this embodiment, by obtaining a target haze image to be defogged, the target haze image is then input into a content encoder and generator of a defogging model to perform defogging on the target haze image, and a target defogging image output by the defogging model is obtained, thereby achieving image defogging, and further ensuring the normal operation of a vision-based outdoor system. At the same time, the content encoder and generator of the defogging model are obtained by unsupervised training based on a training image set consisting of unpaired clear images and haze images, thereby avoiding the limitation of paired image training sets on defogging model training, and the unpaired clear images and haze images can be real images, so as to perform model training based on real data, thereby improving the defogging performance of the defogging model, and further improving the image defogging performance.

[0194] The image defogging device provided by the present invention is described below. The image defogging device described below and the image defogging method described above can be referred to each other.

[0195] An acquisition module is used to acquire the target haze image to be defogged;

[0196] A defogging module is used to input the target haze image into a defogging model, perform defogging on the target haze image, and obtain a target defogging image output by the defogging model. The defogging model is obtained by performing unsupervised training on a training image set consisting of unpaired clear images and haze images. The model to be trained includes a multi-scale attention module for performing fogging conversion processing and defogging conversion processing, and a discriminator for distinguishing between real images of the training image set and generated images of the multi-scale attention module.

[0197] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the image defogging method, which includes: obtaining a target haze image to be defogged; inputting the target haze image into a defogging model, defogging the target haze image, and obtaining a target defogged image output by the defogging model, wherein the defogging model is a training image set composed of unpaired clear images and haze images, and is obtained by unsupervised training of the training model, wherein the training model includes a multi-scale attention module for performing fogging conversion processing and defogging conversion processing, and a discriminator for distinguishing the real image of the training image set from the generated image of the multi-scale attention module.

[0198] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0199] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image defogging method provided by the above-mentioned methods, and the method includes: obtaining a target haze image to be defogged; inputting the target haze image into a defogging model, performing defogging processing on the target haze image, and obtaining a target defogged image output by the defogging model, wherein the defogging model is based on a training image set consisting of unpaired clear images and haze images, and is obtained by performing unsupervised training on a model to be trained, and the model to be trained includes a multi-scale attention module for performing fogging conversion processing and defogging conversion processing, and a discriminator for distinguishing between real images of the training image set and images generated by the multi-scale attention module.

[0200] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the image defogging method provided by the above-mentioned methods, the method comprising: obtaining a target haze image to be defogged; inputting the target haze image into a defogging model, performing defogging processing on the target haze image, and obtaining a target defogged image output by the defogging model, wherein the defogging model is obtained by performing unsupervised training on a training image set consisting of unpaired clear images and haze images, and the model to be trained comprises a multi-scale attention module for performing fogging conversion processing and defogging conversion processing, and a discriminator for distinguishing between real images of the training image set and images generated by the multi-scale attention module.

[0201] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0202] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0203] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image defogging method, characterized in that: include: Obtain a target haze image to be defogged; Inputting the target haze image into a defogging model, performing defogging processing on the target haze image, and obtaining a target defogging image output by the defogging model, wherein the defogging model is obtained by performing unsupervised training on a training image set consisting of unpaired clear images and haze images, and the training model includes a multi-scale attention module for performing haze conversion processing and defogging conversion processing, and a discriminator for distinguishing between a real image of the training image set and an image generated by the multi-scale attention module; The defogging model is trained by the following method: Obtain unpaired clear image training samples and haze image training samples; Based on the clear image training samples, the haze image training samples and the preset loss function, the constructed model to be trained is trained to obtain a defogging model, wherein the model to be trained includes a multi-scale attention module for performing a fogging conversion process and a defogging conversion process, and a discriminator for distinguishing a real image from a generated image, wherein the real image is the clear image training sample or the haze image training sample, and the generated image is a generated image output by the multi-scale attention module; The method of performing model training on the constructed model to be trained based on the clear image training sample, the haze image training sample and a preset loss function to obtain a defogging model includes: Based on the constructed haze encoder of the to-be-trained model, feature extraction is performed on the haze image training sample to obtain haze information; Based on the first multi-scale attention module of the model to be trained and the haze information, performing fogging conversion processing on the clear image training sample to obtain a first fogged image; Based on the first discriminator of the model to be trained and a first adversarial loss function in a preset loss function, performing adversarial loss calculation on the first fogged image to obtain a first adversarial loss corresponding to the haze image; Based on the second multi-scale attention module of the model to be trained, performing dehazing conversion processing on the haze image training sample to obtain a first dehazed image; Based on the second discriminator of the model to be trained and the second adversarial loss function in the preset loss function, performing adversarial loss calculation on the first dehazed image to obtain a second adversarial loss corresponding to a clear image; Based on the first adversarial loss and the second adversarial loss, model training is performed on the model to be trained to obtain a defogging model.

2. The image defogging method according to claim 1, characterized in that: The haze encoder based on the constructed model to be trained performs feature extraction on the haze image training sample to obtain haze information, including: Based on the first convolution unit of the haze encoder, performing a convolution operation on the haze image training sample to obtain a first feature map; Based on the second convolution unit of the haze encoder, feature extraction is performed on the first feature map to obtain a second feature map, and based on the third convolution unit of the haze encoder, feature extraction is performed on the first feature map to obtain a third feature map, and the second feature map and the third feature map have different resolutions; Based on the fourth convolution unit of the haze encoder, performing feature extraction on the second feature map to obtain a fourth feature map, and based on the fifth convolution unit of the haze encoder, performing feature extraction on the third feature map to obtain a fifth feature map; The third characteristic map and the fifth characteristic map are added to obtain haze information.

3. The image defogging method according to claim 1, characterized in that: The first multi-scale attention module based on the model to be trained and the haze information performs fogging conversion processing on the clear image training sample to obtain a first fogged image, including: Based on the first content encoder of the first multi-scale attention module, encoding the clear image training sample to obtain a clear image content feature; Inputting the clear image content features and the haze information into the first generator of the first multi-scale attention module, performing fogging conversion processing, and obtaining a first fogged image output by the first generator; The second multi-scale attention module based on the model to be trained performs defogging conversion processing on the haze image training sample to obtain a first defogging image, including: Based on the second content encoder of the second multi-scale attention module, encoding the haze image training sample to obtain haze image content features; The haze image content features are input into the second generator of the second multi-scale attention module to perform dehazing conversion processing to obtain a first dehazed image output by the second generator.

4. The image defogging method according to claim 1, characterized in that: The loss function of the model to be trained includes adversarial loss, content adversarial loss, cycle consistency loss, dark channel loss, identity mapping loss, reconstruction loss and semantic consistency loss.

5. The image defogging method according to any one of claims 1 to 4, characterized in that: The step of inputting the target haze image into a defogging model, performing defogging processing on the target haze image, and obtaining a target defogging image output by the defogging model includes: Inputting the target haze image into a content encoder of a dehazing model, encoding the target haze image, and obtaining target content features output by the content encoder; The target content features are input into the generator of the defogging model to perform defogging processing, and a target defogging image output by the generator is obtained.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the image defogging method according to any one of claims 1 to 5 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image defogging method according to any one of claims 1 to 5 are implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the image defogging method according to any one of claims 1 to 5 are implemented.

Citation Information

Patent Citations

  • Single image defogging method based on multi-scale self-attention generative adversarial network

    CN112184577A

  • Traffic image defogging method based on improved generative adversarial network

    CN112801902A