Image defogging method and device, electronic equipment and storage medium

Abstract

The application provides an image defogging method and device, electronic equipment and storage medium, wherein the method comprises: determining an image to be defogged; inputting the image to be defogged into a defogging model to obtain a defogged image output by the defogging model; the defogging model comprises a haze image encoder and a semantic decoder; the haze image encoder is used for extracting and fusing semantic features of the image to be defogged at two different levels to obtain haze image semantic features; the semantic decoder is used for obtaining a defogged image based on the haze image semantic features; the defogging model is trained based on unpaired haze sample images and clear sample images, realizes extraction of semantic features of the image to be defogged at different levels, improves feature expression capability of the output haze image semantic features, thereby improving haze feature extraction capability of the defogging model, and further improving defogging quality of the defogging model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image defogging method and device, electronic equipment and storage medium. BACKGROUND

[0002] Haze is a typical atmospheric phenomenon, which is caused when fine particles such as dust accumulate in the air. These particles have a strong absorption and scattering effect on light, thereby weakening the scene radiation reflected by the scene, and causing it to be confused with scattered light. Images or videos taken under such conditions will have serious quality degradation, affecting subsequent intelligent analysis and restricting system performance.

[0003] With the development of semi-supervised and unsupervised transfer learning in deep learning, there are currently application development domain adaptation paradigms or the use of CycleGAN's cycle structure to improve the generalization ability of deep models. In the feature domain, the semantic components and haze components of the haze image are decomposed / reconstructed, the transfer between domains is realized, and the defogging image is obtained.

[0004] However, when decoupling the haze image features, a homogeneous single-branch network structure is often used, which lacks differentiated processing of different characteristic information, and with the deepening of the network structure, information will inevitably be lost in the learning process. SUMMARY

[0005] The present application provides an image defogging method, device, electronic equipment and storage medium to solve the defects of the prior art that using a homogeneous single-branch network structure to decouple the haze image results in a lack of feature information differentiation and information loss.

[0006] The present application provides an image defogging method, comprising:

[0007] determining a to-be-defogged image;

[0008] inputting the to-be-defogged image into a defogging model to obtain a defogged image output by the defogging model;

[0009] The defogging model comprises a haze image encoder and a semantic decoder; the haze image encoder is configured to extract and fuse semantic features of the to-be-defogged image at two different levels to obtain haze image semantic features; the semantic decoder is configured to obtain a defogged image based on the haze image semantic features; and the defogging model is trained based on unpaired haze sample images and clear sample images.

[0010] According to the image defogging method provided by the present application, the haze image encoder comprises a shallow network branch, a deep network branch and a weighted fusion network.

[0011] The acquisition step of the haze image semantic feature is as follows:

[0012] The semantic features of the to-be-dehazed image extracted by the shallow network branch and the semantic features of the to-be-dehazed image extracted by the deep network branch are fused to obtain a fusion feature;

[0013] Based on the weighted fusion network, the channel-level and pixel-level weighted feature fusion is performed on the fusion feature to obtain the haze image semantic feature.

[0014] According to the image dehazing method provided by the application, the shallow network branch includes a shallow first convolution unit, a shallow second convolution unit, a shallow third convolution unit and a shallow residual unit; the deep network branch includes a deep first convolution unit, a deep second convolution unit, a deep third convolution unit, a deep first residual unit, a deep second residual unit and a deep third residual unit;

[0015] The fusion of the semantic features of the to-be-dehazed image extracted by the shallow network branch and the semantic features of the to-be-dehazed image extracted by the deep network branch to obtain a fusion feature includes:

[0016] Based on the to-be-dehazed image, the first shallow feature is determined by applying the shallow first convolution unit, and the first deep feature is determined by applying the deep first convolution unit and the deep first residual unit;

[0017] Based on the first shallow feature and the first deep feature, the first fusion feature is determined;

[0018] Based on the first fusion feature, the second shallow feature is determined by applying the shallow second convolution unit, and the second deep feature is determined by applying the deep second convolution unit and the deep second residual unit;

[0019] Based on the second shallow feature and the second deep feature, the second fusion feature is determined;

[0020] Based on the second fusion feature, the third shallow feature is determined by applying the shallow third convolution unit, and the third deep feature is determined by applying the deep third convolution unit;

[0021] Based on the third shallow feature and the third deep feature, the third fusion feature is determined;

[0022] Based on the third fusion feature, the fourth shallow feature is determined by applying the shallow residual unit, and the fourth deep feature is determined by applying the deep third residual unit;

[0023] Based on the fourth shallow feature and the fourth deep feature, the fusion feature is determined.

[0024] According to the image defogging method provided by the application, the weighted fusion network comprises a first weighted fusion unit, a second weighted fusion unit, a third weighted fusion unit, a first channel-level weighted unit, a second channel-level weighted unit, a third channel-level weighted unit and a pixel-level weighted unit; the first weighted fusion unit, the second weighted fusion unit and the third weighted fusion unit each comprise a channel weighting subunit and a pixel weighting subunit;

[0025] The weighted feature fusion of the fusion features at the channel level and the pixel level is performed based on the weighted fusion network, so as to obtain the haze image semantic features, which comprises:

[0026] Based on the fusion features, the first weighted fusion unit and the first channel-level weighted unit are applied to determine the first channel weighted feature;

[0027] Based on the first channel weighted feature, the second weighted fusion unit is applied to determine the first weighted fusion feature;

[0028] Based on the first weighted fusion feature, the second channel-level weighted unit is applied to determine the second channel-level weighted feature;

[0029] Based on the second channel weighted feature, the third weighted fusion unit is applied to determine the second weighted fusion feature;

[0030] Based on the second weighted fusion feature, the third channel-level weighted unit is applied to determine the third channel-level weighted feature;

[0031] Based on the first weighted fusion feature, the second weighted fusion feature and the third channel weighted feature, the fourth channel-level weighted feature is determined;

[0032] Based on the fourth channel-level weighted feature, the pixel-level weighted unit is applied to determine the haze image semantic features.

[0033] According to the image defogging method provided by the application, the training step of the defogging model is as follows:

[0034] The unpaired haze sample image and clear sample image and the initial model are determined, and the initial haze image encoder, the initial semantic decoder, the initial clear image encoder, the initial reconstruction decoder and the style encoder are determined;

[0035] Based on the initial clear image encoder, style encoder, and initial reconstruction decoder, the haze sample image and the clear sample image are applied to generate a synthetic haze image corresponding to the clear sample image. Simultaneously, based on the initial haze image encoder and the initial semantic decoder, the haze sample image is applied to generate a synthetic clear image corresponding to the clear sample image. Also, based on the initial clear image encoder and the initial semantic decoder, the haze sample image is applied to generate a synthetic clear image corresponding to the haze sample image. Finally, based on the initial clear image encoder, style encoder, and initial reconstruction decoder, the synthetic haze image corresponding to the clear sample image and the synthetic clear image corresponding to the haze sample image are applied to generate a synthetic haze image corresponding to the haze sample image.

[0036] Based on the haze sample image, the corresponding synthesized clear image, the corresponding synthesized haze image, the clear sample image, the corresponding synthesized haze image, and the corresponding synthesized clear image, a loss is determined, and the parameters of the initial model are iterated based on the loss until the loss converges. The initial haze image encoder obtained after training is used as the haze image encoder, and the initial semantic decoder is used as the semantic decoder.

[0037] According to the image dehazing method provided by the present invention, the loss is obtained by weighted summation of cycle consistency loss, semantic consistency loss, target domain adversarial loss, feature representation adversarial loss, consistency mapping loss, self-reconstruction loss, dark channel loss and style feature contrast loss; the style feature contrast loss is calculated based on a style feature contrast learning module; the style feature contrast learning module includes a style encoder and the 9th and 13th layers of a pre-trained VGG19 model.

[0038] According to an image dehazing method provided by the present invention, the formula for the loss function corresponding to the style feature contrast loss is as follows:

[0039]

[0040]

[0041] In the formula, This indicates the style loss in synthesized sharp images; Indicates the style loss of the synthesized haze image; G i This indicates the extraction of hidden features from the i-th layer of the VGG19 model, w i β is the weight coefficient of the i-th layer. id(x,y) represents the preset adjustment coefficient for the i-th layer, where i is 9 or 13; d(x,y) is the mean absolute error between x and y. This refers to the style encoder; J clear Indicates the clear sample image; I haze This represents the haze sample image; J dehaze This represents the synthesized clear image corresponding to the haze sample image; I fahaze This refers to the synthetic haze image corresponding to the clear sample image.

[0042] The present invention also provides an image dehazing apparatus, comprising:

[0043] The determination module is used to determine the image to be dehazed;

[0044] The dehazing module is used to input the image to be dehazed into the dehazing model and obtain the dehazed image output by the dehazing model;

[0045] The dehazing model includes a haze image encoder and a semantic decoder; the haze image encoder is used to extract and fuse the semantic features of the image to be dehazed at two different levels to obtain the haze image semantic features; the semantic decoder is used to obtain the dehazed image based on the haze image semantic features; the dehazing model is trained based on unpaired haze sample images and clear sample images.

[0046] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image dehazing method as described above.

[0047] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image dehazing method as described above.

[0048] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image dehazing method as described above.

[0049] The image dehazing method, apparatus, electronic device, and storage medium provided by this invention extract semantic features of the dehazed image by setting two different-level networks in the haze image encoder, and then fusing the obtained semantic features of the two different levels. This realizes the extraction of semantic features of different levels of the image to be dehazed, improves the feature expression ability of the output haze image semantic features, thereby improving the haze feature extraction ability of the dehazing model, and further improving the dehazing quality of the dehazing model. Attached Figure Description

[0050] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0051] Figure 1 is a flowchart of the image defogging method provided by the present application;

[0052] Figure 2 is a flowchart of the haze image semantic feature acquisition method provided by the present application;

[0053] Figure 3 is a data flowchart of the haze image encoder provided by the present application;

[0054] Figure 4 is a flowchart of the defogging model training method provided by the present application;

[0055] Figure 5 is a network structure diagram of the decoder provided by the present application;

[0056] Figure 6 is a network structure diagram of the style encoder provided by the present application;

[0057] Figure 7 is a data flowchart of the style feature contrast learning module provided by the present application;

[0058] Figure 8 is a network architecture diagram of the initial model provided by the present application;

[0059] Figure 9 is a structural schematic diagram of the image defogging device provided by the present application;

[0060] Figure 10 is a structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0061] In order to make the objects, technical solutions and advantages of the present application clearer, the following will combine the drawings in the present application to clearly and completely describe the technical solutions in the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0062] With the development of deep learning and convolutional neural networks (CNNs), many supervised learning-based methods have been applied to the field of image dehazing. For example, DehazeNet (An end-to-end system for single image haze removal), AOD-Net (An All-in-One Network for Dehazing and Beyond), etc. Compared with traditional methods, these methods directly map the imaging parameters of the haze image or directly solve the clear image through an end-to-end network. With the proposal of the synthetic paired dataset, these methods achieve superior performance on the corresponding dataset. However, in the real world, it is almost impossible to obtain paired clear / haze image sample pairs, and the model trained based on synthetic data often lacks generalization ability when processing real haze images. Some recent studies attempt to explore dehazing strategies that do not require paired samples. The first method is to decouple the haze image in the image domain through the atmospheric scattering model. For example, YOLY (You Only Look Yourself), ZID (Zero-shot Image Dehazing), etc. Such methods usually use multiple subnets to separate the input haze image into three component layers: atmospheric light A, transmission map t(x) and clear image J. Then, the atmospheric scattering model is used to reconstruct the haze image and establish constraints with the input haze image, thereby recovering the clear image. Generally, decoupling in the image domain with the aid of the atmospheric scattering model will introduce errors and lead to inaccurate estimation, because the atmospheric scattering model is only a rough approximation of the real world and contains many assumptions itself. Another method is semi-supervised or unsupervised transfer learning. They develop domain adaptation paradigms or use the cycle structure of CycleGAN to improve the generalization ability of deep models. In the feature domain, the semantic components and haze components of the haze image are decomposed / reconstructed, and the transfer between domains is realized. However, the above methods often use isomorphic single-branch network structures when decoupling the features of the haze image, which lacks distinction in processing different characteristic information, and with the deepening of the network structure, information will inevitably be lost in the learning process.

[0063] Therefore, how to improve the dehazing quality of the unsupervised dehazing model is a technical problem that those skilled in the art are eager to solve.

[0064] To solve the above technical problems, an embodiment of the present application provides an image dehazing method. Figure 1 is a flowchart of the image dehazing method provided by the present application. As shown in Figure 1 , the method comprises:

[0065] Step 110, determining a to-be-dehazed image;

[0066] It should be noted that the defogged image can be any image with fog blur, and the embodiment of the present application does not limit the image format and image size of the image to be defogged.

[0067] In step 120, the image to be defogged is input into the defogging model to obtain a defogged image output by the defogging model.

[0068] The defogging model comprises a haze image encoder and a semantic decoder. The haze image encoder is configured to extract and fuse semantic features of the image to be defogged at two different levels to obtain haze image semantic features. The semantic decoder is configured to obtain a defogged image based on the haze image semantic features. The defogging model is trained based on unpaired haze sample images and clear sample images.

[0069] It is considered that a convolutional network of different depths can extract feature representations of input images at different levels, and then fuse the feature representations at different levels, which can improve the feature expression capability. Therefore, in the embodiment of the present application, the semantic features of the image to be defogged are extracted at two different levels by the image encoder, so as to improve the feature extraction capability and further improve the defogging quality of the defogging model.

[0070] Specifically, a cycle generative adversarial model comprising a haze image encoder and a semantic decoder is constructed, and the cycle generative adversarial model is trained using unpaired haze sample images and clear sample images. After the training is completed, the parameters of the haze image encoder and the parameters of the semantic decoder are fixed, and the defogging model is composed of the haze image encoder and the semantic decoder. Then, the image to be defogged is input into the defogging model to obtain a defogged image output by the defogging model. The haze image encoder is configured to extract and fuse semantic features of the image to be defogged at two different levels to obtain haze image semantic features. The semantic decoder is configured to decode the haze image semantic features to obtain a defogged image.

[0071] It should be noted that the haze image encoder extracts semantic features of the image to be defogged at two different levels to obtain semantic features at two different levels, and then fuses the semantic features at two different levels. The fusion can be feature fusion of corresponding node layers in the two different level networks, or fusion of the features finally extracted by the two different level networks, and the embodiment of the present application does not limit this. In addition, after the fusion of the semantic features is completed, weighted fusion can be performed on each channel and each pixel, and the embodiment of the present application does not limit this. The unpaired haze sample images and clear sample images mean that the haze sample images are randomly selected from a haze sample set, the clear sample images are randomly selected from a clear sample set, and there is no mapping relationship between the haze sample images and the clear sample images.

[0072] The image defogging method provided by the embodiment of the present application extracts semantic features of the image to be defogged by setting two different levels of networks in the haze image encoder, and fuses the obtained semantic features of the two different levels, thereby realizing semantic feature extraction of different levels of the image to be defogged, improving the feature expression capability of the semantic features of the output haze image, and thereby improving the haze feature extraction capability of the defogging model, and further improving the defogging quality of the defogging model.

[0073] Based on the above embodiment, the haze image encoder comprises a shallow network branch, a deep network branch and a weighted fusion network. Figure 2 is a flowchart of the haze image semantic feature acquisition method provided by the present application. As shown in the figure, the haze image semantic feature acquisition steps are as follows: Figure 2

[0074] Step 210, fuse the semantic features of the image to be defogged extracted by the shallow network branch and the semantic features of the image to be defogged extracted by the deep network branch to obtain fusion features.

[0075] Step 220, based on the weighted fusion network, perform channel-level and pixel-level weighted feature fusion on the fusion features to obtain the haze image semantic features.

[0076] It is considered that the fusion features after fusing the semantic features of the two different levels are equally processed for each channel and each pixel, which leads to poor defogging effect. Therefore, the embodiment of the present application improves the defogging effect of the defogging model by performing weighted feature fusion on the channel level and the pixel level.

[0077] Specifically, the semantic features obtained by the shallow network branch and the semantic features obtained by the deep network branch are fused to obtain fusion features; then the weighted fusion network is used to calculate the channel-level weighted weight and the pixel-level weighted weight, and then the fusion features are fused according to the channel-level weighted weight and the pixel-level weighted weight to obtain the haze image semantic features.

[0078] It should be noted that the channel-level weighted weight can be calculated by sequentially passing the semantic features to be weighted through an average pooling layer, a convolution kernel of 1x1 convolution layer, a ReLU activation function layer, a convolution kernel of 1x1 convolution layer and a Sigmoid activation function layer, and the pixel-level weighted weight can be calculated by sequentially passing the semantic features to be weighted through a convolution kernel of 1x1 convolution layer, a ReLU activation function layer, a convolution kernel of 1x1 convolution layer and a Sigmoid activation function layer.

[0079] ​In addition, the weighted fusion is multiple channel-level weighting and pixel-level weighting, the fusion features can be sequentially subjected to channel-level weighted feature fusion and pixel-level weighted feature fusion, and can be subjected to channel-level weighted feature fusion and pixel-level weighted feature fusion across layers, for example, the features obtained through multiple channel-level weighted feature fusion can be subjected to re-fusion, and then the features obtained through the re-fusion are subjected to pixel-level weighted feature fusion, and the embodiments of the present application do not limit this.

[0080] Based on the above embodiments, the present application provides a preferred embodiment, in which the shallow network branch includes a shallow first convolution unit, a shallow second convolution unit, a shallow third convolution unit and a shallow residual unit; the deep network branch includes a deep first convolution unit, a deep second convolution unit, a deep third convolution unit, a deep first residual unit, a deep second residual unit and a deep third residual unit.

[0081] Step 210 includes:

[0082] Step 211, based on the to-be-dehazed image, applying the shallow first convolution unit to determine a first shallow feature, and applying the deep first convolution unit and the deep first residual unit to determine a first deep feature;

[0083] Step 212, based on the first shallow feature and the first deep feature, determining a first fusion feature;

[0084] Step 213, based on the first fusion feature, applying the shallow second convolution unit to determine a second shallow feature, and applying the deep second convolution unit and the deep second residual unit to determine a second deep feature;

[0085] Step 214, based on the second shallow feature and the second deep feature, determining a second fusion feature;

[0086] Step 215, based on the second fusion feature, applying the shallow third convolution unit to determine a third shallow feature, and applying the deep third convolution unit to determine a third deep feature;

[0087] Step 216, based on the third shallow feature and the third deep feature, determining a third fusion feature;

[0088] Step 217, based on the third fusion feature, applying the shallow residual unit to determine a fourth shallow feature, and applying the deep third residual unit to determine a fourth deep feature;

[0089] Step 218, based on the fourth shallow feature and the fourth deep feature, determining a fusion feature.

[0090] It should be noted that the shallow first convolution unit includes a convolution layer with a convolution kernel of 7x7 and a convolution step of 1 and a ReLU activation function layer, the network structures of the shallow second convolution unit and the shallow third convolution unit are the same, and each includes a convolution layer with a convolution kernel of 4x4 and a convolution offset of 2 and a ReLU activation function layer, the shallow residual unit includes 4 residual blocks, and each residual block includes two convolution units, and each convolution unit includes a convolution layer with a convolution kernel of 3x3 and a convolution step of 1 and a ReLU activation function layer. The deep first convolution unit includes a convolution layer with a convolution kernel of 7x7 and a convolution step of 1 and a ReLU activation function layer, the network structures of the deep second convolution unit and the deep third convolution unit are the same, and each includes a convolution layer with a convolution kernel of 4x4 and a convolution offset of 2 and a ReLU activation function layer, the network structures of the deep first residual unit, the deep second residual unit and the deep third residual unit are the same, and each includes 4 residual blocks, and each residual block includes two convolution units, and each convolution unit includes a convolution layer with a convolution kernel of 3x3 and a convolution step of 1 and a ReLU activation function layer.

[0091] In addition, Figure 3 is a data flow diagram of the haze image encoder provided by the present application. As Figure 3 shown, the haze image encoder fuses the semantic features extracted by the shallow network branch and the semantic features of the deep network branch as follows:

[0092] S1, the image to be dehazed is input into the shallow network branch and the deep network branch, the shallow first convolution unit performs convolution operation on the image to be dehazed, and the first shallow feature The deep first convolution unit performs convolution operation on the image to be dehazed, and inputs the feature obtained after the convolution operation into the deep first residual unit to obtain the first deep feature

[0093] S2, the first shallow feature and the first deep feature are fused to obtain the first fusion feature, and the first fusion feature is subjected to convolution operation with a convolution kernel size of 1x1 to obtain the first fusion feature corresponding to the shallow first fusion feature and the first fusion feature corresponding to the deep first fusion feature

[0094] S3, the first fusion feature is input into the shallow second convolution unit to perform convolution operation to obtain the second shallow feature At the same time is input into the deep second convolution unit to perform convolution operation, and the feature obtained after the convolution operation is input into the deep second residual unit to obtain the second deep feature

[0095] S4, the and fusion to obtain the second fusion feature, and the second fusion feature is subjected to a convolution operation with a convolution kernel size of 1x1 to obtain a shallow second fusion feature corresponding to the second fusion feature and a deep second fusion feature corresponding to the second fusion feature

[0096] S5, the is input to the shallow third convolution unit to perform a convolution operation, and a third shallow feature is obtained is input to the deep third convolution unit to perform a convolution operation, and a third deep feature is obtained

[0097] S6, the and fusion to obtain the third fusion feature, and the third fusion feature is subjected to a convolution operation with a convolution kernel size of 1x1 to obtain a shallow third fusion feature corresponding to the third fusion feature and a deep third fusion feature corresponding to the third fusion feature

[0098] S7, the is input to the shallow residual unit to obtain a fourth shallow feature at the same time, the is input to the deep third residual unit to obtain a fourth shallow feature

[0099] S8, the and fusion and convolution operation with a convolution kernel size of 1x1 to obtain a fusion feature

[0100] Based on the above embodiment, a preferred embodiment of the present application is provided, wherein the weighted fusion network comprises a first weighted fusion unit, a second weighted fusion unit, a third weighted fusion unit, a first channel level weighting unit, a second channel level weighting unit, a third channel level weighting unit and a pixel level weighting unit; the first weighted fusion unit, the second weighted fusion unit and the third weighted fusion unit each comprise a channel weighting subunit and a pixel weighting subunit;

[0101] Step 220 comprises:

[0102] Step 221, based on the fusion feature, applying the first weighted fusion unit and the first channel level weighting unit to determine a first channel weighted feature;

[0103] Step 222, based on the first channel weighted feature, applying the second weighted fusion unit to determine a first weighted fusion feature;​

[0104] Step 223, based on the first weighted fusion feature, applying a second channel level weighting unit to determine a second channel level weighting feature;

[0105] Step 224, based on the second channel weighting feature, applying a third weighted fusion unit to determine a second weighted fusion feature;

[0106] Step 225, based on the second weighted fusion feature, applying a third channel level weighting unit to determine a third channel level weighting feature;

[0107] Step 226, based on the first weighted fusion feature, the second weighted fusion feature and the third channel weighting feature, determining a fourth channel level weighting feature;

[0108] Step 227, based on the fourth channel level weighting feature, applying a pixel level weighting unit to determine the haze image semantic feature.

[0109] It should be noted that the network structures of the channel weighting subunit, the first channel level weighting unit, the second channel level weighting unit and the third channel level weighting unit are all the same, all including a pooling kernel of 2x2 and a step of 2 mean pooling layer, a convolution unit including a convolution kernel size of 1x1 and a ReLU activation function layer, and a convolution unit including a convolution kernel size of 1x1 and a Sigmoid activation function layer. The network structures of the pixel weighting subunit and the pixel level weighting unit are all the same, all including a convolution unit including a convolution kernel size of 1x1 and a ReLU activation function layer, and a convolution unit including a convolution kernel size of 1x1 and a Sigmoid activation function layer.

[0110] In addition, as shown in Figure 3 , the data flow of the first weighted fusion unit, the second weighted fusion unit and the third weighted fusion unit is as follows:

[0111] S1', input the semantic feature F into the channel weighting subunit to obtain the channel level weight through the average pooling operation, the convolution operation and the activation operation, multiply the channel level weight with the semantic feature F to obtain the channel weighting feature F'.

[0112] S2', input the channel weighting feature F' into the pixel weighting subunit to obtain the pixel level weight through the convolution operation and the activation operation, multiply the pixel level weight with the channel weighting feature F' to obtain the pixel level weighting feature F".

[0113] As shown in Figure 3 , the data flow of the weighted fusion network is as follows:

[0114] S9, input the fusion feature The first pixel-level weighted feature is input to the first channel-level weighting unit to obtain the first channel-level weighted feature through an average pooling operation, a convolution operation and an activation operation.

[0115] S10, the first channel-level weighted feature is input to the second weighting fusion unit to obtain the first weighted fusion feature, and the first weighted fusion feature is input to the second channel-level weighting unit to obtain the second channel-level weighted feature through an average pooling operation, a convolution operation and an activation operation.

[0116] S11, the second channel-level weighted feature is input to the third weighting fusion unit to obtain the second weighted fusion feature, and the second weighted fusion feature is input to the third channel-level weighting unit to obtain the third channel-level weighted feature through an average pooling operation, a convolution operation and an activation operation.

[0117] S12, the first weighted fusion feature, the second weighted fusion feature and the third channel-level weighted feature are summed to obtain the fourth channel-level weighted feature.

[0118] S13, the fourth channel-level weighted feature is input to the pixel-level weighting unit to obtain the haze image semantic feature through a convolution operation and an activation operation.

[0119] Based on the above embodiment, Figure 4 is a flowchart of the haze removal model training method provided by the present application. As Figure 4 shown, the training steps of the haze removal model are as follows:

[0120] Step 410, determine the unpaired haze sample image and clear sample image, and the initial model; the initial haze image encoder, the initial semantic decoder, the initial clear image encoder, the initial reconstruction decoder and the style encoder;

[0121] Specifically, an initial model containing an initial haze image encoder, an initial semantic decoder, an initial clear image encoder, an initial reconstruction decoder and a style encoder is constructed, and the initial model is a cycle generative adversarial model. The initial haze image encoder and the initial semantic decoder are used to generate a haze removal image according to the haze image; the initial clear image encoder, the initial reconstruction decoder and the initial model of the style encoder are used to generate a haze image according to a clear image, wherein the style editor is used to extract the haze image style feature from the haze image.

[0122] It should be noted that the network structure of the initial haze image encoder and the initial clear image encoder is the same, and all includes a shallow network branch, a deep network branch and a weighted fusion network; the shallow network branch includes a shallow first convolution unit, a shallow second convolution unit, a shallow third convolution unit and a shallow residual unit; the deep network branch includes a deep first convolution unit, a deep second convolution unit, a deep third convolution unit, a deep first residual unit, a deep second residual unit and a deep third residual unit; the shallow first convolution unit includes a convolution layer with a convolution kernel of 7x7 and a convolution step of 1 and a ReLU activation function layer, and the network structure of the shallow second convolution unit and the shallow third convolution unit is the same, and both include a convolution layer with a convolution kernel of 4x4 and a convolution offset of 2 and a ReLU activation function layer, and the shallow residual unit includes 4 residual blocks, each of which contains two convolution units, and each convolution unit contains a convolution layer with a convolution kernel of 3x3 and a convolution step of 1 and a ReLU activation function layer. The deep first convolution unit includes a convolution layer with a convolution kernel of 7x7 and a convolution step of 1 and a ReLU activation function layer, and the network structure of the deep second convolution unit and the deep third convolution unit is the same, and both include a convolution layer with a convolution kernel of 4x4 and a convolution offset of 2 and a ReLU activation function layer, and the network structure of the deep first residual unit, the deep second residual unit and the deep third residual unit is the same, and all include 4 residual blocks, each of which contains two convolution units, and each convolution unit contains a convolution layer with a convolution kernel of 3x3 and a convolution step of 1 and a ReLU activation function layer. The weighted fusion network includes a first weighted fusion unit, a second weighted fusion unit, a third weighted fusion unit, a first channel-level weighting unit, a second channel-level weighting unit, a third channel-level weighting unit and a pixel-level weighting unit; the first weighted fusion unit, the second weighted fusion unit and the third weighted fusion unit all include a channel weighting subunit and a pixel weighting subunit; the network structure of the channel weighting subunit, the first channel-level weighting unit, the second channel-level weighting unit and the third channel-level weighting unit is the same, and all include a pooling layer with a pooling kernel of 2x2 and a step of 2, a convolution unit including a convolution layer with a convolution kernel size of 1x1 and a ReLU activation function layer, and a convolution unit including a convolution layer with a convolution kernel size of 1x1 and a Sigmoid activation function layer. The network structure of the pixel weighting subunit and the pixel-level weighting unit is the same, and all include a convolution unit including a convolution layer with a convolution kernel size of 1x1 and a ReLU activation function layer, and a convolution unit including a convolution layer with a convolution kernel size of 1x1 and a Sigmoid activation function layer.

[0123] Figure 5 The network structure of the decoder provided by the application is shown in the figure. Figure 5As shown, the network structure of the initial semantic decoder and the initial reconstruction decoder is the same, and both include a residual module ResBlocks and three convolution units. ResBlocks includes eight convolution layers Conv5_1 to Conv5_8 with a convolution kernel size of 3x3. Convolution unit Conv5_9 and convolution unit Conv5_10 include a convolution layer with a convolution kernel size of 5x5 and a ReLU activation function layer, and convolution unit Conv5_11 includes a convolution layer with a convolution kernel size of 7x7 and a Tanh activation function layer.

[0124] Figure 6 is the network structure diagram of the style encoder provided by the application. As shown in the figure, Figure 6 the style encoder includes four convolution units. Among them, convolution unit Conv4_1 includes a convolution layer with a convolution kernel size of 7x7 and a ReLU activation function layer, convolution unit Conv4_2 and convolution unit Conv4_3 include a convolution layer with a convolution kernel size of 4x4 and a ReLU activation function layer, and convolution unit Conv4_4 includes a convolution layer with a convolution kernel size of 1x1 and a step of 1.

[0125] Step 420, based on the initial clear image encoder, the style encoder and the initial reconstruction decoder, applying the haze sample image and the clear sample image, generating the synthesized haze image corresponding to the clear sample image, and based on the initial haze image encoder and the initial semantic decoder, applying the synthesized haze image corresponding to the clear sample image, generating the synthesized clear image corresponding to the clear sample image; at the same time, based on the initial haze image encoder and the initial semantic decoder, applying the haze sample image, generating the synthesized clear image corresponding to the haze sample image, and based on the initial clear image encoder, the style encoder and the initial reconstruction decoder, applying the synthesized haze image corresponding to the clear sample image and the synthesized clear image corresponding to the haze sample image, generating the synthesized haze image corresponding to the haze sample image;

[0126] Specifically, the unpaired haze sample images and clear sample images are input, in forward migration, the initial clear image encoder extracts clear image semantic features of the clear sample images, the style encoder extracts haze image style features of the haze sample images, the clear image semantic features and the haze image style features are input to the initial reconstruction decoder to obtain synthesized haze images corresponding to the clear sample images, meanwhile, the initial haze image encoder extracts haze image semantic features of the haze sample images, and then the haze image semantic features are input to the initial semantic decoder to obtain synthesized clear images corresponding to the haze sample images; in reverse migration, the synthesized haze images corresponding to the clear sample images are input to the initial haze image encoder to obtain haze semantic features of the synthesized haze images corresponding to the clear sample images, then the haze semantic features are input to the initial semantic decoder to obtain synthesized clear images corresponding to the clear sample images, meanwhile, the synthesized clear images corresponding to the haze sample images are input to the initial clear image encoder to obtain clear image semantic features of the synthesized clear images corresponding to the haze sample images, the synthesized haze images corresponding to the clear sample images are input to the style encoder to obtain haze image style features of the synthesized haze images corresponding to the clear sample images, the clear image semantic features of the synthesized clear images corresponding to the haze sample images and the haze image style features of the synthesized haze images corresponding to the clear sample images are input to the initial reconstruction decoder to obtain synthesized haze images corresponding to the haze sample images.

[0127] It should be noted that the loss is calculated by 430 after one forward migration and one reverse migration, and the current iteration is completed.

[0128] Step 430, based on the haze sample images, the synthesized clear images corresponding to the haze sample images, the synthesized haze images corresponding to the haze sample images, the clear sample images, the synthesized haze images corresponding to the clear sample images and the synthesized clear images corresponding to the clear sample images, determine the loss, and based on the loss, perform parameter iteration on the initial model until the loss converges, and the obtained initial haze image encoder is used as the haze image encoder and the initial semantic decoder is used as the semantic decoder.

[0129] Specifically, according to the haze sample images, the synthesized clear images corresponding to the haze sample images, the synthesized haze images corresponding to the haze sample images, the clear sample images, the synthesized haze images corresponding to the clear sample images and the synthesized clear images corresponding to the clear sample images obtained by the current round of forward migration and reverse migration, loss calculation is performed to obtain the loss, and then the initial model is iterated according to the loss until the loss converges, and then the parameters of the initial haze image encoder obtained after training are fixed as the haze image encoder in the haze removal model, and the parameters of the initial semantic decoder obtained after training are fixed as the semantic decoder in the haze removal model.

[0130] It should be noted that the loss can include cycle consistency loss, semantic consistency loss, target domain adversarial loss, feature representation adversarial loss, consistent mapping loss, self-reconstruction loss, dark channel loss and style feature contrast loss, and the embodiments of the present application do not limit this.

[0131] Based on the above embodiments, the present application provides a preferred embodiment, and the loss is obtained by weighted sum of cycle consistency loss, semantic consistency loss, target domain adversarial loss, feature representation adversarial loss, consistent mapping loss, self-reconstruction loss, dark channel loss and style feature contrast loss. The style feature contrast loss is calculated according to the style feature contrast learning module, and the style feature contrast learning module includes a style encoder and the 9th and 13th layers of the pre-trained VGG19 model.

[0132] It should be noted that, Figure 7 is the data flow diagram of the style feature contrast learning module provided by the present application. As Figure 7 shown, first, the style features of the images are extracted by the style encoder, and then the 9th and 13th layers of the pre-trained VGG19 model are used to extract deep features as positive and negative samples to construct a contrast space. Specifically, the dehazed image J dehaze is first processed by the style encoder to obtain the style features, and then processed by the convolution unit of the pre-trained VGG19 model to obtain the features. Finally, the features are taken as the anchor points of the contrast space, and the style features of the input clear image and the style features of the haze image are selected as the positive and negative samples respectively to construct the contrast loss. Similarly, the input false haze image I fahaze is processed in the same way to obtain the features as the anchor points to construct a new sample space and contrast loss. In the present application, two contrast spaces are constructed: the "anchor" of the first contrast space CR-1 is the style feature of the dehazed image, and the "positive" and "negative" are the style features of the clear image and the haze image respectively; the "anchor" of the second contrast space CR-2 is the style feature of the reconstructed false haze image, and the "positive" and "negative" are the style features of the haze image and the clear image respectively.

[0133] The data processing steps of the style feature contrast learning module are as follows:

[0134] Step one, the dehazed image J dehaze , the reconstructed false haze image I fahaze , the clear image J clear and the haze image I haze input to the module are extracted to obtain their style features.

[0135] Step two, the style is input to the pre-trained VGG network for feature extraction to obtain deep features.

[0136] Step three, build two contrast space optimization model training. The fog image features are taken as the anchor points of the first contrast space, and the features of the clear image and the fog image are taken as the positive and negative samples respectively; the reconstructed false fog image features are taken as the anchor points of the second contrast space, and the features of the fog image and the clear image are taken as the positive and negative samples respectively, to build a contrast loss constraint model training.

[0137] In addition, the formula of the loss function corresponding to the cycle consistency loss is as follows:

[0138] L CC =E I [||I cychaze -I haze ||1]+E J [||J cycclear -J clear ||1]

[0139] In the formula, E I represents the loss for distinguishing the fog domain, E J represents the loss for distinguishing the clear domain, I haze represents the fog sample image, J clear represents the clear sample image, I cychaze represents the synthesized fog image corresponding to the fog sample image, J cycclear represents the synthesized clear image corresponding to the clear sample image.

[0140] The formula of the loss function corresponding to the semantic consistency loss is as follows:

[0141]

[0142] In the formula, represents the features from the conv5-l of the pre-trained VGG19 model on ImageNet.

[0143] The formula of the loss function corresponding to the target domain adversarial loss is as follows:

[0144]

[0145]

[0146] In the formula, I haze represents the fog sample image, J clear represents the clear sample image, I fahaz represents the synthesized fog image corresponding to the clear sample image, J dehaze represents the synthesized clear image corresponding to the fog sample image, F represents the expected value of the distribution function, P I represents the fog sample distribution, P J represents the clear sample distribution.

[0147] The formula of the loss function corresponding to the feature representation adversarial loss is as follows:

[0148]

[0149] In the formula, E I represents the haze domain, E J represents the clear domain, Dis C represents the feature discriminator, z I represents the haze image semantic feature, z J represents the clear image semantic feature.

[0150] The formula of the loss function corresponding to the consistency mapping loss is as follows:

[0151] L idt =||D J (E I (J clear ))-J clear ||1

[0152] In the formula, J clear represents the clear sample image, E I represents the haze image encoder, D J represents the semantic decoder.

[0153] The formula of the loss function corresponding to the self-reconstruction loss is as follows:

[0154]

[0155] In the formula, E I represents the haze image encoder, D J represents the semantic decoder, E J represents the clear image encoder, D J represents the reconstruction decoder, represents the style encoder, I haze represents the haze sample image, J clear represents the clear sample image.

[0156] The formula of the loss function corresponding to the dark channel loss is as follows:

[0157] L dcp =||DC(D J (E I (I haze )))||1

[0158] In the formula, E I represents the haze image encoder, D J represents the semantic decoder, I hazedenotes a haze sample image, DC denotes a dark channel used for calculating, L dcp for making the pixel value of at least one color channel tend to 0.

[0159] The image defogging method provided by the embodiment of the present application calculates the style feature contrast loss through the style feature contrast learning module, so that the generated synthetic haze image is more realistic, thereby further improving the defogging quality of the defogging model.

[0160] Based on the above embodiment, the present application provides a preferred embodiment, in which the formula of the loss function corresponding to the style feature contrast loss is as follows:

[0161]

[0162]

[0163] In the formula, denotes a synthetic clear image style loss; denotes a synthetic haze image style loss; G i denotes the hidden features of the i-th layer extracted from the VGG19 model, w i is the weight coefficient of the i-th layer, β i denotes the preset adjustment coefficient of the i-th layer, i is 9 or 13; d(x, y) is the mean absolute error between x and y (i.e., L1 loss is calculated, L1 loss function is also called MAE (mean abs error), i.e., the absolute value of the difference between the predicted value and the true value); denotes a style encoder; J clear denotes a clear sample image; I haze denotes a haze sample image; J dehaze denotes a synthetic clear image corresponding to the haze sample image; I fahaze denotes a synthetic haze image corresponding to the clear sample image. Preferably, β9=0.25, β 13 =1.

[0164] According to the above loss functions, the formula of the total loss function is as follows:

[0165]

[0166] In the formula, λ CC =λ sr =λ idt =10, λ dcp =0.5. Wherein,

[0167] Based on the above embodiment, the present application provides a preferred embodiment, Figure 8is the network architecture diagram of the initial model provided by the application. As shown in Figure 8 , a set of unpaired haze sample images I haze and clear sample images J clear are obtained. In the forward migration, the haze sample images are converted into the corresponding synthetic clear images of the haze sample images by haze removal, and the clear sample images are converted into the corresponding synthetic haze images of the clear sample images by haze addition. Specifically, the corresponding haze image encoder E I and the clear image encoder E J are used to encode I haze and J clear to obtain semantic features, Z I and Z J , the haze image style feature Z haze in the haze sample image I h is extracted by the style encoder E ; after Z J and Z h are spliced in the channel dimension, they are input into the reconstruction decoder D I to obtain the synthetic haze image I fahaze corresponding to the clear sample image; Z I is input into the semantic decoder D J to obtain the synthetic clear image J dehaze corresponding to the haze sample image. After completing the forward migration, regression migration is implemented again. Similarly, the synthetic haze image I fahaze of the clear sample image and the synthetic clear image J dehaze corresponding to the haze sample image are converted back into clear images and haze images. Specifically, the corresponding haze image encoder E I and the clear image encoder E J are used to encode I fahaze and J dehaze to obtain semantic features, and the haze image style feature in I fahaze is extracted by the style encoder E and is reconstructed and decoded to finally obtain the synthetic haze image I cychaze corresponding to the haze sample image and the synthetic clear image J cycclear corresponding to the clear sample image. In the process of iterative training, the style feature contrast loss is calculated by the style feature contrast learning module.

[0168] The image haze removal device provided by the application is described below. The image haze removal device described below can be mutually corresponding to the image haze removal method described above.

[0169] Figure 9 is the structure diagram of the image haze removal device provided by the application. As shown inFigure 9 The apparatus includes a determination module 910 and a defogging module 920.

[0170] The determination module 910 is configured to determine a to-be-defogged image.

[0171] The determination module 910 is configured to determine a to-be-defogged image.

[0172] The defogging module 920 is configured to input the to-be-defogged image into a defogging model to obtain a defogged image output by the defogging model.

[0173] The defogging model includes a haze image encoder and a semantic decoder. The haze image encoder is configured to extract and fuse semantic features of the to-be-defogged image at two different levels to obtain haze image semantic features. The semantic decoder is configured to obtain the defogged image based on the haze image semantic features. The defogging model is trained based on unpaired haze sample images and clear sample images.

[0174] The image defogging apparatus provided by the embodiment of the present application includes a determination module configured to determine a to-be-defogged image. The defogging module is configured to input the to-be-defogged image into a defogging model to obtain a defogged image output by the defogging model. The defogging model includes a haze image encoder and a semantic decoder. The haze image encoder is configured to extract and fuse semantic features of the to-be-defogged image at two different levels to obtain haze image semantic features. The semantic decoder is configured to obtain the defogged image based on the haze image semantic features. The defogging model is trained based on unpaired haze sample images and clear sample images.

[0175] Based on any of the above embodiments, the image defogging further includes a haze image semantic feature acquisition module, which includes:

[0176] The fusion sub-module is configured to fuse the semantic features of the to-be-defogged image extracted by the shallow network branch and the semantic features of the to-be-defogged image extracted by the deep network branch to obtain fused features.

[0177] The weighting sub-module is configured to perform channel-level and pixel-level weighted feature fusion on the fused features based on a weighted fusion network to obtain haze image semantic features.

[0178] Based on any of the above embodiments, the fusion sub-module includes:

[0179] The first layer feature sub-module is configured to determine a first shallow layer feature by applying a shallow first convolution unit based on the to-be-defogged image and determine a first deep layer feature by applying a deep first convolution unit and a deep first residual unit.

[0180] The first layer fusion feature submodule is configured to determine a first fusion feature based on the first shallow layer feature and the first deep layer feature.

[0181] The second layer feature submodule is configured to determine a second shallow layer feature by applying a shallow second convolution unit based on the first fusion feature, and determine a second deep layer feature by applying a deep second convolution unit and a deep second residual unit.

[0182] The second layer fusion feature submodule is configured to determine a second fusion feature based on the second shallow layer feature and the second deep layer feature.

[0183] The third layer feature submodule is configured to determine a third shallow layer feature by applying a shallow third convolution unit based on the second fusion feature, and determine a third deep layer feature by applying a deep third convolution unit.

[0184] The third layer fusion feature submodule is configured to determine a third fusion feature based on the third shallow layer feature and the third deep layer feature.

[0185] The fourth layer feature submodule is configured to determine a fourth shallow layer feature by applying a shallow residual unit based on the third fusion feature, and determine a fourth deep layer feature by applying a deep third residual unit.

[0186] The fusion feature submodule is configured to determine a fusion feature based on the fourth shallow layer feature and the fourth deep layer feature.

[0187] Based on any of the above embodiments, the weighting submodule includes:

[0188] The first channel feature submodule is configured to determine a first channel weighted feature by applying a first weighted fusion unit and a first channel level weighting unit based on the fusion feature.

[0189] The first weighted fusion submodule is configured to determine a first weighted fusion feature by applying a second weighted fusion unit based on the first channel weighted feature.

[0190] The second channel feature submodule is configured to determine a second channel level weighted feature by applying a second channel level weighting unit based on the first weighted fusion feature.

[0191] The second weighted fusion submodule is configured to determine a second weighted fusion feature by applying a third weighted fusion unit based on the second channel weighted feature.

[0192] The third channel feature submodule is configured to determine a third channel level weighted feature by applying a third channel level weighting unit based on the second weighted fusion feature.

[0193] The fourth channel feature submodule is configured to determine a fourth channel level weighted feature based on the first weighted fusion feature, the second weighted fusion feature, and the third channel weighted feature.

[0194] The image semantic feature sub-module is configured to determine a haze image semantic feature based on the fourth channel level weighted feature and by applying a pixel level weighting unit.

[0195] According to any of the above embodiments, the image haze removing device further comprises a training module, which comprises:

[0196] The training determination sub-module is configured to determine unpaired haze sample images and clear sample images, and an initial model; an initial haze image encoder, an initial semantic decoder, an initial clear image encoder, an initial reconstruction decoder, and a style encoder.

[0197] The cycle signature sub-module is configured to generate a synthesized haze image corresponding to the clear sample image based on the initial clear image encoder, the style encoder, and the initial reconstruction decoder, and apply the haze sample image and the clear sample image, and generate a synthesized clear image corresponding to the clear sample image based on the initial haze image encoder and the initial semantic decoder, and apply the synthesized haze image corresponding to the clear sample image; at the same time, generate a synthesized clear image corresponding to the haze sample image based on the initial haze image encoder and the initial semantic decoder, and apply the haze sample image, and generate a synthesized haze image corresponding to the haze sample image based on the initial clear image encoder, the style encoder, and the initial reconstruction decoder, and apply the synthesized haze image corresponding to the clear sample image and the synthesized clear image corresponding to the haze sample image.

[0198] The iteration sub-module is configured to determine a loss based on the haze sample image, the synthesized clear image corresponding to the haze sample image, the synthesized haze image corresponding to the haze sample image, the clear sample image, the synthesized haze image corresponding to the clear sample image, and the synthesized clear image corresponding to the clear sample image, and perform parameter iteration on the initial model based on the loss until the loss converges, and obtain the initial haze image encoder as the haze image encoder and the initial semantic decoder as the semantic decoder.

[0199] Figure 10 An example of an entity structure diagram of an electronic device is shown as Figure 10As shown, the electronic device can include a processor 1010, a communications interface 1020, a memory 1030, and a communications bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 complete mutual communication through the communications bus 1040. The processor 1010 can call the logical instructions in the memory 1030 to execute the image defogging method, which includes determining a to-be-defogged image; inputting the to-be-defogged image into a defogging model to obtain a defogged image output by the defogging model; the defogging model includes a haze image encoder and a semantic decoder; the haze image encoder is used to extract and fuse semantic features of the to-be-defogged image at two different levels to obtain haze image semantic features; the semantic decoder is used to obtain the defogged image based on the haze image semantic features; and the defogging model is obtained based on unpaired haze sample images and clear sample images.

[0200] In addition, the logical instructions in the memory 1030 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0201] On the other hand, the present application also provides a computer program product, which includes a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program is executed by a processor, and the computer can execute the image defogging method provided by the above-mentioned method, which includes: determining a to-be-defogged image; inputting the to-be-defogged image into a defogging model to obtain a defogged image output by the defogging model; the defogging model includes a haze image encoder and a semantic decoder; the haze image encoder is used to extract and fuse semantic features of the to-be-defogged image at two different levels to obtain haze image semantic features; the semantic decoder is used to obtain the defogged image based on the haze image semantic features; and the defogging model is obtained based on unpaired haze sample images and clear sample images.

[0202] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the image defogging method provided by any of the above methods, and the method comprises: determining a to-be-defogged image; inputting the to-be-defogged image into a defogging model to obtain a defogged image output by the defogging model; the defogging model comprises a haze image encoder and a semantic decoder; the haze image encoder is configured to extract and fuse semantic features of the to-be-defogged image at two different levels to obtain haze image semantic features; the semantic decoder is configured to obtain the defogged image based on the haze image semantic features; and the defogging model is trained based on unpaired haze sample images and clear sample images.

[0203] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0204] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and necessary universal hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0205] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image defogging method, characterized by, The method comprises the steps of: determining a to-be-dehazed image; inputting the to-be-dehazed image into a dehazing model to obtain a dehazed image output by the dehazing model; the dehazing model comprises a haze image encoder and a semantic decoder; the haze image encoder is used to extract and fuse semantic features of the to-be-dehazed image at two different levels to obtain haze image semantic features; the semantic decoder is used to obtain a dehazed image based on the haze image semantic features; the dehazing model is trained based on unpaired haze sample images and clear sample images; wherein the haze image encoder comprises a shallow network branch, a deep network branch and a weighted fusion network; the haze image semantic feature is obtained by the following steps: fuse the semantic features of the to-be-dehazed image extracted by the shallow network branch and the semantic features of the to-be-dehazed image extracted by the deep network branch to obtain a fusion feature; based on the weighted fusion network, perform channel-level and pixel-level weighted feature fusion on the fusion feature to obtain the haze image semantic feature; wherein the shallow network branch comprises a shallow first convolution unit, a shallow second convolution unit, a shallow third convolution unit and a shallow residual unit; the deep network branch comprises a deep first convolution unit, a deep second convolution unit, a deep third convolution unit, a deep first residual unit, a deep second residual unit and a deep third residual unit; the fusion of the semantic features of the to-be-dehazed image extracted by the shallow network branch and the semantic features of the to-be-dehazed image extracted by the deep network branch to obtain a fusion feature comprises: based on the to-be-dehazed image, apply the shallow first convolution unit to determine a first shallow feature, and apply the deep first convolution unit and the deep first residual unit to determine a first deep feature; based on the first shallow feature and the first deep feature, determine a first fusion feature; based on the first fusion feature, apply the shallow second convolution unit to determine a second shallow feature, and apply the deep second convolution unit and the deep second residual unit to determine a second deep feature; based on the second shallow feature and the second deep feature, determine a second fusion feature; based on the second fusion feature, apply the shallow third convolution unit to determine a third shallow feature, and apply the deep third convolution unit to determine a third deep feature; based on the third shallow feature and the third deep feature, determine a third fusion feature; based on the third fusion feature, apply the shallow residual unit to determine a fourth shallow feature, and apply the deep third residual unit to determine a fourth deep feature; based on the fourth shallow feature and the fourth deep feature, determine the fusion feature.

2. The image defogging method of claim 1, wherein, the weighted fusion network comprises a first weighted fusion unit, a second weighted fusion unit, a third weighted fusion unit, a first channel-level weighting unit, a second channel-level weighting unit, a third channel-level weighting unit and a pixel-level weighting unit; the first weighted fusion unit, the second weighted fusion unit and the third weighted fusion unit each comprise a channel weighting subunit and a pixel weighting subunit; The weighted feature fusion of the channel level and the pixel level is performed on the fusion feature based on the weighted fusion network, to obtain the haze image semantic feature, comprising: Based on the fusion feature, the first weighted fusion unit and the first channel level weighting unit are applied to determine the first channel weighted feature; Based on the first channel weighted feature, the second weighted fusion unit is applied to determine the first weighted fusion feature; Based on the first weighted fusion feature, the second channel level weighting unit is applied to determine the second channel level weighted feature; Based on the second channel weighted feature, the third weighted fusion unit is applied to determine the second weighted fusion feature; Based on the second weighted fusion feature, the third channel level weighting unit is applied to determine the third channel level weighted feature; Based on the first weighted fusion feature, the second weighted fusion feature and the third channel weighted feature, the fourth channel level weighted feature is determined; Based on the fourth channel level weighted feature, the pixel level weighting unit is applied to determine the haze image semantic feature.

3. The image defogging method of claim 1, wherein, The training steps of the dehazing model are as follows: Determine the unpaired haze sample image and clear sample image, and the initial model; initial haze image encoder, initial semantic decoder, initial clear image encoder, initial reconstruction decoder and style encoder; Based on the initial clear image encoder, style encoder and initial reconstruction decoder, the haze sample image and the clear sample image are applied to generate the synthesized haze image corresponding to the clear sample image, and based on the initial haze image encoder and the initial semantic decoder, the synthesized haze image corresponding to the clear sample image is applied to generate the synthesized clear image corresponding to the clear sample image; at the same time, based on the initial haze image encoder and the initial semantic decoder, the haze sample image is applied to generate the synthesized clear image corresponding to the haze sample image, and based on the initial clear image encoder, style encoder and initial reconstruction decoder, the synthesized haze image corresponding to the clear sample image and the synthesized clear image corresponding to the haze sample image are applied to generate the synthesized haze image corresponding to the haze sample image; Based on the haze sample image, the synthesized clear image corresponding to the haze sample image, the synthesized haze image corresponding to the haze sample image, the clear sample image, the synthesized haze image corresponding to the clear sample image and the synthesized clear image corresponding to the clear sample image, the loss is determined, and the initial model is iterated based on the loss until the loss converges, the initial haze image encoder obtained by training is taken as the haze image encoder and the initial semantic decoder is taken as the semantic decoder.

4. The image defogging method of claim 3, wherein, The loss is obtained by weighted summation based on a cycle consistency loss, a semantic consistency loss, a target domain adversarial loss, a feature representation adversarial loss, a consistent mapping loss, a self-reconstruction loss, a dark channel loss, and a style feature contrast loss; the style feature contrast loss is calculated based on a style feature contrast learning module; the style feature contrast learning module comprises a style encoder and the 9th layer and the 13th layer of a pre-trained VGG19 model.

5. The image defogging method of claim 4, wherein, The formula of the loss function corresponding to the style feature contrast loss is as follows: In the formula, indicates the synthesis clear image style loss; indicates the synthesis haze image style loss; G i indicates the hidden features of the i-th layer extracted from the VGG19 model, w i is the weight coefficient of the i-th layer, β i indicates the preset adjustment coefficient of the i-th layer, i is 9 or 13; d(x, y) indicates the mean absolute error between x and y; indicates the style encoder; J clear indicates the clear sample image; I haze indicates the haze sample image; J dehaze indicates the synthesized clear image corresponding to the haze sample image; I fahaze indicates the synthesized haze image corresponding to the clear sample image.

6. An image defogging apparatus characterized by comprising: The method comprises the steps of: determining a to-be-dehazed image; inputting the to-be-dehazed image into a dehazing model to obtain a dehazed image output by the dehazing model; the dehazing model comprises a haze image encoder and a semantic decoder; the haze image encoder is configured to extract and fuse semantic features of the to-be-dehazed image at two different levels to obtain haze image semantic features; the semantic decoder is configured to obtain a dehazed image based on the haze image semantic features; the dehazing model is trained based on unpaired haze sample images and clear sample images; wherein the haze image encoder comprises a shallow network branch, a deep network branch, and a weighted fusion network; the haze image semantic features are obtained by: fusing the semantic features of the to-be-dehazed image extracted by the shallow network branch and the semantic features of the to-be-dehazed image extracted by the deep network branch to obtain fused features; performing channel-level and pixel-level weighted feature fusion on the fused features based on the weighted fusion network to obtain the haze image semantic features; wherein the shallow network branch comprises a shallow first convolution unit, a shallow second convolution unit, a shallow third convolution unit, and a shallow residual unit; the deep network branch comprises a deep first convolution unit, a deep second convolution unit, a deep third convolution unit, a deep first residual unit, a deep second residual unit, and a deep third residual unit; the fusion of the semantic features of the to-be-dehazed image extracted by the shallow network branch and the semantic features of the to-be-dehazed image extracted by the deep network branch to obtain fused features comprises: based on the to-be-dehazed image, applying the shallow first convolution unit to determine a first shallow feature and applying the deep first convolution unit and the deep first residual unit to determine a first deep feature; based on the first shallow feature and the first deep feature, determining a first fused feature; based on the first fused feature, applying the shallow second convolution unit to determine a second shallow feature and applying the deep second convolution unit and the deep second residual unit to determine a second deep feature; based on the second shallow feature and the second deep feature, determining a second fused feature; based on the second fused feature, applying the shallow third convolution unit to determine a third shallow feature and applying the deep third convolution unit to determine a third deep feature; based on the third shallow feature and the third deep feature, determining a third fused feature; applying the shallow residual unit based on the third fusion feature to determine a fourth shallow feature, and applying the deep third residual unit based on the fourth shallow feature to determine a fourth deep feature; determining the fusion feature based on the fourth shallow feature and the fourth deep feature.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the image defogging method of any one of claims 1-5 when executing the program.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the image defogging method of any one of claims 1-5 when executed by the processor.

Images