Image defogging method and device, electronic equipment and computer storage medium

Abstract

The application discloses a haze removal method and device of an image, electronic equipment and a computer storage medium. Specifically, a haze image to be removed is obtained; the haze image is input into a haze removal model, the haze removal model is called to perform haze removal processing on the haze image, a haze-removed image output by the haze removal model is obtained, the haze removal model is obtained by modulating a target component of a sample haze image to obtain a modulation sample, and optimization is performed on the basis of modulation-independent feature invariance loss and modulation-dependent content consistency loss; and the target modulation component of the sample haze image includes transmittance and surface color of the sample haze image. The application obtains rich modulation samples by modulating the target component of the sample haze image, trains the haze removal model on the basis of the modulation-independent feature invariance loss and the modulation-dependent content consistency loss calculated based on the modulation samples, and thus the generalization and stability of the haze removal model are improved, and the haze removal performance of the haze removal model in various scenes is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a haze removal method and device for images, an electronic device, and a computer storage medium. BACKGROUND

[0002] In order to realize haze removal of a single haze image, researchers have proposed a haze removal method based on priori. However, the haze removal method based on priori largely depends on the effectiveness of priori, resulting in poor stability of haze removal effect. Based on the powerful learning ability of neural networks, researchers have proposed a supervised learning method for haze removal of haze images, such as DehazeNet and MSCNN. However, these methods have damaged the stability and generalization of the haze removal model, resulting in poor haze removal performance of the haze removal model. SUMMARY

[0003] In view of the above problems, the present application provides a haze removal method and device for images, an electronic device, and a computer storage medium, to improve the stability and generalization of the image haze removal model, and further improve the haze removal performance of the haze removal model. The specific scheme is as follows:

[0004] A haze removal method for images comprises:

[0005] Obtaining a haze image to be removed;

[0006] Inputting the haze image into a haze removal model, calling the haze removal model to perform haze removal processing on the haze image, and obtaining a haze removal image output by the haze removal model, wherein the haze removal model is obtained by modulating target components of a sample haze image to obtain a modulation sample, and calculating modulation-independent feature invariance loss and modulation-dependent content consistency loss by using the modulation sample; and the target components of the sample haze image include transmittance and surface color of the sample haze image.

[0007] Optionally, the haze removal model comprises a transmittance estimation sub-module, an image restoration sub-module, and an atmospheric light estimation sub-module.

[0008] The calling of the haze removal model to perform haze removal processing on the haze image and the obtaining of the haze removal image output by the haze removal model comprise:

[0009] Calling the image restoration sub-module to process the haze image, and obtaining the haze removal image output by the haze removal model.

[0010] Optionally, the haze removal model is obtained by modulating target components of a sample haze image to obtain a modulation sample, and calculating modulation-independent feature invariance loss and modulation-dependent content consistency loss by using the modulation sample, comprising:

[0011] A first modulation sample is obtained by using the transmittance of the sample haze image and the surface color of the sample haze image after modulation.

[0012] a second modulation sample is obtained by the surface color of the sample haze image and the transmittance of the sample haze image after modulation;

[0013] a modulation-independent feature invariance loss is obtained by calculating a modulation-independent feature invariance loss function based on the sample haze image and the first modulation sample;

[0014] a second modulation-independent feature invariance loss is obtained by calculating a modulation-independent feature invariance loss function based on the sample haze image and the second modulation sample;

[0015] the first modulation-independent feature invariance loss and the second modulation-independent feature invariance loss are added to obtain a modulation-independent feature invariance loss;

[0016] a first modulation-dependent content consistency loss is obtained by calculating a modulation-dependent content consistency loss function based on the second modulation sample and the transmittance of the sample haze image after modulation;

[0017] a second modulation-dependent content consistency loss is obtained by calculating a modulation-dependent content consistency loss function based on the first modulation sample and the surface color of the sample haze image after modulation;

[0018] the first modulation-dependent content consistency loss and the second modulation-dependent content consistency loss are added to obtain a modulation-dependent content consistency loss;

[0019] an initialized model is trained based on the modulation-independent feature invariance loss and the modulation-dependent content consistency loss to obtain a defogging model.

[0020] Optionally, before the initial model is trained based on the modulation-independent feature invariance loss and the modulation-dependent content consistency loss to obtain the defogging model, the method further comprises:

[0021] a reconstruction loss is obtained by calculating a reconstruction loss function based on the sample haze image and the reconstructed sample haze image;

[0022] an adversarial loss is obtained by calculating an adversarial loss function based on the first clear image and the defogging image corresponding to the sample haze image, the content of the first clear image being different from the content of the defogging image corresponding to the sample haze image;

[0023] a reconstruction consistency loss is obtained by calculating a reconstruction consistency loss function based on the first clear image and the second clear image;

[0024] The initialized model is trained based on the modulation-independent feature invariance loss and the modulation-dependent content consistency loss to obtain the defogging model, comprising:

[0025] The total loss is calculated by modulating the feature invariance loss irrelevant to modulation, the content consistency loss relevant to modulation, the reconstruction loss, the adversarial loss and the reconstruction consistency loss, and an initial model is trained to obtain a defogging model.

[0026] A defogging device of an image comprises:

[0027] An acquisition unit is configured to acquire a haze image to be defogged.

[0028] A processing unit is configured to input the haze image into a defogging model, call the defogging model to perform defogging processing on the haze image, and obtain a defogging image output by the defogging model, wherein the defogging model is obtained by modulating a target component of a sample haze image to obtain a modulation sample, and optimizing the modulation sample with a feature invariance loss irrelevant to modulation and a content consistency loss relevant to modulation as targets.

[0029] Optionally, the defogging model comprises a transmittance estimation sub-module, an image restoration sub-module and an atmospheric light estimation sub-module.

[0030] The processing unit calls the defogging model to perform defogging processing on the haze image, and obtains the defogging image output by the defogging model, for calling the image restoration sub-module to process the haze image, and obtaining the defogging image output by the defogging model.

[0031] Optionally, the device further comprises a model training unit configured to obtain a first modulation sample by a transmittance of a sample haze image and a surface color of the sample haze image after modulation.

[0032] The model training unit obtains a second modulation sample by a surface color of the sample haze image and a transmittance of the sample haze image after modulation.

[0033] The model training unit calculates a feature invariance loss function irrelevant to modulation by the sample haze image and the first modulation sample, and obtains a first feature invariance loss irrelevant to modulation.

[0034] The model training unit calculates a feature invariance loss function irrelevant to modulation by the sample haze image and the second modulation sample, and obtains a second feature invariance loss irrelevant to modulation.

[0035] The model training unit adds the first feature invariance loss irrelevant to modulation and the second feature invariance loss irrelevant to modulation to obtain a feature invariance loss irrelevant to modulation.

[0036] The model training unit calculates a content consistency loss function relevant to modulation by the second modulation sample and the transmittance of the sample haze image after modulation, and obtains a first content consistency loss relevant to modulation.

[0037] The first modulation-related content consistency loss function is calculated by the first modulation sample and the surface color of the sample haze image after modulation, and the second modulation-related content consistency loss is obtained.

[0038] The first modulation-related content consistency loss and the second modulation-related content consistency loss are added to obtain the modulation-related content consistency loss.

[0039] The dehazing model is obtained by training the initialized model through the modulation-independent feature invariance loss and the modulation-related content consistency loss.

[0040] Optionally, the model training unit comprises:

[0041] The first modulation sample acquisition unit is configured to obtain the first modulation sample through the transmittance of the sample haze image and the surface color of the sample haze image after modulation.

[0042] The second modulation sample acquisition unit is configured to obtain the second modulation sample through the surface color of the sample haze image and the transmittance of the sample haze image after modulation.

[0043] The first loss acquisition unit is configured to calculate the modulation-independent feature invariance loss function through the sample haze image and the first modulation sample, and obtain the first modulation-independent feature invariance loss.

[0044] The second loss acquisition unit is configured to calculate the modulation-independent feature invariance loss function through the sample haze image and the second modulation sample, and obtain the second modulation-independent feature invariance loss.

[0045] The first loss merging unit is configured to add the first modulation-independent feature invariance loss and the second modulation-independent feature invariance loss to obtain the modulation-independent feature invariance loss.

[0046] The third loss acquisition unit is configured to calculate the modulation-related content consistency loss function through the second modulation sample and the transmittance of the sample haze image after modulation, and obtain the first modulation-related content consistency loss.

[0047] The fourth loss acquisition unit is configured to calculate the modulation-related content consistency loss function through the first modulation sample and the surface color of the sample haze image after modulation, and obtain the second modulation-related content consistency loss.

[0048] The second loss merging unit is configured to add the first modulation-related content consistency loss and the second modulation-related content consistency loss to obtain the modulation-related content consistency loss.

[0049] The output unit is configured to obtain the dehazing model by training the initialized model through the modulation-independent feature invariance loss and the modulation-related content consistency loss.

[0050] An electronic device comprising at least one processor and a memory connected to the processor, wherein:

[0051] The memory is configured to store computer programs or instructions;

[0052] The processor is configured to execute the computer programs or instructions to enable the electronic device to implement the steps of the image defogging method of any one of the above.

[0053] A computer storage medium, the storage medium carrying one or more computer program instructions, when the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the image defogging method of any one of the above.

[0054] By the above technical solutions, the image defogging method, device, electronic device and computer storage medium provided by the present application can obtain modulation samples with different transmittances and surface colors through directional modulation of the transmittance of the haze image and directional modulation of the surface color of the haze image. Based on these modulation samples with different transmittances and surface colors, the defogging model can be trained by modulating the feature invariance loss irrelevant to the task, thereby improving the anti-interference ability of the corresponding sub-module to the task-independent factor, and the defogging model can be trained by modulating the content consistency loss related to the task, thereby improving the sensitivity of the corresponding sub-module to the task-related factor, thereby improving the stability and generalization of the defogging model, and enhancing the defogging performance of the defogging model in various haze scenes. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0056] Figure 1 The flowchart of the image defogging method provided by the embodiment of the present application;

[0057] Figure 2 The structural schematic diagram of the image defogging model provided by the embodiment of the present application;

[0058] Figure 3 The flowchart of the modulation of the haze image provided by the embodiment of the present application;

[0059] Figure 4 The schematic diagram of the feature invariance loss irrelevant to the modulation process;

[0060] Figure 5A schematic diagram of the content consistency loss process provided by the embodiment of the present application is shown in FIG. 1.

[0061] Figure 6 A comparison chart of the image defogging effect of the image defogging method provided by the embodiment of the present application and the existing image defogging methods De-Hamer and D4-Net on the whole image is shown in FIG. 2.

[0062] Figure 7-a and Figure 7-b A comparison chart of the image defogging effect of the image defogging method provided by the embodiment of the present application and other image defogging methods on the haze image in the SOTS data set in terms of image detail processing is shown in FIG. 3.

[0063] Figure 8 A comparison chart of the image defogging effect of the image defogging method provided by the embodiment of the present application and other image defogging methods on the indoor image data set in the SOTS in terms of two image quality evaluation indexes of PSNR and SSIM is shown in FIG. 4.

[0064] Figure 9-a and Figure 9-b A comparison chart of the image defogging effect of the image defogging method provided by the embodiment of the present application and other image defogging methods on the haze image in the O-HAZE and Pan data set in terms of image detail processing is shown in FIG. 5.

[0065] Figure 10 A comparison chart of the image defogging effect of the image defogging method provided by the embodiment of the present application and other image defogging methods on the outdoor image data set in the O-HAZE and Pan in terms of two image quality evaluation indexes of PSNR and SSIM is shown in FIG. 6.

[0066] Figure 11-a and Figure 11-b A comparison chart of the image defogging effect of the image defogging method provided by the embodiment of the present application and other image defogging methods on the real haze image is shown in FIG. 7.

[0067] Figure 12 A schematic diagram of the image defogging device provided by the embodiment of the present application is shown in FIG. 8.

[0068] Figure 13 A schematic diagram of the electronic device provided by the embodiment of the present application is shown in FIG. 9. DETAILED DESCRIPTION

[0069] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0070] In order to realize the haze removal of a single haze image, researchers propose a prior-based method for haze removal. However, the prior-based method largely depends on the effectiveness of the prior, resulting in poor stability of the haze removal effect. Based on the powerful learning ability of neural networks, researchers propose to use a supervised learning method to remove haze from haze images, such as DehazeNet and MS-CNN. However, these methods compromise the stability and generalization of the haze removal model, resulting in poor haze removal performance of the haze removal model.

[0071] In order to improve the stability and generalization of the haze removal model and enhance the haze removal performance of the haze removal model in various haze scenes, the present application provides a haze removal method for unpaired images based on target component modulation. By directional modulation of the transmittance of the haze image and directional modulation of the surface color of the haze image, modulation samples with different target component parameters can be obtained. Based on these modulation samples with different target component parameters, the haze removal network can be trained by modulation-independent feature invariance loss to improve the anti-interference ability of the model to task-independent factors, and by modulation-dependent content consistency loss to improve the sensitivity of the model to task-related factors, thereby improving the stability and generalization of the haze removal model and enhancing the haze removal performance of the haze removal model in various haze scenes.

[0072] The present application will be described in detail below in conjunction with specific embodiments.

[0073] Embodiment One

[0074] Figure 1 A flowchart of the haze removal method for images provided by the embodiments of the present application is shown.

[0075] Figure 1 Steps S10 to S30 can be understood as the process of constructing a haze removal model, and step S40 can be understood as the process of applying the haze removal model to remove haze from the haze image input to the haze removal model to obtain a haze-removed image.

[0076] In some embodiments, steps S10 to S30 can be performed by one device, and step S40 can be performed by another device. In other embodiments, the execution process of steps S10 to S30 and the execution process of step S40 can be performed by the same device.

[0077] S10, Prepare the data set used for model training.

[0078] In order to improve the stability and generalization of the haze removal model provided by the embodiments of the present application, the embodiments of the present application prepare different data sets for two different image haze removal scenes, indoor images and outdoor images.

[0079] For the image defogging scene of indoor images, the embodiments of the present application adopt the ITS dataset, which is a subset of the RESIDE dataset, containing 13990 pairs of synthetic indoor images generated using real depth values, including a variety of indoor scenes. Specifically, the haze images and clear images in the ITS dataset are one-to-one paired, and the 13990 pairs of haze images and clear images are paired images generated based on the depth values of real scene images. During the training process, the image pairs are randomly shuffled to realize unpaired image training.

[0080] For the image defogging scene of outdoor images, the embodiments of the present application adopt the RESIDE-unpaired dataset, which is a high-quality outdoor image dataset customized for the image defogging task of unpaired images, containing 2903 outdoor haze images and 3577 outdoor clear images, and all images in the RESIDE-unpaired dataset are taken from the real world, and the haze images and clear images are not one-to-one paired, that is, the images in the dataset are different haze images and different clear images.

[0081] It should be noted that in the embodiments of the present application, the images in the dataset can also be randomly cropped and randomly horizontally flipped for data enhancement to improve the performance of model training.

[0082] S20, construct an image defogging model and obtain modulation samples for model training.

[0083] The process of synthesizing haze images can be described by the atmospheric scattering model (ASM), that is,

[0084] I(x) = J(x) t(x) + A (1-t(x))

[0085] In the formula, I(x) represents the corresponding haze image at pixel (x), J(x) and t(x) represent the corresponding clear image and the transmittance at pixel (x) of the haze image, respectively, and A is the background atmospheric light in the given haze image. Assuming that the haze is homogeneous, the mathematical expression of the transmittance t(x) is:

[0086] t(x) = e -β·d(x)

[0087] In the formula, β is the scattering coefficient that determines the intensity of the haze, and d(x) is the scene depth of the haze image.

[0088] Generally, the value of atmospheric light A for a haze image can be estimated according to the pixel value of the sky region of the haze image or the highest brightness of the whole haze image, and the atmospheric light value can also be estimated from the average value of the 0.1% brightest pixels in the dark channel image of the haze image.

[0089] Based on this, the structural diagram of the defogging model provided by the embodiments of the present application is as shown in Figure 3 which consists of three sub-modules: transmittance estimation submodule G t , image restoration submodule G J and atmospheric light estimation submodule A-Model. The transmittance estimation submodule G t has a structure of 8 down-sampling layers and 8 up-sampling layers based on Unet, the image restoration submodule G J has a ResNet structure of 9 residual blocks, and the atmospheric light estimation submodule A-Model estimates the atmospheric light value from the average value of the 0.1% brightest pixels in the dark channel image of the haze image.

[0090] Specifically, for a given haze image I, after processing by the transmittance estimation submodule G t and the image restoration submodule G J , the transmittance image and the defogging image of the haze image obtained are denoted as t I and J I respectively.

[0091] Figure 3 The diagram shows the schematic diagram of the transmittance modulation of the haze image and the surface color modulation of the defogging image provided by the embodiments of the present application.

[0092] As shown in Figure 3 , M t can be understood as a transmittance modulation submodule, and M J can be understood as a surface color modulation submodule. For the transmittance image t I of the haze image I, after processing by the transmittance modulation submodule M t , the modulated transmittance image of the haze image I can be obtained Similarly, for the defogging image J I of the haze image I, after processing by the surface color modulation submodule M J , the modulated defogging image can be obtained

[0093] The transmittance modulation submodule M t processes the transmittance image t I to obtain The process is as follows:

[0094] From the modeling formula of the transmittance above, the transmittance t I can be represented as:

[0095]

[0096] In the formula, β I d represents the scattering coefficient of haze image I. I (x) represents the scene depth of haze image I.

[0097] Taking the logarithm of both sides of the above equation, we get:

[0098] ln(t I (x))=-β I ·d I (x)

[0099] Furthermore, by multiplying both sides of the above equation by the coefficient α and then taking the exponent on both sides, we can obtain:

[0100]

[0101] In the formula, the coefficient α is an adjustable positive variable. The scattering coefficient obtained by modulation is α·β I A new transmittance map.

[0102] Since the scene depth d is determined only by the distance from the object to the camera, the scene depth d is fixed in the same scene. Therefore, the transmittance of the haze image I can be changed by controlling the value of α.

[0103] Therefore, according to the Atmospheric Scattering Model (ASM), the modulated haze image I can be obtained. Mt U Mt It can be represented as:

[0104]

[0105] In the formula, ⊙ represents element-wise multiplication, and A I J represents the atmospheric light value of haze image I. I This is the original dehazed image corresponding to haze image I. The modulated transmittance diagram, U Mt This is a modulated image of the haze. (Source: [Insert Image Here]) Figure 4 It can be seen that the modulated haze image U Mt It is the modulated transmittance And the original dehazed image J I Haze image synthesized using the Atmospheric Scattering Model (ASM) U Mt .

[0106] It should be noted that the transmittance t of the modulated haze image I provided in this embodiment of the application is... I get During the process, The image contains complete scene depth information dI, which allows the modulated image I to... Mt It has a more realistic effect.

[0107] Surface color modulation submodule M J Processing dehazed images J I get The process is as follows:

[0108] This application's embodiments target dehazing images J I Two different surface color modulation strategies are proposed, one of which is for dehazed images J I Saturation modulation of the surface color is performed on dehazed images. I The surface color is gamma corrected.

[0109] Specifically, for dehazed image J I Saturation modulation of surface color is actually the process of modifying the original dehazed image J. I The surface color saturation is multiplied by a coefficient β, where β is an adjustable positive variable, and this is applied to the original dehazed image J. I Multiplying the surface color saturation by a coefficient β can change the dehazed image J. I The purity of the surface color, thereby achieving modulation and dehazing of the image J. I The surface color is modulated to obtain the dehazed image.

[0110] Specifically, for dehazed image J I Gamma correction of surface color is essentially adjusting the dehazed image. I The gamma curve was used to simulate the dehazed image J by adjusting different γ values. I Different exposure levels are used to achieve modulated dehazing of the image J. I The surface color is modulated to obtain the dehazed image.

[0111] The above two modulation dehazing images J I Methods to change the surface color can all alter the dehazed image. I The appearance of surface color is determined, thus providing different color attributes for the same scene, enriching the diversity of samples, and further improving the generalization of the dehazing model. Furthermore, regardless of the dehazed image J... I Whether it's saturation modulation or gamma correction of surface color, it's all based on the dehazed image. IThe operation of the middle pixel is thus, the pixels with the same color in the original haze image still have the same color in the modulated haze image, which helps to preserve the edge information of the pixels in the haze image, and the accurate matching with the transmittance map of the original haze image can be achieved through the edge information of the pixels in the haze image, and a more realistic modulated haze image is obtained.

[0112] It should be noted that the modulation of the surface color of the defogging image J I is achieved by randomly performing saturation modulation and gamma correction, and the specific modulation mode of the surface color of the defogging image J I each time is not limited.

[0113] By modulating the surface color of the defogging image J I , a modulated defogging image J is obtained. Then, the modulated defogging image J is combined with the corresponding transmittance map t I , and a modulated haze image I Mj is obtained. Mj I

[0114]

[0115] In the formula, is the element multiplication, A I is the atmospheric light value of the haze image I, is the modulated defogging image, t I is the transmittance map corresponding to the haze image I, I Mj is the modulated haze image. It can be known from Figure 3 that the modulated haze image I Mj is a haze image I Mj synthesized by the modulated defogging image J and the corresponding transmittance map t I through the atmospheric scattering model (ASM).

[0116] The embodiments of the present application obtain the modulated haze images I Mt and I Mj by directional modulation of the transmittance of the haze image and directional modulation of the surface color of the haze image, and based on a large number of modulated haze images I Mt and I Mj as modulation samples, the image defogging model proposed in the embodiments of the present application is trained, which can effectively improve the generalization of the defogging model and enhance the defogging performance of the defogging model.

[0117] S30, training the defogging model by using the images in the data set and the modulation samples.

[0118] The training process of the image defogging model provided in the embodiments of the present application can be performed on an NVIDIA Tesla V100 using a Pytorch framework. In order to optimize the image defogging model proposed in the embodiments of the present application, the Adam optimizer is used to optimize the defogging model.

[0119] Specifically, the parameters of the Adam optimizer can be set to β1=0.9, β2=0.999, the learning rate can be set to 0.0001, the batch size can be set to 4, the value range of the variable α and the variable β can be set to [0.5, 1.5], the value range of the variable γ can be set to [0.6, 0.9], and the visibility of the dark light area can be increased.

[0120] In the embodiments of the present application, in addition to the reconstruction loss function L rec , the adversarial loss function L adv and the reconstruction consistency loss function L idt used in conventional model training, the modulation-independent feature invariance loss function L mfi and the modulation-related content consistency loss function L mcc are also proposed to train and optimize the image defogging model proposed in the embodiments of the present application.

[0121] Figure 2 The schematic diagram of the reconstruction loss function L rec , the adversarial loss function L adv and the consistency loss function L idt that need to be calculated in the conventional model training provided in the embodiments of the present application is shown in FIG.

[0122] As can be seen from Figure 2 , the reconstruction loss function L rec is obtained from the reconstructed haze image I rec and the original haze image I. Specifically, the reconstruction loss function L rec is obtained by performing pixel consistency constraint on the reconstructed haze image U rec and the original haze image I.

[0123] The mathematical expression of the reconstruction loss function L rec is as follows:

[0124] L rec =||U rec -I||1

[0125] In the formula, U rec is the reconstructed haze image, and the reconstruction principle still conforms to the imaging principle of the atmospheric scattering model (ASM), that is, U rec =JI ⊙t I +A I ⊙(1-t I ), ||·||1 is the norm of L1.

[0126] Although the reconstructed haze image U rec It is the transmittance map t of the original haze image I. I Atmospheric light value A I and dehazed image J I Synthesized, but reconstructed, haze image U rec It still cannot be made exactly the same as the original haze image I. This is achieved by calculating the reconstruction loss function L. rec This can ensure the transmittance map t I And dehazed image J I Within the solution space constrained by the Atmospheric Scattering Model (ASM).

[0127] Depend on Figure 2 It can be seen that the adversarial loss function L adv It is derived from the dehazed image J I The image obtained is a clear, real image J. Specifically, in this embodiment of the application, a discriminator (PatchGAN) can be used to identify the dehazed image J. I And a true, clear image J is used to obtain the adversarial loss function L. adv Adversarial loss function L adv The mathematical expression is:

[0128] L adv =E J [log D(J)]+E I [log(1-D(G J (I)))]

[0129] In the formula, D is the discriminator and E is the mathematical expectation.

[0130] It should be noted that the embodiments of this application calculate the adversarial loss function L. adv The dehazed image J used I The true clear image J and the real clear image I can be two unrelated images; that is, the true clear image J does not need to be the dehazed image J corresponding to the hazy image I. I Adversarial loss function L adv By calculating the dehazed image J I The similar distribution between the two clear images, J and J, can encourage the dehazing model to pay more attention to the texture details of the image when processing it. This helps the dehazing model provided in this application to produce a more realistic dehazing effect and retain as much detail information as possible in the dehazed image.

[0131] Reconstructing the consistency loss function Lidt is obtained by inputting the clear image J into the image restoration submodule G J , specifically, inputting the clear image J into the image restoration submodule G J , the image restoration submodule G J brings the output result and the clear image J into the reconstruction consistency loss function L idt to obtain the reconstruction consistency loss. The mathematical expression of the reconstruction consistency loss function L idt is as follows:

[0132] L idt =||G J (J)-J||1

[0133] In the formula, ||·||1 is the L1 norm.

[0134] It should be noted that the image restoration submodule G J is used to remove the haze component in the original haze image I to obtain the haze-removed image J I , if the clear image J is input into the image restoration submodule G J , the obtained result is still a clear image, therefore, by inputting the clear image J into the image restoration submodule G J , the output result is brought into the expression of the reconstruction consistency loss function L idt to obtain the reconstruction consistency loss, and the obtained reconstruction consistency loss can be used to suppress the image restoration submodule G J from introducing unnecessary color distortion and structure damage.

[0135] Figure 4 The modulation-independent feature invariance loss function L mfi provided by the embodiment of the present application is shown in the schematic diagram of the process.

[0136] For a given haze image I, the embodiment of the present application defines the modulation-independent feature invariance loss function as L mfi , which can be known from the following formula: Figure 4 It can be known from the formula that the modulation-independent feature invariance loss function L mfi is composed of two parts and , wherein

[0137]

[0138] In the formula, E t and E J represent the corresponding encoders of G t and G J , respectively, and ||·||1 is the L1 norm. Therefore, the mathematical expression of the modulation-independent feature invariance loss function L mfi is as follows:

[0139]

[0140] It should be noted that, due to the transmittance estimation submodule G t The goal is to obtain the transmittance map t of haze image I. I Therefore, the transmittance estimation submodule G t The key focus should be on the transmittance t that affects haze image I. I The scattering coefficient β and scene depth d.

[0141] It should also be noted that the transmittance estimation submodule G t The corresponding encoder E t Processing modulated haze images U Mj When compared with the original haze image I, the extracted features should be the modulated haze image U. Mj The transmittance t of the original haze image I, and by Figure 3 It can be seen that the modulated haze image U Mj The difference between haze image U and the original haze image I is that haze image U Mj The surface color has been modulated, therefore the transmittance estimation submodule G t The corresponding encoder E t Extracting the modulated haze image I Mj When comparing the features of the original haze image I, the forced transmittance estimation submodule G can be used. t The corresponding encoder E t The extracted features are the modulated haze image U Mj The same transmittance t features as the original haze image I can be used to enhance the transmittance estimation submodule G. t The corresponding encoder E t The ability to resist interference from features unrelated to transmittance t can, for example, enhance the transmittance estimation submodule G. t The corresponding encoder E t The ability to resist interference from the surface color characteristics of haze images.

[0142] It should be noted that, due to the image restoration submodule G J The goal is to obtain the dehazed image J from haze image I. I Therefore, the image restoration submodule G J The focus should be on scene content unrelated to smog concentration, namely the image restoration submodule G. J The image that should be given special attention is the dehazed image J, which corresponds to the haze image I. I Related information, such as the dehazed image J which is the focus of this application embodiment. I The surface color, because of the dehazed image J IThe surface color of the haze image I does not affect the haze density of the haze image I.

[0143] It should be further explained that the image restoration submodule G J The corresponding encoder E J In processing the modulated haze image I Mt When the original haze image I, the extracted features should be the modulated haze image U Mt The scene content in the original haze image I that is irrelevant to the haze density, such as the surface color of the haze image I, is irrelevant to the modulated haze image U Figure 4 It can be seen that the modulated haze image I Mt The difference between the original haze image I and the modulated haze image U Mt is that the transmittance of the haze image I is modulated, so the image restoration submodule G J The corresponding encoder E J In extracting the features of the modulated haze image U Mt The original haze image I, the image restoration submodule G J The corresponding encoder E J The extracted features are the modulated haze image U Mt The surface color of the original haze image I is the same, and the image restoration submodule G J The corresponding encoder E J The anti-interference ability of extracting features irrelevant to the surface color, for example, the image restoration submodule G J The corresponding encoder E J The anti-interference ability of extracting the transmittance features of the haze image.

[0144] The embodiment of the present application can improve the transmittance estimation submodule G mfi The corresponding encoder E t The anti-interference ability of extracting the transmittance features of the haze image. t The corresponding encoder E J The anti-interference ability of extracting the transmittance features of the haze image. J The corresponding encoder E t The anti-interference ability of extracting the transmittance features of the haze image. J The anti-interference ability of extracting the transmittance features of the haze image.

[0145] Figure 5 The embodiment of the present application provides a modulation haze image related content consistency loss function process.

[0146] The embodiment of the present application defines the modulation related content consistency loss function as L mcc The corresponding encoder E Figure 5It can be seen that the modulation-related content consistency loss function L mcc Composed of two parts and Composition, in which,

[0147]

[0148] Therefore, we can obtain the modulation-related content consistency loss function L. mcc The mathematical expression is:

[0149]

[0150] It should be noted that, due to the transmittance estimation submodule G t The goal is to obtain the transmittance map t of haze image I. I Moreover, the modulated haze image I Mt It is the modulated transmittance And the original dehazed image J I Haze image synthesized using the Atmospheric Scattering Model (ASM) U Mt The modulation-related content consistency loss function L... mmcc The first part of the loss function It is the modulated haze image I Mt and modulated transmittance Therefore, by calculating the modulation-related content consistency loss function L, we can obtain... mmcc The first part of the loss function This can improve the transmittance estimation submodule G t The accuracy of obtaining the transmittance map t of haze images from haze image analysis.

[0151] It should also be noted that, due to the image restoration submodule G J The goal is to obtain the dehazed image J from haze image I. I The modulated haze image I Mj It is a modulated dehazed image And the original transmittance map t I Haze image synthesized using the Atmospheric Scattering Model (ASM) I Mj Modulation-related content consistency loss function L mmc The second part of the loss function c It is the modulated haze image I Mj and modulated dehazed image The obtained, and here the modulated dehazed image It is derived from the dehazed image J I It is obtained through surface color modulation, therefore, the relevant content consistency loss function L is modulated. mmccthe second part of the loss function The image restoration submodule G J The accuracy of the surface color of the haze image obtained by analyzing the haze image.

[0152] The embodiment of the present application calculates the content consistency loss function L mcc The transmittance estimation submodule G t The accuracy of the transmittance map t and the image restoration submodule G J The accuracy of the surface color of the haze image obtained by analyzing the haze image, thereby improving the haze removal performance of the haze removal model.

[0153] The embodiment of the present application will configure different weights according to the above five loss functions, and finally obtain the overall loss function. The expression of the overall loss function is:

[0154] L total = λ1L rec + λ2L adv + λ3L idt + λ4L mfi + λ5L mcc

[0155] In the formula, λ1 to λ5 are the weights of balancing different loss functions, which can be set as λ1 = 1, λ2 = 0.05, λ3 = 1, λ4 = 0.1, and λ5 = 0.2.

[0156] It should be noted that the numerical value of the weight of balancing different loss functions in the overall loss function provided by the embodiment of the present application can be set as the above numerical value, or the numerical value of the weight can be adjusted according to the change in the actual model training process. The purpose of setting the weight of different loss functions is to minimize the overall loss function obtained finally, so that the haze removal model provided by the embodiment of the present application can achieve the most ideal haze removal performance.

[0157] S40, inputting the haze image into the haze removal model to obtain a haze-removed image.

[0158] Figure 6 The image haze removal method provided by the embodiment of the present application is compared with the haze removal effect of the existing image haze removal methods De-Hamer and D4-Net for the whole image.

[0159] Figure 6 (a) is the original haze image, Figure 6 (b) is the haze-removed image processed by the haze removal method De-Hamer, Figure 6 (c) is the haze-removed image processed by the haze removal method D4-Net, Figure 6 (d) is the haze-removed image obtained by processing the haze removal model provided by the embodiment of the present application.

[0160] As can be seen from Figure 6 , the dehazed image obtained by the dehazing model provided in the embodiments of the present application is the clearest among the three dehazed images, especially the outline of the mountain peak part in the dehazed image processed by the dehazing model provided in the embodiments of the present application is the clearest, the dehazed image is closer to the real clear image, and the dehazing effect is the best.

[0161] The embodiments of the present application use SOTS, O-HAZE and Pan data sets to evaluate the performance of the dehazing model provided in the embodiments of the present application. Among them, the indoor image subset of the SOTS data set contains 500 pairs of synthesized haze images and clear images of indoor images, that is, 500 indoor clear images and 500 haze images synthesized according to the 500 indoor clear images. The O-HAZE data set contains 45 pairs of haze images and clear images of outdoor images in real scenes. The Pan data set contains 96 pairs of haze images and clear images of outdoor images generated using the Make3D data set.

[0162] Figure 7-a And Figure 7-b The comparison chart of the image dehazing method (TCM) provided in the embodiments of the present application and other image dehazing methods in terms of image detail processing for haze images in the SOTS data set.

[0163] Figure 7-a And Figure 7-b It can be understood that the qualitative comparison chart of the image dehazing method (TCM) provided in the embodiments of the present application and other image dehazing methods on the indoor data set SOTS.

[0164] As can be seen from Figure 7-a and Figure 7-b , compared with the dark channel prior dehazing method (DCP) and the non-local prior dehazing method (NLD) and some dehazing methods based on unsupervised learning, such as CycleGAN, DistentGAN, RefineDNet, YOLY, and D4-Net, the image dehazing method provided in the embodiments of the present application can restore more original structures of potential details in the image, thereby obtaining the best dehazing effect closest to the real image. For example Figure 7-a and Figure 7-bThe comparison chart of the details of the table and the door handle in the defogging image obtained by removing the haze from the haze image in the indoor data set SOTS by using the image defogging method provided in the embodiments of the present application and other defogging methods is shown in the figure. It can be seen from the figure that the image defogging method provided in the embodiments of the present application can clearly restore more details in the image, and the door handle and the table in the defogging image after removing the haze also retain more details. Meanwhile, the obtained defogging image is also the defogging image closest to the real clear image.

[0165] In order to obtain the best performance comparison result, the image defogging method (TCM) provided in the embodiments of the present application is compared with the dark channel prior defogging method (DCP), the non-local prior defogging method (NLD), and some defogging methods based on unsupervised learning, such as CycleGAN, DistentGAN, RefineDNet, YOLY, and D4-Net. The data set and the defogging methods mentioned above are used, and the peak signal to noise ratio (PSNR) and the structural similarity (SSIM) in the image quality evaluation index are used as the evaluation index to quantitatively evaluate the defogging performance of the defogging methods mentioned above and the defogging method provided in the embodiments of the present application.

[0166] Figure 8 The comparison chart of the PSNR and the SSIM in the image quality evaluation index of the image defogging method provided in the embodiments of the present application and other image defogging methods for the SOTS indoor image data set.

[0167] Figure 8 It can be understood that the quantitative comparison result of the image defogging method (TCM) provided in the embodiments of the present application and other defogging methods on the indoor data set SOTS.

[0168] From Figure 8 It can be seen from the figure that the image defogging method (TCM) provided in the embodiments of the present application obtains the highest score in the peak signal to noise ratio (PSNR) and the structural similarity (SSIM) two image quality evaluation indexes. Especially, compared with the most advanced image defogging method D4-Net based on unsupervised learning, the PSNR of the image defogging method (TCM) provided in the embodiments of the present application is improved by 1.51 dB, and the SSIM score also leads by 0.023 points.

[0169] Figure 9-a andFigure 9-b The image defogging method (TCM) provided by the embodiments of the present application and other image defogging methods are compared in terms of effects on image detail processing of haze images in the O-HAZE and Pan data sets.

[0170] Figure 9-a And Figure 9-b It can be understood that the image defogging method (TCM) provided by the embodiments of the present application and other image defogging methods are compared in terms of qualitative comparison on the O-HAZE and Pan data sets.

[0171] From Figure 9-a And Figure 9-b It can be seen from the above that the image defogging method (TCM) provided by the embodiments of the present application can effectively restore the original structure of the objects in the haze image compared with the dark channel prior defogging method (DCP) and the non-local prior defogging method (NLD) and some defogging methods based on unsupervised learning, such as CycleGAN, DistentGAN, RefineDNet, YOLY, and D4-Net. For example Figure 9-a And Figure 9-b The comparison of the ground and trees in the defogging image obtained after removing the haze from one haze image in the outdoor data set O-HAZE and the Pan data set by the image defogging method provided by the embodiments of the present application and other defogging methods is shown in FIGS. 1 and 2. It can be seen from the figures that the image defogging method provided by the embodiments of the present application can well restore the original structure of the ground tiles and trees, and the quality of the defogging image obtained by the present application is the highest compared with the quality of the defogging image obtained by other methods.

[0172] Figure 10 The comparison results of the image defogging method (TCM) provided by the embodiments of the present application and other defogging methods on the outdoor data set O-HAZE and the Pan data set.

[0173] From Figure 10As can be seen from the above, the image defogging method (TCM) provided in the embodiments of the present application obtains the highest scores in the two image quality evaluation indexes of peak signal to noise ratio (PSNR) and structural similarity (SSIM) in both the O-HAZE outdoor data set and the Pan outdoor data set. In the O-HAZE outdoor data set, compared with the current most advanced unsupervised defogging method D4-Net, the PSNR of the image defogging method (TCM) provided in the embodiments of the present application is improved by 1.31 dB, and the score of SSIM also leads by 0.1009. In the Pan outdoor data set, compared with the current most advanced unsupervised defogging method D4-Net, the PSNR of the image defogging method (TCM) provided in the embodiments of the present application is improved by 0.22 dB, and the score of SSIM also leads by 0.0043.

[0174] The image defogging method (TCM) provided in the embodiments of the present application not only compares the defogging effects of other image defogging methods on the foggy images in the data set, but also compares the defogging effects of other image defogging methods on the real scene foggy images.

[0175] Figure 11-a And Figure 11-b The image defogging method (TCM) provided in the embodiments of the present application and other image defogging methods are compared in the defogging effect of the real foggy image.

[0176] From Figure 11-a And Figure 11-b As can be seen from the above, compared with the dark channel prior defogging method (DCP) and two new methods of supervised learning based on UHD and DeHamer and some defogging methods based on unsupervised learning such as RefineDNet, YOLY and D4-Net, the image defogging method provided in the embodiments of the present application can better restore the detail information in the foggy image, and the edge contour information of the building main body in the foggy image can be clearly seen in the defogging image, greatly improving the visibility of the defogging image.

[0177] Embodiment Two

[0178] Figure 12 A schematic diagram of the image defogging device provided in the embodiments of the present application is shown.

[0179] As Figure 12 shown, the image defogging device comprises:

[0180] The acquisition unit 1201 is configured to acquire a haze image to be defogged.

[0181] The processing unit 1202 is configured to input the haze image into a defogging model, call the defogging model to perform defogging processing on the haze image, and obtain a defogging image output by the defogging model. The defogging model is obtained by modulating a target component of a sample haze image to obtain a modulation sample, and calculating a modulation-independent feature invariance loss and a modulation-dependent content consistency loss. The target component of the sample haze image includes a transmittance and a surface color of the sample haze image.

[0182] The model training unit 1203 is configured to obtain a first modulation sample by the transmittance of the sample haze image and the surface color of the sample haze image after modulation.

[0183] The model training unit 1203 is configured to obtain a second modulation sample by the surface color of the sample haze image and the transmittance of the sample haze image after modulation.

[0184] The model training unit 1203 is configured to calculate a modulation-independent feature invariance loss function by the sample haze image and the first modulation sample, and obtain a first modulation-independent feature invariance loss.

[0185] The model training unit 1203 is configured to calculate a modulation-independent feature invariance loss function by the sample haze image and the second modulation sample, and obtain a second modulation-independent feature invariance loss.

[0186] The model training unit 1203 is configured to add the first modulation-independent feature invariance loss and the second modulation-independent feature invariance loss to obtain a modulation-independent feature invariance loss.

[0187] The model training unit 1203 is configured to calculate a modulation-dependent content consistency loss function by the second modulation sample and the transmittance of the sample haze image after modulation, and obtain a first modulation-dependent content consistency loss.

[0188] The model training unit 1203 is configured to calculate a modulation-dependent content consistency loss function by the first modulation sample and the surface color of the sample haze image after modulation, and obtain a second modulation-dependent content consistency loss.

[0189] The model training unit 1203 is configured to add the first modulation-dependent content consistency loss and the second modulation-dependent content consistency loss to obtain a modulation-dependent content consistency loss.

[0190] The model training unit 1203 is configured to train an initial model by the modulation-independent feature invariance loss and the modulation-dependent content consistency loss to obtain the defogging model.

[0191] Embodiment three

[0192] Figure 13 A schematic diagram of an electronic device provided by an embodiment of the present application is shown.

[0193] ReferenceFigure 13 As shown in FIG. 1, a structural schematic diagram of an electronic device suitable for implementing the electronic device in the embodiments of the present application is shown. The electronic device in the embodiments of the present application can include, but is not limited to, a fixed terminal such as a mobile phone, a notebook computer, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a desktop computer, and the like. Figure 13 The electronic device shown is only an example and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0194] As shown in FIG. 1, a structural schematic diagram of an electronic device suitable for implementing the electronic device in the embodiments of the present application is shown. The electronic device in the embodiments of the present application can include, but is not limited to, a fixed terminal such as a mobile phone, a notebook computer, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a desktop computer, and the like. Figure 13 As shown in FIG. 1, a structural schematic diagram of an electronic device suitable for implementing the electronic device in the embodiments of the present application is shown. The electronic device in the embodiments of the present application can include, but is not limited to, a fixed terminal such as a mobile phone, a notebook computer, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a desktop computer, and the like.

[0195] Generally, the following devices can be connected to the I / O interface 1305: input devices 1306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 1307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 1308 including, for example, a memory card, a hard disk, and the like; and communication devices 1309. The communication devices 1309 can allow the electronic device to communicate with other devices wirelessly or via a wire to exchange data. Although Figure 13 An electronic device with various devices is shown, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices can be implemented or provided instead.

[0196] Embodiment Four

[0197] The embodiments of the present application provide a computer readable storage medium applied to an electronic device. The computer readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device can implement the image defogging method in embodiment one.

[0198] It should be noted that the computer-readable medium or media used to provide the storage function described herein can be external or internal to an apparatus and can comprise one or more storage mediums of any appropriate type including, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor technology, or any appropriate combination thereof. Examples of a computer-readable medium include, but are not limited to: a portable computer diskette; a hard disk; a random access memory (RAM); a read-only memory (ROM); an erasable programmable read-only memory (EPROM or Flash memory); an optical fiber; a portable compact disc read-only memory (CD-ROM); an optical storage device; a magnetic storage device; or any suitable combination of the foregoing. In the present disclosure, a computer-readable storage medium can be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium can include a computer-readable storage medium and / or a computer-readable transmission medium. A computer-readable transmission medium can include any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate or transport programming for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable medium discussed herein can receive the computer-readable program code in the form of a carrier wave. The program code can be transmitted in a computer-readable medium in the form of signals, whether or not they fall within the definition of carrier waves. Computer-readable media and storage media do not include propagated signals per se.

[0199] Although preferred embodiments of the application have been described, those skilled in the art will recognize that additional modifications and variations are possible in light of the above teachings. It is therefore intended that the appended claims cover all such modifications and changes as fall within the scope of the application.

[0200] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or arrangement, but rather are used to distinguish one element from another, and do not imply singular or plural. Also, the use of "including," "containing," or "comprising" and variations thereof throughout this disclosure is intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

Claims

1. A method of defogging an image, characterized by, The method comprises the following steps: obtain a haze image to be defogged; input the haze image into a defogging model, call the defogging model to perform defogging processing on the haze image, and obtain a defogged image output by the defogging model, wherein the defogging model is obtained by modulating a target component of a sample haze image to obtain a modulation sample, and optimizing a modulation-independent feature invariance loss and a modulation-dependent content consistency loss as a target; the target component of the sample haze image includes the transmittance and surface color of the sample haze image; the defogging model is obtained by modulating a target component of a sample haze image to obtain a modulation sample, and optimizing a modulation-independent feature invariance loss and a modulation-dependent content consistency loss as a target, comprising: obtain a first modulation sample by the transmittance of the sample haze image and the surface color of the sample haze image after modulation; obtain a second modulation sample by the surface color of the sample haze image and the transmittance of the sample haze image after modulation; calculate a modulation-independent feature invariance loss function by the sample haze image and the first modulation sample to obtain a first modulation-independent feature invariance loss; calculate a modulation-independent feature invariance loss function by the sample haze image and the second modulation sample to obtain a second modulation-independent feature invariance loss; add the first modulation-independent feature invariance loss and the second modulation-independent feature invariance loss to obtain the modulation-independent feature invariance loss; calculate a modulation-dependent content consistency loss function by the second modulation sample and the transmittance of the sample haze image after modulation to obtain a first modulation-dependent content consistency loss; calculate a modulation-dependent content consistency loss function by the first modulation sample and the surface color of the sample haze image after modulation to obtain a second modulation-dependent content consistency loss; add the first modulation-dependent content consistency loss and the second modulation-dependent content consistency loss to obtain the modulation-dependent content consistency loss; 2. The method of defogging an image according to claim 1, wherein, train an initial model by the modulation-independent feature invariance loss and the modulation-dependent content consistency loss to obtain the defogging model. The defogging model comprises a transmittance estimation sub-module, an image restoration sub-module, and an atmospheric light estimation sub-module; wherein calling the defogging model to perform defogging processing on the haze image to obtain a defogged image output by the defogging model comprises:

3. The method of defogging an image according to claim 1, wherein, calling the image restoration sub-module to process the haze image to obtain a defogged image output by the defogging model. Before training the initial model by the modulation-independent feature invariance loss and the modulation-dependent content consistency loss to obtain the defogging model, the method further comprises the following steps: calculate a reconstruction loss function based on the sample haze image and the reconstructed sample haze image to obtain a reconstruction loss; calculate an adversarial loss function based on a clear image and a defogged image corresponding to the sample haze image to obtain an adversarial loss, wherein the content of the clear image is different from the content of the defogged image corresponding to the sample haze image; calculate a reconstruction consistency loss function based on the clear image and the defogged image corresponding to the clear image, to obtain a reconstruction consistency loss; wherein the training of the initial model by the modulation-independent feature invariance loss and the modulation-dependent content consistency loss to obtain the defogging model comprises: calculate a total loss by the modulation-independent feature invariance loss, the modulation-dependent content consistency loss, the reconstruction loss, the adversarial loss, and the reconstruction consistency loss, to train the initial model to obtain the defogging model.

4. An image defogging apparatus characterized by comprising: comprise: an acquisition unit, configured to acquire a haze image to be defogged; a processing unit, configured to input the haze image into a defogging model, call the defogging model to perform defogging processing on the haze image, and obtain a defogged image output by the defogging model, the defogging model being obtained by modulating a target component of a sample haze image to obtain a modulation sample, and calculating a modulation-independent feature invariance loss and a modulation-dependent content consistency loss based on the modulation sample; the target component of the sample haze image comprising a transmittance and a surface color of the sample haze image; a model training unit, configured to obtain a first modulation sample by the transmittance of the sample haze image and the surface color of the sample haze image after modulation; obtain a second modulation sample by the surface color of the sample haze image and the transmittance of the sample haze image after modulation; calculate a modulation-independent feature invariance loss function based on the sample haze image and the first modulation sample, to obtain a first modulation-independent feature invariance loss; calculate a modulation-independent feature invariance loss function based on the sample haze image and the second modulation sample, to obtain a second modulation-independent feature invariance loss; add the first modulation-independent feature invariance loss and the second modulation-independent feature invariance loss to obtain the modulation-independent feature invariance loss; calculate a modulation-dependent content consistency loss function based on the second modulation sample and the transmittance of the sample haze image after modulation, to obtain a first modulation-dependent content consistency loss; calculate a modulation-dependent content consistency loss function based on the first modulation sample and the surface color of the sample haze image after modulation, to obtain a second modulation-dependent content consistency loss; add the first modulation-dependent content consistency loss and the second modulation-dependent content consistency loss to obtain the modulation-dependent content consistency loss; train the initial model by the modulation-independent feature invariance loss and the modulation-dependent content consistency loss to obtain the defogging model.

5. The fog removal device of the image according to claim 4, wherein The defogging model comprises a transmittance estimation sub-module, an image restoration sub-module, and an atmospheric light estimation sub-module. The processing unit calls the defogging model to perform defogging processing on the haze image, to obtain a defogged image output by the defogging model, for: calling the image restoration sub-module to process the haze image, to obtain a defogged image output by the defogging model.

6. The image defogging apparatus according to claim 4, wherein The model training unit comprises: a first modulation sample obtaining unit configured to obtain a first modulation sample from a transmittance of the sample haze image and a surface color of the sample haze image after modulation; a second modulation sample obtaining unit configured to obtain a second modulation sample from the surface color of the sample haze image and the transmittance of the sample haze image after modulation; a first loss obtaining unit configured to calculate a modulation-irrelevant feature invariance loss function from the sample haze image and the first modulation sample, and obtain a first modulation-irrelevant feature invariance loss; a second loss obtaining unit configured to calculate a modulation-irrelevant feature invariance loss function from the haze image and the second modulation sample, and obtain a second modulation-irrelevant feature invariance loss; a first loss combining unit configured to add the first modulation-irrelevant feature invariance loss and the second modulation-irrelevant feature invariance loss to obtain the modulation-irrelevant feature invariance loss; a third loss obtaining unit configured to calculate a modulation-relevant content consistency loss function from the second modulation sample and the transmittance of the sample haze image after modulation, and obtain a first modulation-relevant content consistency loss; a fourth loss obtaining unit configured to calculate a modulation-relevant content consistency loss function from the first modulation sample and the surface color of the sample haze image after modulation, and obtain a second modulation-relevant content consistency loss; a second loss combining unit configured to add the first modulation-relevant content consistency loss and the second modulation-relevant content consistency loss to obtain the relevant content consistency loss; an output unit configured to train an initialized model by using the modulation-irrelevant feature invariance loss and the modulation-relevant content consistency loss, and obtain the dehazing model.

7. An electronic device, comprising: An electronic device including at least one processor and a memory connected to the processor, wherein: the memory is configured to store computer programs or instructions; the processor is configured to execute the computer programs or instructions to enable the electronic device to implement the dehazing method of the image according to any one of claims 1 to 3.

8. A computer storage medium, characterized in that, The storage medium carries one or more computer program instructions, which, when executed by the electronic device, enable the electronic device to implement the dehazing method of the image according to any one of claims 1 to 3.

Images