Fog degradation zero sample defogging method and device based on pseudo atmospheric light image

Abstract

The invention relates to the technical field of computer vision image enhancement, and provides a fog degradation zero-sample defogging method and device based on a pseudo-atmospheric light image, which combine a target transmission image determined by a target transmission image generator and a target atmospheric light image determined by a target atmospheric light image generator and combine an atmospheric scattering model. According to the method, the to-be-defogged image is defogged, the fuzzy influence caused by haze can be quickly removed, the details and the color reality sense of the image are reserved as much as possible, and the implementation principle is simple and easy to implement. Moreover, according to the method, a target transmission image generator and a target atmospheric light image generator are obtained through a fog degradation zero sample learning method, that is, a to-be-defogged sample image and a fog degradation image are utilized in the training process, and the reference basis of zero sample learning is increased, so that the dependence on large-scale paired image data can be reduced; and the model parameters of the atmospheric scattering model can be estimated more accurately, and the defogging quality and visual effect of the zero sample learning image are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision image enhancement, and particularly to a fog degradation zero-sample defogging method and device based on a pseudo-atmospheric light image. Background Art

[0002] Haze is a special weather condition caused by micron-sized particles suspended in the atmosphere, which makes the sky hazy and reduces the clarity, contrast, and color of images. Images taken under haze conditions will significantly affect the performance of computer vision. To solve this problem, existing technologies usually adopt zero-sample defogging methods to defog images taken under haze conditions.

[0003] However, existing zero-sample defogging methods only rely on the input image itself during model training, lacking other references, and usually rely on manually designed priors to estimate the parameters of the atmospheric scattering model (i.e., the Koschmieder model), which easily leads to the output risk dominated by prior features. That is, when the prior features are estimated inaccurately, the defogging effect of the output defogged image will be biased. Summary of the Invention

[0004] The present invention provides a fog degradation zero-sample defogging method and device based on a pseudo-atmospheric light image to solve the defects existing in the prior art.

[0005] The present invention provides a fog degradation zero-sample defogging method based on a pseudo-atmospheric light image, including: Obtaining an image to be defogged; Based on a target transmission map generator, determining the target transmission map of the image to be defogged, based on a target atmospheric light map generator, determining the target atmospheric light map of the image to be defogged, and based on the target transmission map and the target atmospheric light map, combining with the atmospheric scattering model, defogging the image to be defogged; Wherein, the training steps of the target transmission map generator and the target atmospheric light map generator include: Based on an initial atmospheric light map generator, determining a first initial atmospheric light map of a defogging sample image, and based on the first initial atmospheric light map, constructing a first pseudo-atmospheric light map; Based on the first initial atmospheric light map, determining the fog-degraded image of the defogging sample image, and based on the initial atmospheric light map generator, determining a second initial atmospheric light map of the fog-degraded image, and based on the second initial atmospheric light map, constructing a second pseudo-atmospheric light map; Based on an initial transmission map generator, using the defogging sample image and the first pseudo-atmospheric light map as a basis, generating a first initial transmission map and a first pseudo-atmospheric light transmission map, and using the fog-degraded image and the second pseudo-atmospheric light map as a basis, generating a second initial transmission map and a second pseudo-atmospheric light transmission map; Based on the first initial atmospheric light map, the second initial atmospheric light map, the first initial transmission map, the first pseudo-atmospheric light transmission map, the second initial transmission map, and the second pseudo-atmospheric light transmission map, iteratively train the initial atmospheric light map generator and the initial transmission map generator to obtain the target atmospheric light map generator and the target transmission map generator.

[0006] According to a fog-degraded zero-shot defogging method based on pseudo-atmospheric light images provided by the present invention, the step of iteratively training the initial atmospheric light map generator and the initial transmission map generator based on the first initial atmospheric light map, the second initial atmospheric light map, the first initial transmission map, the first pseudo-atmospheric light transmission map, the second initial transmission map, and the second pseudo-atmospheric light transmission map to obtain the target atmospheric light map generator and the target transmission map generator includes: Calculate the atmospheric light map similarity loss based on the first initial atmospheric light map and the second initial atmospheric light map; Calculate the transmission relationship loss based on the first initial transmission map and the second initial transmission map; Based on the first pseudo-atmospheric light transmission map and the second pseudo-atmospheric light transmission map, construct a pseudo-atmospheric light loss for constraining the expected transmittance of the pseudo-atmospheric light map to be zero; Based on the atmospheric light map similarity loss, the transmission relationship loss, and the pseudo-atmospheric light loss, iteratively train the initial atmospheric light map generator and the initial transmission map generator to obtain the target atmospheric light map generator and the target transmission map generator.

[0007] According to a fog-degraded zero-shot defogging method based on pseudo-atmospheric light images provided by the present invention, the step of constructing a pseudo-atmospheric light loss for constraining the expected transmittance of the pseudo-atmospheric light map to be zero based on the first pseudo-atmospheric light transmission map and the second pseudo-atmospheric light transmission map includes: Based on the first pseudo-atmospheric light transmission map and the second pseudo-atmospheric light transmission map, establish a fused transmission map; Calculate the smooth L1 loss of the pixel values of each pixel point in the fused transmission map; Based on the mean value of the L1 losses corresponding to each pixel point in the fused transmission map, determine the pseudo-atmospheric light loss.

[0008] According to a fog-degraded zero-shot defogging method based on pseudo-atmospheric light images provided by the present invention, the step of constructing a first pseudo-atmospheric light map based on the first initial atmospheric light map includes: Determine a first copy map of the first initial atmospheric light map, and add random noise to the first copy map to obtain the first pseudo-atmospheric light map; Constructing a second pseudo-atmospheric light map based on the second initial atmospheric light map includes: Determining a second copy map of the second initial atmospheric light map, and adding random noise to the second copy map to obtain the second pseudo-atmospheric light map.

[0009] According to a zero-sample defogging method for fog degradation based on pseudo-atmospheric light images provided by the present invention, the random noise is selected from a Gaussian distribution with a mean of zero and a standard deviation of 0.01.

[0010] According to a zero-sample defogging method for fog degradation based on pseudo-atmospheric light images provided by the present invention, the target transmission map generator includes an input convolutional layer, a batch normalization (Batch Norm) layer, a leaky rectified linear unit (Leaky ReLU) layer, an encoder, a decoder, and an output convolutional layer connected in sequence; the encoder includes a plurality of downsampling layers, and the decoder includes a plurality of upsampling layers; The plurality of downsampling layers and the plurality of upsampling layers are skip-connected.

[0011] According to a zero-sample defogging method for fog degradation based on pseudo-atmospheric light images provided by the present invention, the target atmospheric light map generator includes a channel processing branch, an image processing branch, and a feature processing main path; the channel processing branch and the image processing branch are both connected to the feature processing main path; The channel processing branch is used to extract features of different channels of the image to be defogged, and determine a dark channel feature map based on the features of different channels; The image processing branch is used to extract features of the image to be defogged to obtain an image feature map; The feature processing main path is used to combine and flatten the dark channel feature map and the image feature map to obtain the target atmospheric light map.

[0012] The present invention also provides a zero-sample defogging device for fog degradation based on pseudo-atmospheric light images, including: An image acquisition module, configured to acquire an image to be defogged; An image defogging module, configured to determine a target transmission map of the image to be defogged based on a target transmission map generator, determine a target atmospheric light map of the image to be defogged based on a target atmospheric light map generator, and defog the image to be defogged by combining an atmospheric scattering model based on the target transmission map and the target atmospheric light map; A generator training module, configured to execute the training steps of the target transmission map generator and the target atmospheric light map generator: Determining a first initial atmospheric light map of a sample image to be defogged based on an initial atmospheric light map generator, and constructing a first pseudo-atmospheric light map based on the first initial atmospheric light map; Based on the first initial atmospheric light map, determine the fog-degraded image of the to-be-dehazed sample image, and based on the initial atmospheric light map generator, determine the second initial atmospheric light map of the fog-degraded image. Based on the second initial atmospheric light map, construct a second pseudo-atmospheric light map; Based on the initial transmission map generator, using the to-be-dehazed sample image and the first pseudo-atmospheric light map as a basis, generate a first initial transmission map and a first pseudo-atmospheric light transmission map, and using the fog-degraded image and the second pseudo-atmospheric light map, generate a second initial transmission map and a second pseudo-atmospheric light transmission map; Based on the first initial atmospheric light map, the second initial atmospheric light map, the first initial transmission map, the first pseudo-atmospheric light transmission map, the second initial transmission map, and the second pseudo-atmospheric light transmission map, perform iterative training on the initial atmospheric light map generator and the initial transmission map generator to obtain the target atmospheric light map generator and the target transmission map generator.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the fog-degraded zero-sample dehazing method based on a pseudo-atmospheric light image as described in any one of the above.

[0014] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the fog-degraded zero-sample dehazing method based on a pseudo-atmospheric light image as described in any one of the above.

[0015] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the fog-degraded zero-sample dehazing method based on a pseudo-atmospheric light image as described in any one of the above.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The fog degradation zero - sample defogging method and device based on pseudo - atmospheric light images provided by the present invention combine the target transmission map determined by the target transmission map generator and the target atmospheric light map determined by the target atmospheric light map generator, and combine the atmospheric scattering model to defog the image to be defogged. It can quickly remove the blur caused by haze, while trying to retain the details and color realism of the image, and the implementation principle is simple and feasible. Moreover, this method obtains the target transmission map generator and the target atmospheric light map generator through the fog degradation zero - sample learning method, that is, in the training process, not only the sample images to be defogged are used, but also the fog - degraded images are used, increasing the reference basis for zero - sample learning. It can not only reduce the dependence on large - scale paired image data, but also more accurately estimate the model parameters of the atmospheric scattering model, significantly improving the quality and visual effect of zero - sample learning image defogging, providing more reliable data support for industrial production and manufacturing, and improving the automation level. In addition, by introducing the first pseudo - atmospheric light map and the second pseudo - atmospheric light map when generating the first initial transmission map and the second initial transmission map respectively, self - supervised learning of the initial transmission map generator can be realized, enhancing the adaptability and generalization ability of the target transmission map generator to different haze degrees, thereby improving the robustness of the defogging effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0018] Figure 1 is one of the schematic flowcharts of the fog degradation zero - sample defogging method based on pseudo - atmospheric light images provided by the present invention; Figure 2 is the schematic diagram of the training steps of the target transmission map generator and the target atmospheric light map generator in the fog degradation zero - sample defogging method based on pseudo - atmospheric light images provided by the present invention; Figure 3 is the schematic diagram of the training structure of the target transmission map generator and the target atmospheric light map generator in the fog degradation zero - sample defogging method based on pseudo - atmospheric light images provided by the present invention; Figure 4 is the schematic diagram of the structure of the initial transmission map generator and the target transmission map generator in the fog degradation zero - sample defogging method based on pseudo - atmospheric light images provided by the present invention; Figure 5 is the schematic diagram of the structure of the initial atmospheric light map generator and the target atmospheric light map generator in the fog degradation zero - sample defogging method based on pseudo - atmospheric light images provided by the present invention; Figure 6It is a schematic structural diagram of a fog degradation zero - sample defogging device based on a pseudo - atmospheric light image provided by the present invention; Figure 7 It is a schematic structural diagram of an electronic device provided by the present invention. Detailed implementation manners

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts shall fall within the protection scope of the present invention.

[0020] Since the existing zero - sample defogging methods only rely on the input image itself during model training and lack other references, and when the prior feature estimation is inaccurate, it will lead to deviations in the defogging effect of the output defogged image. Based on this, an embodiment of the present invention provides a fog degradation zero - sample defogging method based on a pseudo - atmospheric light image.

[0021] Figure 1 It is a schematic flowchart of a fog degradation zero - sample defogging method based on a pseudo - atmospheric light image provided in an embodiment of the present invention. As Figure 1 shown, the method includes: S11, obtaining the image to be defogged; S12, based on the target transmission map generator, determining the target transmission map of the image to be defogged, based on the target atmospheric light map generator, determining the target atmospheric light map of the image to be defogged, and based on the target transmission map and the target atmospheric light map, combining with the atmospheric scattering model, defogging the image to be defogged; Among them, as Figure 2 shown, the training steps of the target transmission map generator and the target atmospheric light map generator include: S21, based on the initial atmospheric light map generator, determining the first initial atmospheric light map of the defogging sample image, and based on the first initial atmospheric light map, constructing the first pseudo - atmospheric light map; S22, based on the first initial atmospheric light map, determining the fog - degraded image of the defogging sample image, and based on the initial atmospheric light map generator, determining the second initial atmospheric light map of the fog - degraded image, and based on the second initial atmospheric light map, constructing the second pseudo - atmospheric light map; S23, based on the initial transmission map generator, using the defogging sample image and the first pseudo - atmospheric light map as the basis, generating the first initial transmission map and the first pseudo - atmospheric light transmission map, and using the fog - degraded image and the second pseudo - atmospheric light map to generate the second initial transmission map and the second pseudo - atmospheric light transmission map; S24. Based on the first initial atmospheric light map, the second initial atmospheric light map, the first initial transmission map, the first pseudo-atmospheric light transmission map, the second initial transmission map, and the second pseudo-atmospheric light transmission map, iteratively train the initial atmospheric light map generator and the initial transmission map generator to obtain the target atmospheric light map generator and the target transmission map generator.

[0022] Specifically, in the embodiment of the present invention, the zero-shot defogging method based on pseudo-atmospheric light images is executed by a zero-shot defogging device based on pseudo-atmospheric light images. This device can be configured in an electronic device, which can be a computer or an image acquisition device. The computer can be a local computer or a cloud computer. The local computer can be a computer, a tablet, etc., and no specific limitation is made here. The image acquisition device can be a camera, which can have the function of zero-shot defogging based on pseudo-atmospheric light images.

[0023] First, execute step S11 to obtain the image to be defogged. The image to be defogged refers to a foggy image with a defogging requirement. The image to be defogged is a color image and can have 3 channels.

[0024] Then, execute step S12 to introduce a target transmission map generator and a target atmospheric light map generator. Both the target transmission map generator and the target atmospheric light map generator can be generators obtained through training.

[0025] Input the image to be defogged into the target transmission map generator, and the target transmission map output by the target transmission map generator can be obtained. The pixel value of each pixel point in the target transmission map is the transmittance of the corresponding pixel point in the image to be defogged.

[0026] Input the image to be defogged into the target atmospheric light map generator, and the target atmospheric light map output by the target atmospheric light map generator can be obtained. The pixel value of each pixel point in the target atmospheric light map is the atmospheric light value of the corresponding pixel point in the image to be defogged.

[0027] In the embodiment of the present invention, both the target transmission map and the target atmospheric light map are used as model parameters of the atmospheric scattering model. By using the target transmission map and the target atmospheric light map and combining the atmospheric scattering model, defogging is performed on the image to be defogged, and a clear image can be obtained.

[0028] Among them, the atmospheric scattering model can be the Koschmieder model, which is a physical model used to describe atmospheric visibility. By analyzing the characteristics of light scattering and absorption in the atmosphere, it explains the relationship between visibility and suspended particles in the air (such as dust, haze, etc.).

[0029] The atmospheric scattering model can be expressed by the following formula: ; Among them, I is the image to be dehazed, and J is the clear image. is the target transmission map, and is the target atmospheric light map.

[0030] The training structures of the target transmission map generator and the target atmospheric light map generator are as Figure 3 shown. The above step S22 is executed after step S21. The execution order of step S23 and step S21 can be set as needed. It can either execute step S21 first and then step S23, or execute step S23 first and then step S21, or execute step S21 and step S23 simultaneously. No specific limitation is made here.

[0031] In step S21, the hazy sample image I1 is input into the initial atmospheric light map generator A-Net, and the first initial atmospheric light map output by the initial atmospheric light map generator can be obtained . Using the first initial atmospheric light map , by introducing random noise, the first pseudo-atmospheric light map PALI1 can be constructed.

[0032] In step S22, using the first initial atmospheric light map and combining it with the hazy sample image I1, the fog-degraded image I2 of the hazy sample image I1 can be determined.

[0033] Here, if the original clear sample image is J1, using the atmospheric scattering model, we can have: ; Among them, is the transmission map corresponding to the hazy sample image I1, is the atmospheric light map corresponding to the hazy sample image I1.

[0034] Using the atmospheric scattering model to degrade the hazy sample image I1, the fog-degraded image I2 can be generated: ; Among them, is the transmission map corresponding to the fog-degraded image I2, is the atmospheric light map corresponding to the fog-degraded image I2.

[0035] Substituting the formula of into the formula of , we can get: ; Since the atmospheric light intensities are approximately equal, that is, , then we can further get: .

[0036] It can be seen from this that generating a fog-degraded image I2 from the foggy sample image I1 to be dehazed is equivalent to generating a fog-degraded image I2 from the undamaged original clear sample image J1 through perturbation for equivalence. Constraining the degradation of the foggy sample image I1 using the atmospheric scattering model is equivalent to the controlled perturbation of the model parameters. Furthermore, can be regarded as a perturbation factor α, and the relationship between the fog-degraded image I2 and the foggy sample image I1 can be expressed as: .

[0037] When determining the fog-degraded image I2, substitute the first initial atmospheric light map into the above formula, that is: , and set , which can be set as needed. For example, it can be set to or other values.

[0038] After that, input the fog-degraded image I2 into the initial atmospheric light map generator A-Net, and the second initial atmospheric light map of the fog-degraded image I2 output by the initial atmospheric light map generator A-Net can be obtained. Using the second initial atmospheric light map , a second pseudo-atmospheric light map PALI2 can be constructed by introducing random noise.

[0039] In step S23, input the foggy sample image I1 and the first pseudo-atmospheric light map PALI1 into the initial transmission map generator T-Net together, and the first initial transmission map output by the initial transmission map generator T-Net and the first pseudo-atmospheric light transmission map can be obtained. Input the fog-degraded image I2 and the second pseudo-atmospheric light map PALI2 into the initial transmission map generator T-Net together, and the second initial transmission map output by the initial transmission map generator T-Net and the second pseudo-atmospheric light transmission map can be obtained.

[0040] Finally, execute step S24. Combine the first initial atmospheric light map , the second initial atmospheric light map , the first initial transmission map , the first pseudo-atmospheric light transmission map , the second initial transmission map and the second pseudo-atmospheric light transmission map , and calculate the training loss. The training loss can include at least one of the atmospheric light map similarity loss, the transmission relationship loss, and the pseudo-atmospheric light loss that constrains the expected transmittance of the pseudo-atmospheric light map to be zero.

[0041] Using the training loss, the initial atmospheric light map generator A-Net and the initial transmission map generator T-Net can be iteratively trained. When the preset number of iterations is reached or the training loss converges, the target atmospheric light map generator and the target transmission map generator can be obtained. Among them, the preset number of iterations can be set as needed and is not specifically limited here.

[0042] In the haze degradation zero-shot defogging method based on the pseudo-atmospheric light image provided in the embodiment of the present invention, by combining the target transmission map determined by the target transmission map generator and the target atmospheric light map determined by the target atmospheric light map generator, and combining the atmospheric scattering model, the image to be defogged is defogged. It can quickly remove the blurring effect caused by haze, and at the same time try to retain the details and color realism of the image, and the implementation principle is simple and feasible. Moreover, this method obtains the target transmission map generator and the target atmospheric light map generator through the haze degradation zero-shot learning method, that is, in the training process, not only the image samples to be defogged are used, but also the haze-degraded images are used, increasing the reference basis for zero-shot learning. It can not only reduce the dependence on large-scale paired image data, but also more accurately estimate the model parameters of the atmospheric scattering model, significantly improving the quality and visual effect of zero-shot learning image defogging, providing more reliable data support for industrial production and manufacturing, and improving the automation level. In addition, by introducing the first pseudo-atmospheric light map and the second pseudo-atmospheric light map when generating the first initial transmission map and the second initial transmission map respectively, self-supervised learning of the initial transmission map generator can be realized, enhancing the adaptability and generalization ability of the target transmission map generator to different haze degrees, thereby improving the robustness of the defogging effect.

[0043] Based on the above embodiments, as Figure 3 shown, the iterative training of the initial atmospheric light map generator and the initial transmission map generator based on the first initial atmospheric light map, the second initial atmospheric light map, the first initial transmission map, the first pseudo-atmospheric light transmission map, the second initial transmission map, and the second pseudo-atmospheric light transmission map to obtain the target atmospheric light map generator and the target transmission map generator includes: Calculating the atmospheric light map similarity loss based on the first initial atmospheric light map and the second initial atmospheric light map; Calculating the transmission relationship loss based on the first initial transmission map and the second initial transmission map; Based on the first pseudo-atmospheric light transmission map and the second pseudo-atmospheric light transmission map, constructing a pseudo-atmospheric light loss for constraining the expected transmittance of the pseudo-atmospheric light map to be zero; Based on the atmospheric light map similarity loss, the transmission relationship loss, and the pseudo-atmospheric light loss, the initial atmospheric light map generator and the initial transmission map generator are iteratively trained to obtain the target atmospheric light map generator and the target transmission map generator.

[0044] Specifically, during the iterative training of the initial atmospheric light map generator A-Net and the initial transmission map generator T-Net, the first initial atmospheric light map and the second initial atmospheric light map can be used to calculate the atmospheric light map similarity loss to constrain the consistency between the first initial atmospheric light map and the second initial atmospheric light map .

[0045] The atmospheric light map similarity loss can be calculated by the following formula: ; where ||·||2 is the L2 norm.

[0046] Using the first initial transmission map and the second initial transmission map , the transmission relationship loss is calculated to constrain the consistency between the first initial transmission map and the second initial transmission map .

[0047] The transmission relationship loss can be calculated by the following formula: ; where x is a pixel point, is the pixel value of pixel point x in the first initial transmission map, and is the pixel value of pixel point x in the second initial transmission map

[0048] Using the first pseudo-atmospheric light transmission map and the second pseudo-atmospheric light transmission map , combined with the assumption that the expected transmittance of the pseudo-atmospheric light map is zero, a pseudo-atmospheric light loss for constraining the expected transmittance of the pseudo-atmospheric light map to be zero can be constructed.

[0049] Finally, by performing a weighted sum of the atmospheric light map similarity loss , the transmission relationship loss , and the pseudo-atmospheric light loss , the training loss can be obtained. That is: ; Among them, is the training loss, , , are the weights of the atmospheric light map similarity loss , the transmission relationship loss and the pseudo-atmospheric light loss respectively, which can be set as needed and are not specifically limited here.

[0050] Using the training loss, the initial atmospheric light map generator and the initial transmission map generator can be iteratively trained to obtain the target atmospheric light map generator and the target transmission map generator.

[0051] In the embodiments of the present invention, the atmospheric light map similarity loss, the transmission relationship loss and the pseudo-atmospheric light loss are considered simultaneously, which can ensure the transmission Figure 1 consistency and the atmospheric light Figure 1 consistency, and constrain the expected transmittance of the pseudo-atmospheric light map to be zero, which can enable the target atmospheric light map generator and the target transmission map generator to have the function of retaining image texture and color details, so that the dehazed clear image is not distorted, providing higher accuracy and robustness for subsequent work.

[0052] Based on the above embodiments, constructing a pseudo-atmospheric light loss for constraining the expected transmittance of the pseudo-atmospheric light map to be zero based on the first pseudo-atmospheric light transmission map and the second pseudo-atmospheric light transmission map includes: Based on the first pseudo-atmospheric light transmission map and the second pseudo-atmospheric light transmission map, establish a fused transmission map; Calculate the smooth L1 loss of the pixel values of each pixel point in the fused transmission map; Based on the mean value of the L1 losses corresponding to each pixel point in the fused transmission map, determine the pseudo-atmospheric light loss.

[0053] Specifically, using the first pseudo-atmospheric light transmission map and the second pseudo-atmospheric light transmission map , a fused transmission map can be established, and this fused transmission map can be obtained by weighted summing the first pseudo-atmospheric light transmission map and the second pseudo-atmospheric light transmission map , and the weights during their weighted summing can be set as needed. In order to make the first pseudo-atmospheric light transmission map and the second pseudo-atmospheric light transmission map contribute equally when calculating the loss, the fused transmission map is expressed as: .

[0054] Therefore, the pseudo-atmospheric light loss The formula can be: ; where N is the number of pixel points in the fusion transmission map, x is the pixel point, is the pixel value of the pixel point x in the fusion transmission map, is the smooth L1 loss.

[0055] It can be understood that the pseudo-atmospheric light loss is actually and the expected transmittance of the difference.

[0056] When the independent variable is the error e, it can be calculated by the following formula:

[0057] In the embodiments of the present invention, the smooth L1 loss can be equivalent to combining the L1 norm and the L2 norm, which can reduce the influence of small errors and have a strong penalty for large errors. When the error is less than 1, the quadratic term is used, which is similar to the processing method of the L2 norm; when is greater than or equal to 1, the linear term, that is, is used, which is similar to the processing method of the L1 norm.

[0058] Based on the assumption that the expected transmittance of the pseudo-atmospheric light map is zero, through the pseudo-atmospheric light loss the initial transmission map generator T-Net is self-supervised trained to generate a transmission map corresponding to the pseudo-atmospheric light map, so that the transmission map is encouraged to be a transmission map with an expected transmittance close to zero. Furthermore, the target transmission map generator can assign a smaller transmission value to the foreground object in the defogging image that is similar in color to the target atmospheric light map.

[0059] It should be noted that since the pseudo-atmospheric light map has no structured textures such as edges or corners and its pixel distribution is relatively uniform, training the initial transmission map generator T-Net can avoid misjudging the structured pixel region that is similar in color to the atmospheric light map but has textures as the far region. Here, the far region refers to the darker region in the transmission map.

[0060] On the basis of the above embodiments, constructing the first pseudo-atmospheric light map based on the first initial atmospheric light map includes: Determining a first copy map of the first initial atmospheric light map and adding random noise to the first copy map to obtain the first pseudo-atmospheric light map; Constructing the second pseudo-atmospheric light map based on the second initial atmospheric light map includes: Determine a second copy of the second initial atmospheric light map, and add random noise to the second copy to obtain the second pseudo-atmospheric light map.

[0061] Specifically, when constructing the first pseudo-atmospheric light map, the first initial atmospheric light map can be determined first, and the pixel values of each pixel point in the first copy H1 of the first initial atmospheric light map are exactly the same as those of each pixel point in the first initial atmospheric light map . By adding random noise to the first copy H1, the first pseudo-atmospheric light map can be obtained.

[0062] Similarly, when constructing the second pseudo-atmospheric light map, the second initial atmospheric light map can be determined first, and the pixel values of each pixel point in the second copy H2 of the first initial atmospheric light map are exactly the same as those of each pixel point in the second initial atmospheric light map . By adding random noise to the second copy H2, the second pseudo-atmospheric light map can be obtained.

[0063] In the embodiments of the present invention, by adding random noise, the first pseudo-atmospheric light map and the second pseudo-atmospheric light map can be quickly obtained, thereby improving the training efficiency.

[0064] Based on the above embodiments, the random noise can be selected from a Gaussian distribution with a mean of zero and a standard deviation of 0.01. This Gaussian distribution is the normal distribution.

[0065] It can be understood that the random noise selected when constructing the first pseudo-atmospheric light map and the second pseudo-atmospheric light map can be the same or different, and no specific limitation is made here.

[0066] Based on the above embodiments, as Figure 4 shown, the structures of the initial transmission map generator and the target transmission map generator can both include an input convolution layer, a batch normalization layer, an activation layer, an encoder, a decoder, and an output convolution layer that are connected in sequence. The encoder includes multiple downsampling (Down Sample) layers, and the decoder includes multiple upsampling (Up Sample) layers. Figure 4 Taking the example of including four downsampling layers connected in sequence and four upsampling layers connected in sequence for illustration.

[0067] The multiple downsampling layers and the multiple upsampling layers are skip-connected, that is, Figure 4 in the first downsampling layer is connected to the fourth upsampling layer, the second downsampling layer is connected to the third upsampling layer, the third downsampling layer is connected to the second upsampling layer, and the fourth downsampling layer is connected to the first upsampling layer. Among them, the input convolution layer can be a 3×3 convolution layer, and the output convolution layer can be a 1×1 convolution layer.

[0068] Assume that the input of the input convolutional layer is an image with a size of 256×256×3. First, it passes through a 3×3 convolutional layer. Without changing the image size, the number of channels is changed to 64. After passing through the batch normalization layer and the activation layer, it is input to the encoder for downsampling. Here, the downsampling is achieved through the pooling layer. Each downsampling operation will halve the size of the image, reduce the resolution of the image, and extract the deep features of the image. After the encoder extracts the features, the features are input to the decoder for upsampling. On the contrary, after each upsampling operation, the size of the image will double. The upsampling uses transposed convolution to increase the resolution of the image and recover the image from the deep features.

[0069] The skip connection is to combine the deep features and the shallow features in the channel dimension to make up for the loss of the detailed information of the image. The skip arrow in the figure refers to combining the input of the downsampling layer with the output of the upsampling layer. For example, 32×32×512 is the input size of the fourth downsampling layer, and the output size after downsampling is 16×16×512. This output is used as the input of the first upsampling layer. After upsampling, the output size is 32×32×512. Therefore, after the skip connection, the obtained feature size is 32×32×1024. After passing through the convolutional layer to reduce the number of channels, the final output feature size is 32×32×256. This is a complete encoding and decoding process of the skip connection. The encoding and decoding processes of other skip connections are based on the same principle. Finally, through a 1×1 convolutional layer, the image is adjusted to 3 channels to obtain the output transmission map.

[0070] Based on the above embodiments, as Figure 5 shown, the structures of the initial atmospheric light map generator and the target atmospheric light map generator can both include three channel processing branches, one image processing branch, and one feature processing main path. The three channel processing branches correspond to the red channel (R), the green channel (G), and the blue channel (B) respectively.

[0071] Both the channel processing branch and the image processing branch are connected to the feature processing main path.

[0072] The channel processing branch can include a 9*9 convolutional layer (Conv 9*9), a 3*3 convolutional layer (Conv 3*3), and a splicing layer (C). The feature maps are extracted through Conv 9*9 and Conv 3*3, and a dimension is added through the splicing layer to combine the three feature maps. The minimum value is extracted in the newly added dimension to obtain the dark channel feature map (DCF).

[0073] The image processing branch can also include Conv 9*9 and Conv 3*3 to extract the image feature maps from the image to be dehazed.

[0074] The feature processing main path includes a multiplication layer , a Conv 3*3, a flattening layer (Flatten), and a linear layer (Linear). Through the flattening layer, an image feature map with 64 channels can be obtained, and through the linear layer, an atmospheric light map with 3 channels can be obtained.

[0075] As Figure 6 shown, based on the above embodiments, an apparatus for zero-shot defogging of fog degradation based on a pseudo-atmospheric light image is provided in an embodiment of the present invention, including: An image acquisition module 61, configured to acquire an image to be defogged; An image defogging module 62, configured to determine a target transmission map of the image to be defogged based on a target transmission map generator, determine a target atmospheric light map of the image to be defogged based on a target atmospheric light map generator, and defog the image to be defogged based on the target transmission map and the target atmospheric light map in combination with an atmospheric scattering model; A generator training module 63, configured to execute training steps of the target transmission map generator and the target atmospheric light map generator: Determine a first initial atmospheric light map of a sample image to be defogged based on an initial atmospheric light map generator, and construct a first pseudo-atmospheric light map based on the first initial atmospheric light map; Determine a fog-degraded image of the sample image to be defogged based on the first initial atmospheric light map, determine a second initial atmospheric light map of the fog-degraded image based on the initial atmospheric light map generator, and construct a second pseudo-atmospheric light map based on the second initial atmospheric light map; Based on an initial transmission map generator, generate a first initial transmission map and a first pseudo-atmospheric light transmission map based on the sample image to be defogged and the first pseudo-atmospheric light map, and generate a second initial transmission map and a second pseudo-atmospheric light transmission map based on the fog-degraded image and the second pseudo-atmospheric light map; Iteratively train the initial atmospheric light map generator and the initial transmission map generator based on the first initial atmospheric light map, the second initial atmospheric light map, the first initial transmission map, the first pseudo-atmospheric light transmission map, the second initial transmission map, and the second pseudo-atmospheric light transmission map to obtain the target atmospheric light map generator and the target transmission map generator.

[0076] Based on the above embodiments, in the apparatus for zero-shot defogging of fog degradation based on a pseudo-atmospheric light image provided in an embodiment of the present invention, the generator training module is specifically configured to: Calculate an atmospheric light map similarity loss based on the first initial atmospheric light map and the second initial atmospheric light map; Calculate a transmission relationship loss based on the first initial transmission map and the second initial transmission map; Construct a pseudo-atmospheric light loss for constraining the expected transmittance of the pseudo-atmospheric light map to be zero based on the first pseudo-atmospheric light transmission map and the second pseudo-atmospheric light transmission map; Iteratively train the initial atmospheric light map generator and the initial transmission map generator based on the atmospheric light map similarity loss, the transmission relationship loss, and the pseudo-atmospheric light loss to obtain the target atmospheric light map generator and the target transmission map generator.

[0077] Based on the above embodiments, in the fog degradation zero-shot defogging device based on pseudo-atmospheric light images provided in the embodiments of the present invention, the generator training module is specifically configured to: Establish a fused transmission map based on the first pseudo-atmospheric light transmission map and the second pseudo-atmospheric light transmission map; Calculate the smooth L1 loss of the pixel values of each pixel point in the fused transmission map; Determine the pseudo-atmospheric light loss based on the mean value of the L1 losses corresponding to each pixel point in the fused transmission map.

[0078] Based on the above embodiments, in the fog degradation zero-shot defogging device based on pseudo-atmospheric light images provided in the embodiments of the present invention, the generator training module is specifically configured to: Determine a first copy of the first initial atmospheric light map, and add random noise to the first copy to obtain the first pseudo-atmospheric light map; The constructing the second pseudo-atmospheric light map based on the second initial atmospheric light map includes: Determine a second copy of the second initial atmospheric light map, and add random noise to the second copy to obtain the second pseudo-atmospheric light map.

[0079] Based on the above embodiments, in the fog degradation zero-shot defogging device based on pseudo-atmospheric light images provided in the embodiments of the present invention, the random noise is selected from a Gaussian distribution with a mean of zero and a standard deviation of 0.01.

[0080] Based on the above embodiments, in the fog degradation zero-shot defogging device based on pseudo-atmospheric light images provided in the embodiments of the present invention, the target transmission map generator includes an input convolutional layer, a batch normalization (Batch Norm) layer, an activation (Leaky ReLU) layer, an encoder, a decoder, and an output convolutional layer connected in sequence; the encoder includes a plurality of downsampling layers, and the decoder includes a plurality of upsampling layers; The plurality of downsampling layers and the plurality of upsampling layers are skip-connected.

[0081] Based on the above embodiments, in the fog degradation zero-shot defogging device based on pseudo-atmospheric light images provided in the embodiments of the present invention, the target atmospheric light map generator includes a channel processing branch, an image processing branch, and a feature processing main path; the channel processing branch and the image processing branch are both connected to the feature processing main path; The channel processing branch is used to extract features from different channels of the image to be defogged, and determine a dark channel feature map based on the features of different channels; The image processing branch is used to extract features from the image to be defogged to obtain an image feature map; The feature processing main path is used to combine and flatten the dark channel feature map and the image feature map to obtain the target atmospheric light map.

[0082] Specifically, the functions of each module in the fog degradation zero-shot defogging device based on pseudo-atmospheric light images provided in the embodiments of the present invention correspond one-to-one to the operation processes of each step in the above method embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and the embodiments of the present invention will not be elaborated herein.

[0083] Figure 7 An example of a schematic physical structure diagram of an electronic device is shown as Figure 7 shown. The electronic device may include: a processor (Processor) 810, a communication interface (Communications Interface) 820, a memory (Memory) 830, and a communication bus 840. Among them, the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logical instructions in the memory 830 to execute the fog degradation zero-shot defogging method based on pseudo-atmospheric light images provided in each of the above embodiments.

[0084] In addition, when the above logical instructions in the memory 830 are implemented in the form of a software functional unit and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes.

[0085] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the fog degradation zero-sample defogging method based on the pseudo-atmospheric light image provided in the above-mentioned various embodiments.

[0086] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the fog degradation zero-sample defogging method based on the pseudo-atmospheric light image provided in the above-mentioned various embodiments.

[0087] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0088] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or equivalently replace some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A zero-sample defogging method for fog degradation based on pseudo-atmospheric light images, characterized in that, Including: Obtain the image to be dehazed; Based on a target transmission map generator, determine the target transmission map of the image to be dehazed, based on a target atmospheric light map generator, determine the target atmospheric light map of the image to be dehazed, and based on the target transmission map and the target atmospheric light map, combine with the atmospheric scattering model to dehaze the image to be dehazed; Among them, the training steps of the target transmission map generator and the target atmospheric light map generator include: Based on an initial atmospheric light map generator, determine the first initial atmospheric light map of the dehazing sample image, and based on the first initial atmospheric light map, construct a first pseudo-atmospheric light map; Based on the first initial atmospheric light map, determine the fog-degraded image of the dehazing sample image, and based on the initial atmospheric light map generator, determine the second initial atmospheric light map of the fog-degraded image, and based on the second initial atmospheric light map, construct a second pseudo-atmospheric light map; Based on an initial transmission map generator, using the dehazing sample image and the first pseudo-atmospheric light map as a basis, generate a first initial transmission map and a first pseudo-atmospheric light transmission map, and using the fog-degraded image and the second pseudo-atmospheric light map as a basis, generate a second initial transmission map and a second pseudo-atmospheric light transmission map; Based on the first initial atmospheric light map, the second initial atmospheric light map, the first initial transmission map, the first pseudo-atmospheric light transmission map, the second initial transmission map, and the second pseudo-atmospheric light transmission map, perform iterative training on the initial atmospheric light map generator and the initial transmission map generator to obtain the target atmospheric light map generator and the target transmission map generator.

2. The zero-sample defogging method for fog degradation based on pseudo-atmospheric light images according to claim 1, wherein The performing iterative training on the initial atmospheric light map generator and the initial transmission map generator based on the first initial atmospheric light map, the second initial atmospheric light map, the first initial transmission map, the first pseudo-atmospheric light transmission map, the second initial transmission map, and the second pseudo-atmospheric light transmission map to obtain the target atmospheric light map generator and the target transmission map generator includes: Based on the first initial atmospheric light map and the second initial atmospheric light map, calculate the atmospheric light map similarity loss; Based on the first initial transmission map and the second initial transmission map, calculate the transmission relationship loss; Based on the first pseudo-atmospheric light transmission map and the second pseudo-atmospheric light transmission map, construct a pseudo-atmospheric light loss that constrains the expected transmittance of the pseudo-atmospheric light map to be zero; Based on the atmospheric light map similarity loss, the transmission relationship loss, and the pseudo-atmospheric light loss, perform iterative training on the initial atmospheric light map generator and the initial transmission map generator to obtain the target atmospheric light map generator and the target transmission map generator.

3. The zero-sample defogging method for fog degradation based on pseudo-atmospheric light images according to claim 2, wherein The constructing a pseudo-atmospheric light loss that constrains the expected transmittance of the pseudo-atmospheric light map to be zero based on the first pseudo-atmospheric light transmission map and the second pseudo-atmospheric light transmission map includes: Based on the first pseudo-atmospheric light transmission map and the second pseudo-atmospheric light transmission map, establish a fused transmission map; Calculate the smooth L1 loss of the pixel values of each pixel point in the fused transmission map; Based on the mean value of the L1 losses corresponding to each pixel point in the fused transmission map, determine the pseudo-atmospheric light loss.

4. The zero-shot defogging method for fog degradation based on pseudo-atmospheric light images according to any one of claims 1-3, characterized in that Constructing a first pseudo-atmospheric light map based on the first initial atmospheric light map includes: Determining a first copy of the first initial atmospheric light map and adding random noise to the first copy to obtain the first pseudo-atmospheric light map; Constructing a second pseudo-atmospheric light map based on the second initial atmospheric light map includes: Determining a second copy of the second initial atmospheric light map and adding random noise to the second copy to obtain the second pseudo-atmospheric light map.

5. The zero-sample defogging method for fog degradation based on a pseudo-atmospheric light image according to claim 4, characterized in that The random noise is selected from a Gaussian distribution with a mean of zero and a standard deviation of 0.

01.

6. The zero-sample defogging method for fog degradation based on a pseudo-atmospheric light image according to any one of claims 1 to 3, characterized in that, The target transmission map generator includes an input convolutional layer, a batch normalization (Batch Norm) layer, a leaky rectified linear unit (Leaky ReLU) layer, an encoder, a decoder, and an output convolutional layer connected in sequence; the encoder includes a plurality of downsampling layers, and the decoder includes a plurality of upsampling layers; The plurality of downsampling layers and the plurality of upsampling layers are skip-connected.

7. The zero-sample defogging method for fog degradation based on pseudo-atmospheric light images according to any one of claims 1-3, characterized in that, The target atmospheric light map generator includes a channel processing branch, an image processing branch, and a feature processing main path; Both the channel processing branch and the image processing branch are connected to the feature processing main path; The channel processing branch is used to extract features from different channels of the foggy image to be dehazed and determine a dark channel feature map based on the features of different channels; The image processing branch is used to extract features from the foggy image to be dehazed to obtain an image feature map; The feature processing main path is used to combine and flatten the dark channel feature map and the image feature map to obtain the target atmospheric light map.

8. A fog degradation zero - sample defogging device based on a pseudo - atmospheric light image, characterized in that, Including: An image acquisition module for acquiring a foggy image to be dehazed; An image dehazing module for determining a target transmission map of the foggy image to be dehazed based on the target transmission map generator, determining a target atmospheric light map of the foggy image to be dehazed based on the target atmospheric light map generator, and dehazing the foggy image to be dehazed by combining the atmospheric scattering model based on the target transmission map and the target atmospheric light map; A generator training module for performing the training steps of the target transmission map generator and the target atmospheric light map generator: Determining a first initial atmospheric light map of a foggy sample image based on an initial atmospheric light map generator and constructing a first pseudo-atmospheric light map based on the first initial atmospheric light map; Determining a fog-degraded image of the foggy sample image based on the first initial atmospheric light map, determining a second initial atmospheric light map of the fog-degraded image based on the initial atmospheric light map generator, and constructing a second pseudo-atmospheric light map based on the second initial atmospheric light map; Based on an initial transmission map generator, generating a first initial transmission map and a first pseudo-atmospheric light transmission map based on the foggy sample image and the first pseudo-atmospheric light map, and generating a second initial transmission map and a second pseudo-atmospheric light transmission map based on the fog-degraded image and the second pseudo-atmospheric light map; Based on the first initial atmospheric light map, the second initial atmospheric light map, the first initial transmission map, the first pseudo-atmospheric light transmission map, the second initial transmission map, and the second pseudo-atmospheric light transmission map, iteratively train the initial atmospheric light map generator and the initial transmission map generator to obtain the target atmospheric light map generator and the target transmission map generator.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the fog degradation zero-shot defogging method based on pseudo-atmospheric light images according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium, having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the fog degradation zero-shot defogging method based on pseudo-atmospheric light images according to any one of claims 1-7.

Images