Single image defogging method and device based on image prior and global block aggregation

Abstract

The present invention discloses a single image defogging method and device based on image prior and global block aggregation. The method includes: using multiple pre-established feature defoggers to convert a foggy image into enhanced defogging features at multiple scales; a feature defogger includes two parallel branches, in branch one, global clear features are aggregated according to image self-similarity to obtain a global defogging feature; in branch two, feature extraction provides more local features for the global defogging feature; the features output by the two branches are fused to obtain an enhanced defogging feature at one scale, and the global defogging feature and the local detail feature are added element by element during the fusion process; finally, the enhanced defogging features at multiple scales are input into a fog-free image decoder to recover a clear image after defogging. The present invention can effectively improve the defogging performance in complex scenes. In particular, while reducing the generation of visual artifacts, the image details are more accurately restored to generate a more realistic clear image.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a single image defogging method and device based on image prior and global block aggregation. Background Art

[0002] Fog is a common atmospheric phenomenon caused by the scattering and absorption of light by suspended particles in the atmosphere. Due to the limitations of contrast and visibility in foggy scenes, the imaging quality of images will be significantly reduced, which often causes color shift and detail distortion in foggy images. Therefore, the performance of high-level visual tasks such as scene understanding and target detection in foggy scenes is greatly limited. Defogging images to produce clear images has received widespread attention. The purpose of image defogging is to restore degraded images affected by fog to clear images. Early defogging methods were mainly based on priors from statistical rules or visual clues of clear images. However, artificial assumptions and priors are not always valid, especially in real and changeable foggy scenes. This makes it easy for such methods to obtain distorted defogging images. With the development of deep learning, many methods use convolutional neural networks and self-attention mechanisms to predict atmospheric model parameters (such as transmission maps, atmospheric light) or clear images. Such methods can break the limitations of assumptions and achieve better defogging effects. However, these methods have several inherent problems: (1) Lack of consideration of image priors. This leads to visual artifacts in the defogging images and increases unnecessary computational consumption. (2) Lack of consideration of the complementarity of local and global information. Over-emphasizing local information makes it difficult for the defogging algorithm to restore dense fog areas, while relying on global information also leads to unclear and blurred details. In order to solve these three problems, the present invention proposes a non-local defogging module to search for similar clear features in a global scope for the restoration of foggy features. In particular, unlike the traditional self-attention mechanism, each feature block focuses on similar clear features of a larger scale in a global scope. These similar large-scale blocks provide more clear clues for the restoration of foggy features. Furthermore, the present invention proposes a local feature enhancement module for introducing local features (such as texture details, edges) to enhance defogging features, which enhances the network's expressive power. In addition, the present invention proposes to optimize the training of the defogging network based on adversarial learning based on dark channel priors, so that the network focuses on dense fog areas and fully learns the difference between foggy images and non-fog images. At the same time, SSIM Loss is introduced to optimize the network so that the generated defogging image is closer to human visual perception. Summary of the Invention

[0003] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology, and to improve the defogging performance in complex scenes, especially in non-uniform fog scenes, by establishing a single image defogging system based on image prior and global block aggregation. At the same time, it can reduce the generation of visual artifacts in the defogged image, truly restore the color and texture details of the image, and improve the authenticity of the defogged image.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] In a first aspect, the present invention provides a single image defogging method based on image prior and global block aggregation, comprising the following steps:

[0006] The foggy image is converted into enhanced defogging features at multiple scales using multiple pre-established feature defogging machines. The foggy image is first input to the first feature defogging machine.

[0007] A feature dehazer consists of two parallel processing branches. In branch one, the non-local dehazing module first divides the foggy features into multiple feature blocks and embeds positional codes. Subsequently, several adjacent feature blocks are aggregated to obtain clue blocks. The global dehazing features are obtained by weighted aggregation of the global clue blocks.

[0008] In branch 2, the local feature enhancement module extracts local information from the foggy features, providing more local features for the global defogging feature, thereby improving the feature expression capability of the entire feature defogging device;

[0009] The features output by the two branches are fused to obtain an enhanced dehazing feature at one scale, which is used as the output of the first feature dehazer. The output of the first feature dehazer is used as the input of the second feature dehazer. Through the iteration of multiple feature dehazers, enhanced dehazing features at multiple scales are finally obtained. In the fusion process, the global dehazing feature and the local detail feature are added element by element.

[0010] Finally, the enhanced dehazing features at multiple scales are input into the haze-free image decoder to restore the clear dehazed image.

[0011] As a preferred technical solution, the non-local defogging module implements the averaging of fog features based on overlapping sliding windows and linear transformation. The input features are two-dimensional, and the averaging operation is as follows:

[0012] First, use mirror filling to fill one row or one column of pixels at each of the four boundaries of the input feature;

[0013] Afterwards, a sliding window is used to extract blocks by setting a step size. Each time the window slides, the pixels within the window are used as the basic pixels of the corresponding feature block.

[0014] Finally, the basic pixels are linearly transformed to output a feature block. The sliding window traverses the entire feature map, so that the foggy features are evenly divided into multiple feature blocks. Each feature block is linearly transformed to generate the corresponding position code, which is embedded into the feature block by element-by-element addition.

[0015] As a preferred technical solution, the non-local dehazing module performs weighted aggregation of clue blocks based on the non-local method and multi-head global self-attention, where the generation process of the clue blocks is as follows:

[0016] First, a non-overlapping sliding window is used to scan the feature block. The multiple feature blocks in the sliding window with the window size as the step size are regarded as a large-scale feature block.

[0017] These divided large-scale blocks are then input into a linear transformation layer to extract features, thereby obtaining multiple clue blocks. The clue blocks are used for subsequent aggregation and provide valuable information for the restoration of foggy features. In the weighted aggregation process of clue blocks, multiple parallel restoration heads are used to aggregate clue blocks of different global ranges. Each restoration head establishes the correlation between each feature block and the global clue block, and then uses a non-local method to aggregate the global clue block. The process of establishing the relationship between feature blocks and clue blocks in the global scope is described as follows:

[0018]

[0019] where q i ,k j It is obtained by linearly transforming the feature block and clue block respectively, and the cosine similarity is used to calculate q i ,k j The weight relationship between them, σ is the Softmax activation function, C represents the channel dimension of the feature block, w i,j Represents the similarity between feature block i and clue block j. Through the above operations, the similarity between each feature block and all clue blocks is calculated. Based on these similarities, the global clue blocks are weighted aggregated. This aggregation process is based on the non-local method and is defined as:

[0020] o=wv

[0021] Among them, w is the weight matrix of the feature block and the clue block, v is the linear transformation output of the clue block, and the aggregated output o is the aggregated feature output by a restoration head. Finally, the outputs of multiple restoration heads are exchanged through a feedforward network to obtain the global dehazing feature.

[0022] As a preferred technical solution, the image prior is defined as a cross-scale reproducibility property, which means that a small block in a natural image will appear on the same image in the form of multiple scales, that is, the natural image is divided into multiple small image blocks of the same scale, and these small image blocks of the same scale will be repeated in the form of multiple scales in the global image; this property is extended to the feature space, and in the feature space, the same feature block will be repeated on the image in the form of multiple scales. It is in this way that the non-local defogging module finds similar clear clue blocks for feature blocks that are severely degraded by fog, thereby providing more information for the restoration of severely degraded feature blocks.

[0023] As an optimal technical solution, the local feature enhancement module is implemented through local operations. The input of the local feature enhancement module is a two-dimensional foggy feature. First, local features are extracted using maximum pooling, where the size of the pooling kernel is 2×2; finally, a 1×1 convolution operation is used to reorganize the feature dimensions.

[0024] As a preferred technical solution, the following steps are also included: using adversarial learning based on dark channel prior to perform network optimization, and introducing the SSIM loss function to optimize and improve the dehazing performance, specifically:

[0025] Use dark channel prior DCP to construct the contrast loss function, which is:

[0026]

[0027] Where G(·) represents the proposed defogging method, G(I) represents the defogging image output by the defogging method, For feature splicing, splicing is performed on the feature channel, G(I) DCP It represents the dark channel component of the dehazed image. This component is calculated using the dark channel prior. The dehazed image is input into the contrast loss function for identification. If the dehazed image has a poor dehazed effect, the contrast loss function will generate a relatively large value. On the contrary, if the dehazed image has a good dehazed effect, the loss function will generate a relatively small value. The contrast learning mechanism based on the dark channel prior is good at focusing on the dense fog area in the image, so that the network generates a more realistic dehazed image and improves the dehazed performance. The calculation of the dark channel component is defined as:

[0028]

[0029] Among them, J c (y) is a color channel of the image, y∈Ω(x) is a local patch centered at x, It means finding the minimum value of three channels. It is a minimum filter, which results in the minimum value of the local area. The dark channel prior indicates that in the non-sky area of ​​the image, the grayscale value of the dark channel component in the fog-free area is close to 0, while the dark channel in the foggy area has a relatively large grayscale value. The dark channel component enables the adversarial learning in our method to focus more on the foggy area of ​​the image, better identify the authenticity of the image, and thus improve the dehazing performance of the model;

[0030] The SSIM loss function measures the structural similarity between the dehazed image and its corresponding true clear image. It encourages the dehazed image to be closer to the perception of the human visual system. The SSIM value of the two images is defined as:

[0031]

[0032] Among them, μ x and are the mean and variance of x respectively. xy is the covariance of x and y. C1 and C2 are constants used to maintain stability. The SSIM value ranges from 0 to 1, and the SSIM loss function is defined as:

[0033] L S =1-SSIM(G(I),J)

[0034] The closer the value of the SSIM loss function is to 0, the closer the dehazed image is to the real haze-free image in terms of structural similarity, thereby optimizing the dehazing performance of the network.

[0035] As an optimal technical solution, the haze-free image decoder is designed based on deconvolution operation and ResNet Block. Deconvolution is used to implement upsampling operation, where the convolution kernel size is 3×3, the stride is 2, and the padding is 1. One ResNet Block contains three residual convolution operations.

[0036] In a second aspect, the present invention further provides a single image defogging system based on image prior and global block aggregation, which is applied to the single image defogging method based on image prior and global block aggregation, including a non-local defogging module, a local feature enhancement module, a fog-free image decoder, and an adversarial learning module;

[0037] The non-local defogging module is used to restore foggy images or foggy features to obtain enhanced defogging features. First, the input is block-segmented and linearly transformed to obtain multiple feature blocks. Adjacent feature blocks are then linearly transformed to output clue blocks. Multiple restoration heads are then established. In each restoration head, clue blocks are aggregated based on a non-local method to achieve feature restoration. The outputs of the multiple restoration heads are input into a feedforward network for fusion, so that feature information from different ranges is exchanged, and finally the restored features are obtained.

[0038] The local feature enhancement module is used to extract local features from the input foggy features. These extracted features are subsequently fused with the global defogging features to obtain enhanced defogging features. The enhanced defogging features have rich global and local information, making the texture and color restoration of the defogging results more accurate;

[0039] The fog-free image decoder is used to output the fog-free image. Multiple feature defoggers output enhanced defogging features of multiple scales. These features are input into a pre-established fog-free image decoder, and finally output a clear image after defogging.

[0040] The adversarial learning module is based on dark channel prior adversarial learning, which is used to optimize the training process of the defogging network. By identifying the defogging image generated by the defogging network and feeding it back to the network's back propagation process, the dark channel component enables the network to focus on restoring dense fog areas.

[0041] In a third aspect, the present invention provides an electronic device, comprising:

[0042] at least one processor; and,

[0043] a memory communicatively connected to the at least one processor; wherein,

[0044] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the single image dehazing method based on image prior and global block aggregation.

[0045] In a fourth aspect, the present invention further provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the single image dehazing method based on image prior and global block aggregation.

[0046] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0047] 1. The non-local dehazing module proposed in this paper is used to restore severely degraded dense fog features. It leverages the cross-scale block recurrence property to globally aggregate clear cues to restore severely degraded local features, rather than relying on local aggregation. Specifically, unlike previous methods based on convolutional neural networks and traditional self-attention mechanisms, the clear cues in this method come from large-scale feature patches, enabling the network to produce dehazing results with fewer visual artifacts.

[0048] 2. The proposed network is based on a proposed local feature enhancement module for enhancing dehazing features. This module provides a shortcut for transferring local features while preserving them, which provides sufficient local detail information for enhancing dehazing features. This improves the feature representation capability of the proposed network and reduces the loss of detail, which helps produce ideal dehazing results.

[0049] 3. The present invention optimizes the proposed dehazing network by using a contrastive learning optimization method based on dark channel prior, which encourages the network to focus on degraded areas and utilize the dark channel components of the image, thereby improving the authenticity of the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0051] Figure 1 This is a flow chart of a single image dehazing method based on image prior and global block aggregation according to an embodiment of the present invention;

[0052] Figure 2 This is a network diagram based on image prior and global block aggregation according to an embodiment of the present invention;

[0053] Figure 3 is a schematic diagram of adversarial learning based on dark channel prior in an embodiment of the present invention;

[0054] Figure 4 is a schematic diagram of a non-local defogging module in an embodiment of the present invention;

[0055] Figure 5 This is a comparison chart of the defogging effects of the present invention and several advanced defogging algorithms;

[0056] Figure 6 2 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0058] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0059] See also Figure 1 The single image defogging method based on image prior and global block aggregation in this embodiment includes the following steps:

[0060] S1. Multiple feature dehazers are pre-established to convert fog features into enhanced dehazing features at multiple scales. The input foggy image is first fed into the first feature dehazer.

[0061] The S2 feature dehazing module has two parallel processing branches. In branch 1, the non-local dehazing module first divides the foggy features into multiple feature blocks, and position information is embedded in these feature blocks. Several adjacent feature blocks are then aggregated to form clue blocks. Dehazing features are obtained through weighted aggregation of these clue blocks.

[0062] Furthermore, step S2 is specifically as follows:

[0063] S21. Use a sliding window to divide the feature map into multiple feature blocks. Then perform a linear transformation on each feature block to generate the corresponding position code, which is embedded into the feature block by element-by-element addition.

[0064] Furthermore, the fog feature averaging operation in the non-local dehazing module is implemented using overlapping sliding windows and linear transformations. We first apply mirror padding to fill a row or column of pixels at each of the four edges of the input feature. Block extraction is performed using a 3×3 sliding window with a stride of 2. Each time the window slides, the pixels within the window serve as the basis pixels for a feature block, which is then linearly transformed to output the feature block. Specifically, the sliding window traverses the entire feature map.

[0065] S22. Several adjacent feature blocks are then aggregated to form clue blocks. We use non-overlapping sliding windows to partition adjacent feature blocks, treating the blocks within the sliding window as a single large-scale block. We then perform feature extraction on these large-scale blocks to obtain multiple clue blocks. This extraction process is implemented using linear transformations.

[0066] S23. Calculate the weight of each feature block relative to all clue blocks globally, and then perform weighted aggregation of the clue blocks based on the weighted ...

[0067] Furthermore, in this embodiment, the non-local dehazing module is based on a non-local method and multi-head global self-attention. During the weighted aggregation of clue blocks, multiple parallel restoration heads are used to aggregate clue blocks of different global scopes. Each restoration head establishes a correlation between each feature block and the global clue block, and then uses a non-local method to aggregate the global clue block. In particular, the process of establishing the relationship between feature blocks and clue blocks at the global scale can be described as follows:

[0068]

[0069] where q i ,k j is obtained by linearly transforming the feature block and clue block respectively. Cosine similarity is used to calculate the weight relationship between them. σ is the Softmax activation function. C represents the number of channel dimensions of the feature block. w i,j Represents the similarity between feature block i and clue block j. The above operations calculate the similarity between each feature block and all clue blocks. Based on these similarities, we perform weighted aggregation on the global clue blocks. This aggregation process is based on the non-local method and is defined as:

[0070] o=wv

[0071] Where w is the weight matrix between the feature block and the clue block. v is the linear transformation output of the clue block. The aggregate output o is the aggregated feature output of a restoration head. Finally, the outputs of multiple restoration heads are exchanged through a feedforward network to obtain the global dehazing feature.

[0072] In branch 2, foggy features are processed by the local feature enhancement module for local feature extraction. First, we use max pooling with a 2×2 kernel size for local feature extraction. Finally, a 1×1 convolution operation is used to reshape the feature dimensions. This ultimately provides more local details to the global dehazing features output by the non-local dehazing module, enhancing the expressive power of the dehazing network.

[0073] S4. The defogging features obtained by the non-local defogging module and the local enhanced features obtained by the local feature enhancement module are fused by element-by-element addition and then input into the next feature defogging unit. In one embodiment of the present application, four feature defogging units are used in the proposed defogging network. The global aggregation times of the non-local defogging unit are set to [3, 5, 15, 10], and the corresponding parallel restoration heads are set to [1, 2, 5, 8]. Each feature defogging unit uses only one parallel local feature enhancement module.

[0074] S5. Multiple feature dehazers will output multiple corresponding enhanced dehazing features, which are all input into the haze-free image decoder for haze-free image restoration. The decoder is based on the ResNet Block and upsampling design.

[0075] S5. Optimize the dehazing network proposed in steps S1 to S4 using adversarial learning based on dark channel priors and the SSIM loss function. Adversarial learning effectively identifies the authenticity of the dehazed image output by the dehazing network, enabling the network to generate more ideal dehazed images. Meanwhile, the Focal Frequency Loss ensures that the generated image is closer to the human eye's perception, thereby obtaining optimal network parameters and achieving robust dehazing results.

[0076] This example uses adversarial learning based on dark channel priors for network optimization and introduces the SSIM loss function to optimize and improve dehazing performance. Specifically:

[0077] The contrast loss function is constructed using the dark channel prior DCP. The contrast loss function is:

[0078]

[0079] Among them, G(·) represents the dehazing method we proposed. G(I) represents the dehazed image processed by our dehazing network. is feature concatenation, G(I) DCP Represents the dark channel component of the dehazed image, which is calculated using the dark channel prior. The dehazed image is input into the contrast loss function for identification. If the dehazed image is poor, the contrast loss function will generate a relatively large value; conversely, if the dehazed image is good, the loss function will generate a relatively small value. Through the contrast learning mechanism based on the dark channel prior, the network can generate more realistic dehazed images, improving the model performance; in particular, the calculation of the dark channel component can be defined as:

[0080]

[0081] Among them, J c (y) is a color channel of the image, and y∈Ω(x) is a local patch centered at x. It means finding the minimum value of three channels. is a minimum filter, resulting in a local minimum. The dark channel prior indicates that in non-sky areas of an image, the grayscale value of the dark channel component in fog-free areas is close to 0, while foggy areas have relatively large grayscale values. By calculating the dark channel prior, the contrastive learning in our method can focus more on foggy areas of the image, better identifying the image's authenticity, thereby improving the model's dehazing performance.

[0082] The SSIM loss function measures the structural similarity between the dehazed image and its corresponding true image. It encourages the dehazed image to be closer to the perception of the human visual system. The SSIM value of the two images is defined as:

[0083]

[0084] Among them, μ x and are the mean and variance of x respectively. xy is the covariance of x and y.

[0085] C1 and C2 are constants used to maintain stability. The SSIM value ranges from 0 to 1. The SSIM loss function is defined as:

[0086] L S =1-SSIM(G(I),J)

[0087] The closer the value of the SSIM loss function is to 0, the closer the defogging image is to the real fog-free image in terms of structural similarity, thereby optimizing the defogging performance of the network.

[0088] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0089] Based on the same concept as the single image defogging method based on image prior and global block aggregation in the above-mentioned embodiment, the present invention also provides a single image defogging system based on image prior and global block aggregation, which can be used to execute the above-mentioned single image defogging method based on image prior and global block aggregation. For ease of explanation, the structural diagram of the embodiment of the single image defogging system based on image prior and global block aggregation only shows the parts related to the embodiment of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation of the device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0090] See also Figure 2,In another embodiment of the present application, a single image defogging system based on image prior and global block aggregation is provided, the system including a local defogging module, a local feature enhancement module, a haze-free image decoder to adversarial learning module;

[0091] The non-local defogging module is used to restore foggy images or foggy features to obtain enhanced defogging features. First, the input is block-segmented and linearly transformed to obtain multiple feature blocks. Adjacent feature blocks are then linearly transformed to output clue blocks. Multiple restoration heads are then established. In each restoration head, clue blocks are aggregated based on a non-local method to achieve feature restoration. The outputs of the multiple restoration heads are input into a feedforward network for fusion, so that feature information from different ranges is exchanged, and finally the restored features are obtained.

[0092] The local feature enhancement module is used to extract local features from the input foggy features. These extracted features are subsequently fused with the global defogging features to obtain enhanced defogging features. The enhanced defogging features have rich global and local information, making the texture and color restoration of the defogging results more accurate;

[0093] The fog-free image decoder is used to output the fog-free image. Multiple feature defoggers output enhanced defogging features of multiple scales. These features are input into a pre-established fog-free image decoder, and finally output a clear image after defogging.

[0094] The adversarial learning module is based on dark channel prior adversarial learning, which is used to optimize the training process of the defogging network. By identifying the defogging image generated by the defogging network and feeding it back to the network's back propagation process, the dark channel component enables the network to focus on restoring dense fog areas.

[0095] It should be noted that the single image defogging system based on image prior and global block aggregation of the present invention corresponds one-to-one to the single image defogging method based on image prior and global block aggregation of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the single image defogging method based on image prior and global block aggregation are all applicable to the embodiment of the single image defogging based on image prior and global block aggregation. For specific contents, please refer to the description in the embodiment of the method of the present invention. No further details will be given here. This is hereby declared.

[0096] In addition, in the implementation of the single image dehazing system based on image prior and global block aggregation in the above embodiment, the logical division of each program module is only an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the convenience of software implementation or the configuration requirements of the corresponding hardware. That is, the internal structure of the single image dehazing system based on image prior and global block aggregation is divided into different program modules to complete all or part of the functions described above.

[0097] Please refer again Figure 2 , the single image defogging system based on image prior and global block aggregation in this application is specifically:

[0098] Figure 2 In the non-local dehazing module, the white block on the far left processes the feature map with the largest scale. As we go to the right, the size of the feature map decreases by a factor of 1 / 2. This is because we perform a linear transformation when constructing the feature block, reducing the corresponding elements.

[0099] Figure 2 In the haze-free image decoder, each block first undergoes a deconvolution process to increase the size of the feature map, which makes the feature map increase from left to right.

[0100] Figure 2 In the figure, the white and light black structures on the left are the non-local dehazing module and the local feature enhancement module respectively; these two modules form a parallel structure for processing feature maps simultaneously.

[0101] Figure 2 and 4 In , “□” means accumulating elements pixel by pixel;

[0102] Figure 3 It shows the principle of contrastive learning based on dark channel prior in the present invention.

[0103] In addition to SSIM loss and contrastive learning loss, the present invention also uses L1 loss and perceptual loss as basic loss functions;

[0104] Figure 3 The dehazed image in Figure 2 Generated in the dehazing network;

[0105] The present invention uses the following weights for training: L1 loss weight is 1, perceptual loss weight is 2, contrastive learning loss function loss weight based on dark channel prior is 0.5, and SSIM loss weight is 1;

[0106] We trained the model for 200 epochs on two NVIDIA RTX-2080TI GPUs with an initial learning rate of 0.0001. We randomly cropped the image using a 240×240 window and randomly rotated the cropped image, using the resulting image as training input.

[0107] Figure 4 Schematic diagram of the structure and polymerization of non-local defogging in the present invention.

[0108] Figure 4 The H in it represents the number of recovery heads, and L represents the number of aggregations;

[0109] Figure 4 The block aggregation in is defined as follows:

[0110]

[0111] where q i ,k j is obtained by linearly transforming the feature block and clue block respectively. Cosine similarity is used to calculate the weight relationship between them. σ is the Softmax activation function. C represents the number of channel dimensions of the feature block. w i,j Represents the similarity between feature block i and clue block j. The above operations calculate the similarity between each feature block and all clue blocks. Based on these similarities, we perform weighted aggregation on the global clue blocks. This aggregation process is based on the non-local method and is defined as:

[0112] o=wv

[0113] Where w is the weight matrix between the feature block and the clue block. v is the linear transformation output of the clue block. The aggregate output o is the aggregated feature output of a restoration head. Finally, the outputs of multiple restoration heads are exchanged through a feedforward network to obtain the global dehazing feature.

[0114] Figure 5 The results of the subjective analysis of the method of the present invention are shown.

[0115] Figure 5 As shown, the method of the invention can better cope with foggy scenes under complex conditions, fully remove fog while avoiding the generation of artifacts, and better restore the texture details of the image.

[0116] Table 1 shows the objective evaluation results of our method. We use peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) as evaluation metrics. In each pair, the left value represents PSNR, and the right value represents SSIM. Higher values ​​indicate that the generated image is closer to a standard definition image.

[0117] Table 1

[0118]

[0119] See also Figure 6 In one embodiment, an electronic device for implementing a single image defogging method based on image prior and global block aggregation is provided. The electronic device 100 may include a first processor 101, a first memory 102 and a bus, and may also include a computer program stored in the first memory 102 and executable on the first processor 101, such as a single image defogging program 103 based on image prior and global block aggregation.

[0120] The first memory 102 includes at least one type of readable storage medium, including a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the first memory 102 may be an internal storage unit of the electronic device 100, such as a mobile hard disk of the electronic device 100. In other embodiments, the first memory 102 may also be an external storage device of the electronic device 100, such as a plug-in mobile hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 100. Furthermore, the first memory 102 may include both an internal storage unit of the electronic device 100 and an external storage device. The first memory 102 can be used not only to store application software and various types of data installed on the electronic device 100, such as the code of the single image dehazing program 103 based on image priors and global block aggregation, but can also be used to temporarily store data that has been output or is about to be output.

[0121] In some embodiments, the first processor 101 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The first processor 101 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the first memory 102, as well as calling data stored in the first memory 102, to perform various functions of the electronic device 100 and process data.

[0122] Figure 6 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 6 The structure shown does not constitute a limitation on the electronic device 100 , and the electronic device 100 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0123] The single image defogging program 103 based on image prior and global block aggregation stored in the first memory 102 of the electronic device 100 is a combination of multiple instructions. When executed in the first processor 101, it can achieve the following:

[0124] The foggy image is converted into enhanced defogging features at multiple scales using multiple pre-established feature defogging machines. The foggy image is first input to the first feature defogging machine.

[0125] A feature dehazer consists of two parallel processing branches. In branch one, the non-local dehazing module first divides the foggy features into multiple feature blocks and embeds positional codes. Subsequently, several adjacent feature blocks are aggregated to obtain clue blocks. The global dehazing features are obtained by weighted aggregation of the global clue blocks.

[0126] In branch 2, the local feature enhancement module extracts local information from the foggy features, providing more local features for the global defogging feature, thereby improving the feature expression capability of the entire feature defogging device;

[0127] The features output by the two branches are fused to obtain an enhanced dehazing feature at one scale, which is used as the output of the first feature dehazer. The output of the first feature dehazer is used as the input of the second feature dehazer. Through the iteration of multiple feature dehazers, enhanced dehazing features at multiple scales are finally obtained. In the fusion process, the global dehazing feature and the local detail feature are added element by element.

[0128] Finally, the enhanced dehazing features at multiple scales are input into the haze-free image decoder to restore the clear dehazed image.

[0129] Furthermore, if the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0130] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0131] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. Single image dehazing method based on image prior and global block aggregation, characterized by: The steps include: The foggy image is converted into enhanced defogging features at multiple scales using multiple pre-established feature defogging machines. The foggy image is first input to the first feature defogging machine. A feature dehazer consists of two parallel processing branches. In branch one, the non-local dehazing module first divides the foggy features into multiple feature blocks and embeds positional codes. Subsequently, several adjacent feature blocks are aggregated to obtain clue blocks. The global dehazing features are obtained by weighted aggregation of the global clue blocks. In branch 2, the local feature enhancement module extracts local information from the foggy features, providing more local features for the global defogging feature, thereby improving the feature expression capability of the entire feature defogging device; The features output by the two branches are fused to obtain an enhanced dehazing feature at one scale, which is used as the output of the first feature dehazer. The output of the first feature dehazer is used as the input of the second feature dehazer. Through the iteration of multiple feature dehazers, enhanced dehazing features at multiple scales are finally obtained. In the fusion process, the global dehazing feature and the local detail feature are added element by element. Finally, the enhanced dehazing features at multiple scales are input into the haze-free image decoder to restore the clear dehazed image.

2. The single image dehazing method based on image prior and global block aggregation according to claim 1, characterized in that: The non-local defogging module performs an averaging operation on foggy features based on overlapping sliding windows and linear transformation. The input features are two-dimensional, and the averaging operation is as follows: First, use mirror filling to fill one row or one column of pixels at each of the four boundaries of the input feature; Afterwards, a sliding window is used to extract blocks by setting a step size. Each time the window slides, the pixels within the window are used as the basic pixels of the corresponding feature block. Finally, the basic pixels are linearly transformed to output a feature block. The sliding window traverses the entire feature map, so that the foggy features are evenly divided into multiple feature blocks. Each feature block is linearly transformed to generate the corresponding position code, which is embedded into the feature block by element-by-element addition.

3. The single image dehazing method based on image prior and global block aggregation according to claim 1, characterized in that: The non-local dehazing module performs weighted aggregation of clue blocks based on the non-local method and multi-head global self-attention, where the clue block generation process is as follows: First, a non-overlapping sliding window is used to scan the feature block. The multiple feature blocks in the sliding window with the window size as the step size are regarded as a large-scale feature block. These divided large-scale blocks are then input into a linear transformation layer to extract features, thereby obtaining multiple clue blocks. The clue blocks are used for subsequent aggregation and provide valuable information for the restoration of foggy features. In the weighted aggregation process of clue blocks, multiple parallel restoration heads are used to aggregate clue blocks of different global ranges. Each restoration head establishes the correlation between each feature block and the global clue block, and then uses a non-local method to aggregate the global clue block. The process of establishing the relationship between feature blocks and clue blocks in the global scope is described as follows: where q i ,k j It is obtained by linearly transforming the feature block and clue block respectively, and the cosine similarity is used to calculate q i ,k j The weight relationship between them, σ is the Softmax activation function, C represents the channel dimension of the feature block, w i,j Represents the similarity between feature block i and clue block j. Through the above operations, the similarity between each feature block and all clue blocks is calculated. Based on these similarities, the global clue blocks are weighted aggregated. This aggregation process is based on the non-local method and is defined as: o=wv Among them, w is the weight matrix of the feature block and the clue block, v is the linear transformation output of the clue block, and the aggregated output o is the aggregated feature output by a restoration head. Finally, the outputs of multiple restoration heads are exchanged through a feedforward network to obtain the global dehazing feature.

4. The single image dehazing method based on image prior and global block aggregation according to claim 1, characterized in that: The image prior is defined as the cross-scale reproducibility property, which means that a small block in a natural image will appear in the same image at multiple scales. That is, the natural image is divided into multiple small image blocks of the same scale, and these small image blocks of the same scale will be repeated in the image at multiple scales. This property is extended to the feature space, where the same feature block will be repeated in the image at multiple scales. The non-local dehazing module finds similar clear clue blocks for feature blocks that are severely degraded by fog in this way, thereby providing more information for restoring severely degraded feature blocks.

5. The single image dehazing method based on image prior and global block aggregation according to claim 1, characterized in that: The local feature enhancement module is implemented through local operations. The input of the local feature enhancement module is a two-dimensional fog feature. First, local features are extracted using maximum pooling, where the size of the pooling kernel is 2×2; finally, a 1×1 convolution operation is used to reorganize the feature dimensions.

6. The single image dehazing method based on image prior and global block aggregation according to claim 1, characterized in that: The following steps are also included: using adversarial learning based on dark channel prior to perform network optimization, and introducing the SSIM loss function to optimize and improve dehazing performance, specifically: Use dark channel prior DCP to construct the contrast loss function, which is: Where G(·) represents the proposed defogging method, G(I) represents the defogging image output by the defogging method, For feature splicing, splicing is performed on the feature channel, G(I) DCP It represents the dark channel component of the dehazed image. This component is calculated using the dark channel prior. The dehazed image is input into the contrast loss function for identification. If the dehazed image has a poor dehazed effect, the contrast loss function will generate a relatively large value. On the contrary, if the dehazed image has a good dehazed effect, the loss function will generate a relatively small value. The contrast learning mechanism based on the dark channel prior is good at focusing on the dense fog area in the image, so that the network generates a more realistic dehazed image and improves the dehazed performance. The calculation of the dark channel component is defined as: Among them, J c (y is a color channel of the image, y∈Ω(x) is a local block centered at x, It means finding the minimum value of three channels. It is a minimum filter, which results in the minimum value of the local area. The dark channel prior indicates that in the non-sky area of ​​the image, the grayscale value of the dark channel component in the fog-free area is close to 0, while the dark channel in the foggy area has a relatively large grayscale value. The dark channel component enables the adversarial learning in our method to focus more on the foggy area of ​​the image, better identify the authenticity of the image, and thus improve the dehazing performance of the model; The SSIM loss function measures the structural similarity between the dehazed image and its corresponding true clear image. It encourages the dehazed image to be closer to the perception of the human visual system. The SSIM value of the two images is defined as: Among them, μ x and are the mean and variance of x, σ xy is the covariance of x and y, C1 and C2 are constants used to maintain stability, the SSIM value ranges from 0 to 1, and the SSIM loss function is defined as: L S =1-SSIM(G(I),J) The closer the value of the SSIM loss function is to 0, the closer the dehazed image is to the real haze-free image in terms of structural similarity, thereby optimizing the dehazing performance of the network.

7. The single image dehazing method based on image prior and global block aggregation according to claim 1, characterized in that: The haze-free image decoder is designed based on deconvolution operations and ResNet Block. Deconvolution is used to implement upsampling operations, where the convolution kernel size is 3×3, the stride is 2, and the padding is 1. One ResNet Block contains three residual convolution operations.

8. Single image dehazing system based on image prior and global block aggregation, characterized by: A single image defogging method based on image prior and global block aggregation applied to any one of claims 1-7, comprising a non-local defogging module, a local feature enhancement module, a haze-free image decoder, and an adversarial learning module; The non-local defogging module is used to restore foggy images or foggy features to obtain enhanced defogging features. First, the input is block-segmented and linearly transformed to obtain multiple feature blocks. Adjacent feature blocks are then linearly transformed to output clue blocks. Multiple restoration heads are then established. In each restoration head, clue blocks are aggregated based on a non-local method to achieve feature restoration. The outputs of the multiple restoration heads are input into a feedforward network for fusion, so that feature information from different ranges is exchanged, and finally the restored features are obtained. The local feature enhancement module is used to extract local features from the input foggy features. These extracted features are subsequently fused with the global defogging features to obtain enhanced defogging features. The enhanced defogging features have rich global and local information, making the texture and color restoration of the defogging results more accurate; The fog-free image decoder is used to output the fog-free image. Multiple feature defoggers output enhanced defogging features of multiple scales. These features are input into a pre-established fog-free image decoder, and finally output a clear image after defogging. The adversarial learning module is based on dark channel prior adversarial learning, which is used to optimize the training process of the defogging network. By identifying the defogging image generated by the defogging network and feeding it back to the network's back propagation process, the dark channel component enables the network to focus on restoring dense fog areas.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to execute the single image dehazing method based on image prior and global block aggregation as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the single image defogging method based on image prior and global block aggregation according to any one of claims 1 to 7 is implemented.

Images