An image defogging method and system based on haze positioning and disentanglement

Abstract

The application discloses an image defogging method and system based on haze positioning and disentanglement. Training data and a defogging network are constructed, the defogging network comprising an encoder, a haze positioning module, a disentanglement module and a decoder; the haze positioning module is used for outputting haze-related feature information of regional distinguishability, and the disentanglement module is used for guiding training of the defogging network; a loss function is designed, and the defogging network is trained by using the training data to form a trained defogging network; the trained defogging network is tested by using a no-reference image quality evaluation index, and when the performance is good, the network parameters are fixed to form a final defogging network. The haze positioning module and the disentanglement module are introduced, and various types of haze can be removed; compared with existing image defogging methods, the application has obvious advantages in haze removal quality evaluation, haze removal stability, haze removal accuracy and haze removal rapidity, and the restored haze-free image is more natural.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision, and specifically to an image dehazing method and system based on haze localization and deentanglement. Background Technology

[0002] Severe weather phenomena such as smog and dust pollution commonly reduce visibility, severely interfering with the imaging quality of visual systems. For visual tasks that rely on image information, such as autonomous driving assistance, nearshore marine monitoring, and drone aerial photography, image quality degradation caused by smog pollution can pose serious safety risks. To eliminate smog interference, the academic community has conducted in-depth research on visual restoration methods for degraded images. Although many representative dehazing methods have been proposed, eliminating smog in real-world scenarios still faces significant challenges.

[0003] Influenced by environmental factors such as atmospheric light intensity and wind speed variations, the distribution of haze in natural scenes often exhibits a non-uniform state. This non-uniform distribution of haze interferes with the acquisition of edge features and texture information in images to varying degrees, leading to image quality degradation. To recover clear images from natural haze images, previous works have used heuristically designed local or global priors to focus on hazy regions in the image and model them to recover clear images. However, dehazing methods based on handcrafted priors often require ideal data assumptions and constraints from robust prior information. Due to the non-uniform distribution of haze in real-world scenes, dehazing results are poor when the obtained actual prior information deviates from perfect data assumptions. Unlike dehazing methods based on heuristically designed priors, many deep learning-based dehazing methods estimate physical model parameters or directly learn the mapping relationship between hazy images and clear images by designing deep, complex network structures. However, due to the complexity of real-world haze distribution, these methods still have significant room for optimization when processing natural haze images with different densities and depths.

[0004] One current technique is the real image dehazing method proposed in the paper "RIDCP: Revitalizing Real Image Dehazing via High-Quality Codebook Priors," which proposes a new paradigm for real image dehazing from the perspectives of synthesizing more realistic haze data and introducing more robust priors into the network. This technique first proposes a haze data generation pipeline based on a traditional atmospheric light scattering model, incorporating various degrading factors. Second, it pre-trains a VQGAN on a large-scale high-quality dataset to obtain a discrete codebook containing high-quality priors (HQPs). After replacing the negative effects of haze with HQPs, the decoder equipped with a novel normalized feature alignment module can effectively utilize high-quality features and produce clear results. The drawback of this technique is that while it achieves excellent dehazing performance on synthetic datasets and real haze datasets in specific scenarios due to its relatively complex deep network structure, it also suffers from a large number of model parameters, high computational complexity, and an inability to remove haze from complex open real-world scenes in real time.

[0005] The second existing technique is an integrated scene restoration framework based on two neural modules proposed in the paper "All-In-One Image Restoration for Unknown Corruption". This technique recovers degraded images based on a contrastive degradation encoder and a degradation guidance module. This method is unaffected by prior knowledge of degradation type and degree, using only observed degraded images for inference. This advantage makes the method more flexible and cost-effective in real-world scenes. The drawback of this technique is that it cannot effectively dehaze haze types such as those caused by nighttime, dust storms, or color interference due to a lack of sufficient prior knowledge. Furthermore, although this technique outperforms many existing deep dehazing methods in terms of performance and speed on synthetic datasets, it still lacks generalization ability and real-time performance when dealing with complex real-world haze. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing methods and propose an image dehazing method and system based on haze localization and disentanglement. The main problem solved by this invention is to design a lightweight network composed of a haze localization module and a disentanglement strategy, thereby achieving lightweight model parameters and low computational complexity while ensuring dehazing generalization and robustness.

[0007] To address the aforementioned problems, this invention proposes an image dehazing method based on haze localization and deentanglement, the method comprising:

[0008] Construct a haze image dataset LDHaze as training data;

[0009] A dehazing network is constructed, comprising an encoder, a haze localization module, a disentanglement module, and a decoder. The encoder extracts features from the haze image, which are then input into the haze localization and disentanglement modules. The haze localization module outputs haze-related feature information that indicates the region's identifiability. The disentanglement module obtains essential weights unrelated to haze, i.e., haze-free background feature weights, which are used to weight the training data and guide the training of the dehazing network through an adaptive loss. The decoder decodes the output features of the haze localization module to obtain the predicted dehazed image.

[0010] Design a loss function and train the dehazing network using the haze image dataset LDHaze to form a trained dehazing network based on haze localization and disentanglement.

[0011] The trained dehazing network based on haze localization and disentanglement is tested using a no-reference-map quality evaluation index. When the network performs well on the no-reference-map quality evaluation index, the network parameters are fixed to form the final dehazing network based on haze localization and disentanglement. Its input is any haze image, and its output is the image dehazing result.

[0012] Preferably, the construction of the haze image dataset LDHaze is used as training data, specifically as follows:

[0013] The sharp labels used to synthesize haze images come from 2061 sharp images and corresponding depth image information in the RESIDE dataset. The construction process of the haze image dataset LDHaze is described by the following formula:

[0014]

[0015] The formula represents a normal distribution within the interval [0.0001, 5]; I represents the haze image to be synthesized, and F... filter (·) represents the blur kernel function; x and y represent the pixel coordinates, and x′ and y′ are the pixel coordinate shift increments; γ represents the generated initial haze image; γ is a constraint factor used to adjust the brightness of the generated haze data image, and is obtained by uniform sampling in the interval [0.2, 0.8]; A is the atmospheric light value, randomly selected from the interval [0.75, 0.9]; t represents the scene transmittance map; R represents the clear image; ε is the noise term, which follows a mean of 0.

[0016] Preferably, the smog positioning module specifically comprises:

[0017] The overall process of the haze positioning module is as follows:

[0018] First, the output features of the last layer of the encoder in the defogging network are explicitly mapped using a random Fourier explicit mapping function. Second, the explicit mapping result is input into three different path branches to obtain path output features. These three path branches are composed of dilated convolutions with different dilation rates. The output features obtained from these three different path branches are concatenated, and then global average pooling, fully connected layers, and Sigmoid operations are performed to obtain a weight matrix. Finally, the weight matrix is ​​multiplied by the output features of the three path branches to obtain weighted output features of the three different path branches. These output features are then concatenated again and subjected to a 1×1 convolution operation to obtain the final output features of the haze localization module, namely, the haze-related features with regional discernibility.

[0019] The specific process of the smog positioning module is as follows:

[0020] First, the random mapping features in the Gaussian kernel Fourier domain are used to represent the high-order nonlinear feature information in the output features of the last layer of the encoder in a low-dimensional explicit manner:

[0021] Let the output feature of the last layer of the encoder be . Estimate a given Gaussian kernel G on each channel using a low-dimensional explicit mapping function z(y). y The formula is as follows:

[0022]

[0023] Among them G y Represents the Gaussian kernel, y a y b These represent the components of feature y; α represents the size of the Gaussian kernel; z represents the value used to estimate a given Gaussian kernel G(y). a y b The low-dimensional explicit mapping function of ); according to Bochner's theorem, if the Gaussian kernel G(y) has translation invariance a y b In the original implicit space dimension Positive definiteness is true if and only if G y It is the Fourier transform of some nonnegative measure p(s); thus, we obtain the following formula:

[0024]

[0025] Where p(s) represents the nonnegativity measure, i represents the imaginary part, and D represents the feature dimension; bias parameter b d ∈[b1…b D From a uniform distribution U [0，2π] Obtained from; From nonnegative measure Weight parameters obtained from sampling;

[0026] Based on the above formula, the random Fourier features of the output features of the last layer of the encoder are obtained. The form of expression:

[0027]

[0028] Secondly, three parallel paths were designed to capture haze-related feature information of different haze densities and depths in different regions of the image. These three paths were constructed based on dilated convolution, which can extract features from receptive fields of different sizes while maintaining image resolution and reducing the number of parameters. In each of the three paths, two 3×3 dilated convolution layers were used with dilation rates of 1, 3, and 5, respectively. The receptive field sizes of the first dilated convolution layer were 3×3, 7×7, and 11×11, respectively, and the receptive field sizes of the second dilated convolution layer were 5×5, 13×13, and 21×21, respectively.

[0029] Considering that the haze-related feature information obtained through these three feature extraction paths still originates from local regions, a self-correction module is designed to model the interdependencies between global and local features in the channel direction and spatial direction of the input feature mapping, respectively. In this self-correction module, the upper branch models the interdependencies between global and local features in the channel direction through global average pooling, 3×3 convolution, fully connected operations, and sigmoid operations; the lower branch models the interdependencies between global and local features in the spatial direction through global average pooling, 1×1 convolution, 3×3 convolution, and sigmoid operations in the height and width directions. Finally, the output features of the upper and lower branches are concatenated and a 1×1 convolution operation is performed to obtain the final output feature of the self-correction module.

[0030] The output features obtained from these three different path branches are concatenated, and then global average pooling, full connection, and Sigmoid operations are performed to obtain the weight matrix.

[0031] Finally, the weight matrix and the output features of the three different path branches are multiplied to obtain the weighted output features of the three different path branches. These features are then fused to further obtain regionally discernible haze-related feature information; the fusion process is as follows:

[0032]

[0033] Where c represents the total feature dimension; F k This represents the output feature of any branch. This represents the feature dimension of the i-th channel in the k-th branch. The fusion features are represented by u and v, where u and v represent the height and width components of the features, and h and w represent the height and width of the output features. After concatenating the features obtained from the three different paths using global average pooling along the channel direction, the final fusion feature is obtained.

[0034]

[0035] Where FC represents a fully connected convolution operation; gc represents a 1×1 convolution operation; This is a multiplication operation; It is the value of the i-th channel of the weight w. i The matrix formed by expansion; This represents the final output feature of the haze positioning module, namely the haze-related features that indicate regional identifiability.

[0036] Preferably, the de-entanglement module specifically comprises:

[0037] The de-entanglement module removes the dependency between essential features and irrelevant features by utilizing the random features in the Gaussian kernel Fourier domain and the characteristics of sample weighting.

[0038] Let Y be the set of output features of the last layer of the encoder in the dehazing network. To make any pair of features y in Y... :，i y :，j If unrelated, then y :，i y :，j The cross-covariance between y :，i y :，j ) = 0; To reduce computational overhead, the Hibert-Schmidt norm of Euclidean space is converted to the Frobenius norm for solution, and the random characteristics in the Fourier domain of the Gaussian kernel function are used for approximate estimation. Therefore, the above y :，i y :，j The cross-covariance between y :，i y :，j Rephrased as:

[0039]

[0040] in,

[0041]

[0042] in, for h n fx(·) are the feature values ​​sampled from the random Fourier feature function space; the number of samples n = 5;

[0043] By introducing stochastic Fourier features and learning a weighted matrix based on them, global decorrelation and standardization are performed on the data to remove the dependency between haze-free background features and haze. Furthermore, the cross-covariance (Cov) of paired features is minimized to make them independent, thus achieving haze deentanglement. Represents sample weights and For any paired feature y ：，i y ：，j The weighted cross-covariance matrix The calculation is performed using the following formula:

[0044]

[0045] in This represents the sample weight of the i-th paired feature. Let represent the sample weight of the j-th paired feature; further, it is expressed by the following formula... Optimize to obtain the optimized weights. That is, the essential weight:

[0046]

[0047] in Represents the sample weights obtained through optimization. The minimum value in, and c represents the total feature dimension; This represents the 2-norm operation.

[0048] Preferably, the design loss function is as follows:

[0049] Use the optimized weights That is, the essential weights are used to weight the training samples, and the training of the dehazing network is guided by constructing the following adaptive loss:

[0050]

[0051] And with

[0052]

[0053] Where x i Represents a haze image, J i The label represents a clear image, and t represents the timestamp for each batch. This represents the predicted fog-free image output; initial weight values. y ：，i y ：，j This represents any pair of features in the feature set Y;

[0054] Furthermore, a contrastive loss function is designed to constrain the haze-free image reconstructed by the network to be closer to the clear sample image and further away from the original haze image; its objective loss function is expressed in the following form:

[0055]

[0056] Among them, g k This represents the k-th hidden layer feature extracted from the VGG-19 model with fixed pre-trained weights; J represents the clear labeled image, and I represents the original haze image; Indicates a positive predicted value. Indicates a negative predicted value; This represents the predicted fog-free image output; further, the contrast loss is defined as follows:

[0057]

[0058] Where α represents the penalty parameter, λ k Represents the weighting coefficient;

[0059] The total loss function can be summarized in the following form:

[0060] Loss = Loss a +Loss b .

[0061] Preferably, the trained dehazing network based on haze localization and disentanglement is tested using a no-reference-graph quality evaluation index. When the network performs well on the no-reference-graph quality evaluation index, the network parameters are fixed to form the final dehazing network based on haze localization and disentanglement. Specifically:

[0062] The test dataset comes from real-world haze data. During testing, haze images of any size are input into the dehazing network in batches.

[0063] By loading the weights of the dehazing network trained on the LDHaze dataset, the haze images in the test dataset are successively input into the dehazing network to obtain the predicted haze-free image output;

[0064] The quality of the predicted fog-free output image is evaluated using the no-reference image quality evaluation indices PM2.5, FADE, and PIQE. If the performance is good on the above no-reference image quality evaluation indices, the network parameters are fixed to form the final defogging network based on fog localization and disentanglement.

[0065] Accordingly, this invention also proposes an image dehazing system based on haze localization and disentanglement, comprising:

[0066] The training data construction unit is used to construct the haze image dataset LDHaze as training data.

[0067] A dehazing network construction unit is used to construct a dehazing network, which includes an encoder, a haze localization module, a deentanglement module, and a decoder. The encoder extracts features from the haze image, which are then input into the haze localization module and the deentanglement module. The haze localization module outputs haze-related feature information that indicates the region's identifiability. The deentanglement module obtains essential weights unrelated to haze, i.e., haze-free background feature weights, which are used to weight the training data and guide the training of the dehazing network by constructing an adaptive loss. The decoder decodes the output features of the haze localization module to obtain the predicted dehazing image.

[0068] The dehazing network training unit is used to design the loss function and train the dehazing network using the haze image dataset LDHaze to form a trained dehazing network based on haze localization and disentanglement.

[0069] The test application unit is used to test the trained haze-based dehazing network using a no-reference-map quality evaluation index. When the network performs well on the no-reference-map quality evaluation index, the network parameters are fixed to form the final haze-based dehazing network. Its input is an arbitrary haze image, and its output is the image dehazing result.

[0070] Implementing this invention has the following beneficial effects:

[0071] This invention introduces a haze localization module and uses a weighted summation method for feature fusion. Unlike many dehazing methods that simply concatenate features, this invention has the following advantages: 1) Learning weights helps the module adaptively adjust the importance of feature information extracted by the three parallel paths, enhancing the network's ability to distinguish and learn haze of different densities and depths; 2) Direct concatenation operations will multiply the number of feature channels, while using positional summation will maintain the original number of channels and will not increase the computational load.

[0072] This invention introduces a haze deentanglement module, which directly removes the dependency between haze and non-haze background features in the relevant feature space by learning weights from training samples, thereby helping the dehazing model focus on learning the essential connection between discriminative features and clear labels.

[0073] The perceptual contrast loss constructed in this invention can bring the distance between the predicted fog-free output and the clear label closer in the implicit distance space, and push the distance between the predicted fog-free output and the hazy image further away, so that the overall network training is updated in the direction of natural dehazing.

[0074] This invention can remove various types of haze. Compared with existing image dehazing methods, it has significant advantages in dehazing quality evaluation, stability, accuracy, and speed. In this invention, the inputs and outputs of different parts of the network are interconnected, extracting and integrating different features, improving efficiency and resulting in a more natural restored haze-free image. Attached Figure Description

[0075] Figure 1 This is a flowchart illustrating the overall process of an image dehazing method based on haze localization and de-entanglement according to an embodiment of the present invention.

[0076] Figure 2 This is a structural diagram of the smog positioning module according to an embodiment of the present invention;

[0077] Figure 3 This is a structural diagram of the extraction path for extracting regionally discernible haze-related feature information according to an embodiment of the present invention;

[0078] Figure 4 This is a structural diagram of the self-correction module according to an embodiment of the present invention;

[0079] Figure 5 This is a diagram showing the overall structure of the defogging network according to an embodiment of the present invention;

[0080] Figure 6 This is a structural diagram of an image dehazing system based on haze localization and de-entanglement according to an embodiment of the present invention. Detailed Implementation

[0081] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0082] Figure 1 This is a general flowchart of an image dehazing method based on haze localization and deentanglement according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0083] S1, construct the haze image dataset LDHaze as training data;

[0084] S2. Construct a dehazing network, which includes an encoder, a haze localization module, a deentanglement module, and a decoder. The encoder extracts features from the haze image, which are then input into the haze localization module and the deentanglement module. The haze localization module outputs haze-related feature information that makes the region identifiable. The deentanglement module obtains essential weights unrelated to haze, i.e., haze-free background feature weights, which are used to weight the training data and guide the training of the dehazing network by constructing an adaptive loss. The decoder decodes the output features of the haze localization module to obtain the predicted dehazed image.

[0085] S3. Design a loss function and train the dehazing network using the LDHaze haze image dataset to form a trained dehazing network based on haze localization and disentanglement.

[0086] S4. The trained dehazing network based on haze localization and disentanglement is tested using a no-reference-map quality evaluation index. When the performance is good on the no-reference-map quality evaluation index, the network parameters are fixed to form the final dehazing network based on haze localization and disentanglement. Its input is any haze image and its output is the image dehazing result.

[0087] Step S1 is as follows:

[0088] The sharp labels used to synthesize haze images come from 2061 sharp images and corresponding depth image information in the RESIDE dataset. The construction process of the haze image dataset LDHaze is described by the following formula:

[0089]

[0090] The formula represents a normal distribution within the interval [0.0001, 5]; I represents the haze image to be synthesized, and F... filter (·) represents the blur kernel function; x and y represent the pixel coordinates, and x′ and y′ are the pixel coordinate shift increments; γ represents the generated initial haze image; γ is a constraint factor used to adjust the brightness of the generated haze data image, and is obtained by uniform sampling in the interval [0.2, 0.8]; A is the atmospheric light value, randomly selected from the interval [0.75, 0.9]; t represents the scene transmittance map; R represents the clear image; ε is the noise term, which follows a mean of 0.

[0091] Step S2 is as follows:

[0092] S2-1, Constructing a fog and haze positioning module:

[0093] like Figure 2 As shown, the overall process of the haze positioning module is as follows:

[0094] First, the output features of the last layer of the encoder in the dehazing network are explicitly mapped using a stochastic Fourier explicit mapping function. Second, the explicit mapping result is input into three different path branches to obtain path output features. These three path branches are composed of dilated convolutions with different dilation rates. The output features obtained from these three different path branches are concatenated, and then global average pooling, fully connected layers, and Sigmoid operations are performed to obtain a weight matrix. Finally, the weight matrix is ​​multiplied by the output features of the three path branches to obtain weighted output features of the three different path branches. These output features are then concatenated again and subjected to a 1×1 convolution operation to obtain the final output features of the haze localization module, namely, the haze-related features with regional discernibility.

[0095] The specific process of the smog positioning module is as follows:

[0096] First, the random mapping features in the Gaussian kernel Fourier domain are used to represent the high-order nonlinear feature information in the output features of the last layer of the encoder in a low-dimensional explicit manner:

[0097] Let the output feature of the last layer of the encoder be... Estimate a given Gaussian kernel G on each channel using a low-dimensional explicit mapping function z(y). y The formula is as follows:

[0098]

[0099] Among them G y Represents the Gaussian kernel, y a y b These represent the components of feature y; α represents the size of the Gaussian kernel; z represents the value used to estimate a given Gaussian kernel G(y). a y b The low-dimensional explicit mapping function of ); according to Bochner's theorem, if the Gaussian kernel G(y) has translation invariance a y b In the original implicit space dimension Positive definiteness is true if and only if G y It is the Fourier transform of some nonnegative measure p(s); thus, we obtain the following formula:

[0100]

[0101] Where p(s) represents the nonnegativity measure, i represents the imaginary part, and D represents the feature dimension; bias parameter b d ∈[b1…b D From a uniform distribution U [0，2π] Obtained from; From nonnegative measure Weight parameters obtained from sampling;

[0102] Based on the above formula, the random Fourier features of the output features of the last layer of the encoder are obtained. The form of expression:

[0103]

[0104] Secondly, three parallel paths are designed to capture haze-related features of different haze densities and depths in different regions of the image. These three paths are constructed based on dilated convolution, which can extract features from receptive fields of different sizes while maintaining image resolution and reducing the number of parameters. The structure of any extraction path used to extract regionally discriminative haze-related features is as follows: Figure 3 As shown; in the three paths, two 3×3 dilated convolution layers were used respectively, and the dilation rates were set to 1, 3, and 5 respectively; the receptive field sizes of the first dilated convolution layer were 3×3, 7×7, and 11×11 respectively; the receptive field sizes of the second dilated convolution layer were 5×5, 13×13, and 21×21 respectively.

[0105] Considering that the haze-related feature information obtained through these three feature extraction paths still originates from local regions, a self-correction module is designed to model the interdependencies between global and local features in the input feature mapping channel direction and spatial direction, respectively; for example... Figure 4 As shown, in this self-correction module, the upper branch models the dependencies between global and local features in the channel direction through global average pooling, 3×3 convolution, fully connected operations, and Sigmoid operations; the lower branch models the dependencies between global and local features in the spatial direction through global average pooling, 1×1 convolution, 3×3 convolution, and Sigmoid operations in the height and width directions; finally, the output features of the upper and lower branches are concatenated and a 1×1 convolution operation is used to obtain the final output feature of the self-correction module.

[0106] The output features obtained from these three different path branches are concatenated, and then global average pooling, full connection, and Sigmoid operations are performed to obtain the weight matrix.

[0107] Finally, the weight matrix and the output features of the three different path branches are multiplied to obtain the weighted output features of the three different path branches. These features are then fused to further obtain regionally discernible haze-related feature information; the fusion process is as follows:

[0108]

[0109] Where c represents the total feature dimension; F kThis represents the output feature of any branch. This represents the feature dimension of the i-th channel in the k-th branch. The fusion features are represented by u and v, where u and v represent the height and width components of the features, and h and w represent the height and width of the output features. After concatenating the features obtained from the three different paths using global average pooling along the channel direction, the final fusion feature is obtained.

[0110]

[0111] Where FC represents the fully connected convolution operation in the network; g c This represents a 1×1 convolution operation; This is a multiplication operation; It is the value of the i-th channel of the weight w. i The matrix formed by expansion; This represents the final output feature of the haze positioning module, namely the haze-related features that indicate regional identifiability.

[0112] S2-2, Constructing the unentanglement module:

[0113] The de-entanglement module removes the dependency between essential features and irrelevant features by utilizing the random features in the Gaussian kernel Fourier domain and the characteristics of sample weighting.

[0114] Let Y be the set of output features of the last layer of the encoder in the dehazing network. To make any pair of features y in Y... ：，i y ：，j If unrelated, then y ：，i y ：，j The cross-covariance between y ：，i y ：，j ) = 0; To reduce computational overhead, the Hibert-Schmidt norm of Euclidean space is converted to the Frobenius norm for solution, and the random characteristics in the Fourier domain of the Gaussian kernel function are used for approximate estimation. Therefore, the above y ：，i y ：，j The cross-covariance between y ：，i y ：，j Rephrased as:

[0115]

[0116] in,

[0117]

[0118] in, for h n (·), f n(·) represents the feature values ​​sampled from the random Fourier feature function space; the number of samples n = 5;

[0119] By introducing stochastic Fourier features and learning a weighted matrix based on them, global decorrelation and standardization are performed on the data to remove the dependency between haze-free background features and haze. Furthermore, the cross-covariance (Cov) of paired features is minimized to make them independent, thus achieving haze deentanglement. Represents sample weights and For any paired feature y ：，i y ：，j The weighted cross-covariance matrix The calculation is performed using the following formula:

[0120]

[0121] in This represents the sample weight of the i-th paired feature. Let represent the sample weight of the j-th paired feature; further, it is expressed by the following formula... Optimize to obtain the optimized weights. Essential weight:

[0122]

[0123] in Represents the sample weights obtained through optimization. The minimum value in, and c represents the total feature dimension; This represents the 2-norm operation.

[0124] S2-3, Based on steps S1, S2-1, and S2-2 above, construct the overall dehazing network, as follows: Figure 5 As shown in the diagram, the encoder and decoder are any existing high-efficiency, lightweight encoder-decoder networks. The haze localization module outputs discriminative haze-related feature information, while the de-entanglement module obtains essential weights unrelated to haze and constructs an adaptive loss to constrain network training, thereby achieving de-entanglement of haze and haze-free features. Negative samples are the input haze images, and positive samples are the corresponding clear label images. These two, along with the predicted haze-free output, are input into the encoder constructed using VGG-19 to construct the contrastive loss.

[0125] Step S3 is as follows:

[0126] Use optimized weights That is, the training samples are essentially weighted, and the training of the dehazing network is guided by the following adaptive loss:

[0127]

[0128] And with

[0129]

[0130] Where x i Represents a haze image, J i The label represents a clear image, and t represents the timestamp for each batch. This represents the predicted fog-free image output; initial weight values. y ：，i y ：，j Represents any pair of features in the feature set Y;

[0131] Furthermore, a contrastive loss function is designed to constrain the haze-free image reconstructed by the network to be closer to the clear sample image and further away from the original haze image; its objective loss function is expressed in the following form:

[0132]

[0133] Among them, g k This represents the k-th hidden layer feature extracted from the VGG-19 model with fixed pre-trained weights; J represents the clear labeled image, and I represents the original haze image; Indicates a positive predicted value. Indicates a negative predicted value; This represents the predicted fog-free image output; further, the contrast loss is defined as follows:

[0134]

[0135] Where α represents the penalty parameter, λ k Represents the weighting coefficient;

[0136] The total loss function can be summarized in the following form:

[0137] Loss = Loss a +Loss b .

[0138] Step S4 is as follows:

[0139] The test dataset comes from real-world haze data. During testing, haze images of any size are input into the dehazing network in batches.

[0140] By loading the weights of the dehazing network trained on the LDHaze dataset, the haze images in the test dataset are successively input into the dehazing network to obtain the predicted haze-free image output.

[0141] The quality of the predicted haze-free output image is evaluated using no-reference-map quality metrics such as PM2.5, FADE, and PIQE. If the performance is good on these metrics, the network parameters are fixed to form the final dehazing network based on haze localization and disentanglement. If the performance does not meet the requirements on the aforementioned no-reference-map evaluation metrics, the model training weights are changed and the test is repeated until the network model performs well on multiple authoritative no-reference-map metrics.

[0142] Accordingly, the present invention also provides an image dehazing system based on haze localization and deentanglement, such as... Figure 6 As shown, it includes:

[0143] Training data construction unit 1 is used to construct the haze image dataset LDHaze as training data;

[0144] Dehazing network construction unit 2 is used to construct a dehazing network, which includes an encoder, a haze localization module, a deentanglement module, and a decoder. The encoder extracts features from the haze image, which are then input into the haze localization module and the deentanglement module. The haze localization module outputs haze-related feature information that indicates the region's identifiability. The deentanglement module obtains essential weights unrelated to haze, i.e., haze-free background feature weights, which are used to weight the training data and guide the training of the dehazing network by constructing an adaptive loss. The decoder decodes the output features of the haze localization module to obtain the predicted dehazing image.

[0145] Dehazing network training unit 3 is used to design the loss function and train the dehazing network using the haze image dataset LDHaze to form a trained dehazing network based on haze localization and disentanglement.

[0146] Test application unit 4 is used to test the trained haze-based dehazing network using a no-reference-map quality evaluation index. When the performance is good on the no-reference-map quality evaluation index, the network parameters are fixed to form the final haze-based dehazing network. Its input is any haze image and its output is the image dehazing result.

[0147] Therefore, this invention introduces a haze localization module and uses a weighted summation method for feature fusion. Unlike many dehazing methods that simply splice features, this invention has the following advantages: 1) Learning weights helps the module adaptively adjust the importance of feature information extracted by the three parallel paths, enhancing the network's ability to distinguish and learn haze of different densities and depths; 2) Direct splicing operations will multiply the number of feature channels, while using positional summation will maintain the original number of channels and will not increase the computational load.

[0148] This invention introduces a haze deentanglement module, which directly removes the dependency between haze and non-haze background features in the relevant feature space by learning weights from training samples, thereby helping the dehazing model focus on learning the essential connection between discriminative features and clear labels.

[0149] The perceptual contrast loss constructed in this invention can bring the distance between the predicted fog-free output and the clear label closer in the implicit distance space, and push the distance between the predicted fog-free output and the hazy image further away, so that the overall network training is updated in the direction of natural dehazing.

[0150] This invention can remove various types of haze. Compared with existing image dehazing methods, it has significant advantages in dehazing quality evaluation, stability, accuracy, and speed. In this invention, the inputs and outputs of different parts of the network are interconnected, extracting and integrating different features, improving efficiency and resulting in a more natural restored haze-free image.

[0151] The foregoing has provided a detailed description of an image dehazing method and system based on haze localization and deentanglement provided by embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An image dehazing method based on haze localization and deentanglement, characterized in that, The method includes: Construct a haze image dataset LDHaze as training data; A dehazing network is constructed, comprising an encoder, a haze localization module, a disentanglement module, and a decoder. The encoder extracts features from the haze image, which are then input into the haze localization and disentanglement modules. The haze localization module outputs haze-related feature information that indicates the region's identifiability. The disentanglement module obtains essential weights unrelated to haze, i.e., haze-free background feature weights, which are used to weight the training data and guide the training of the dehazing network through an adaptive loss. The decoder decodes the output features of the haze localization module to obtain the predicted dehazed image. Design a loss function and train the dehazing network using the haze image dataset LDHaze to form a trained dehazing network based on haze localization and disentanglement. The trained dehazing network based on haze localization and disentanglement is tested using a no-reference-map quality evaluation index. When the network performs well on the no-reference-map quality evaluation index, the network parameters are fixed to form the final dehazing network based on haze localization and disentanglement. Its input is any haze image, and its output is the image dehazing result.

2. The image dehazing method based on haze localization and deentanglement as described in claim 1, characterized in that, The LDHaze image dataset was constructed as training data, specifically as follows: The sharp labels used to synthesize haze images come from sharp images and corresponding depth image information in the RESIDE dataset. The construction process of the LDHaze haze image dataset is described by the following formula: Where I represents the haze image to be synthesized, and F filter (·) represents the blur kernel function; x, y represent the pixel coordinates, and x′, y′ are the pixel coordinate shift increments; The initial haze image is represented; γ is a constraint factor used to adjust the brightness of the generated haze data image; A is the atmospheric light value; t represents the scene transmittance map; R represents the clear image; ε is the noise term.

3. The image dehazing method based on haze localization and deentanglement as described in claim 1, characterized in that, The haze positioning module is specifically as follows: The overall process of the haze positioning module is as follows: First, the output features of the last layer of the encoder in the defogging network are explicitly mapped using a random Fourier explicit mapping function. Second, the explicit mapping result is input into three different path branches to obtain path output features. These three path branches are composed of dilated convolutions with different dilation rates. The output features obtained from these three different path branches are concatenated, and then global average pooling, fully connected layers, and Sigmoid operations are performed to obtain a weight matrix. Finally, the weight matrix is ​​multiplied by the output features of the three path branches to obtain weighted output features of the three different path branches. These output features are then concatenated again and subjected to a 1×1 convolution operation to obtain the final output features of the haze localization module, namely, the haze-related features with regional discernibility. The specific process of the smog positioning module is as follows: First, the random mapping features in the Gaussian kernel Fourier domain are used to represent the high-order nonlinear feature information in the output features of the last layer of the encoder in a low-dimensional explicit manner: Let the output feature of the last layer of the encoder be . Estimate a given Gaussian kernel G on each channel using a low-dimensional explicit mapping function z(y). y The formula is as follows: Among them G y Represents the Gaussian kernel, y a y b These represent the components of feature y; α represents the size of the Gaussian kernel; z represents the value used to estimate a given Gaussian kernel G(y). a ,y b The low-dimensional explicit mapping function of ); according to Bochner's theorem, if the Gaussian kernel G(y) has translation invariance a ,y b In the original implicit space dimension Positive definiteness is true if and only if G y It is the Fourier transform of some nonnegative measure p(s); thus, we obtain the following formula: Where p(s) represents the nonnegativity measure, i represents the imaginary part, and D represents the feature dimension; bias parameter b d ∈[b1…b D From a uniform distribution U [0,2π] Obtained from; From nonnegative measure Weight parameters obtained from sampling; Based on the above formula, the random Fourier features of the output features of the last layer of the encoder are obtained. The form of expression: Secondly, three parallel paths were designed to capture haze-related feature information of different haze densities and depths in different regions of the image. These three paths were constructed based on dilated convolution, which can extract features from receptive fields of different sizes while maintaining image resolution and reducing the number of parameters. In each of the three paths, two 3×3 dilated convolution layers were used with dilation rates of 1, 3, and 5, respectively. The receptive field sizes of the first dilated convolution layer were 3×3, 7×7, and 11×11, respectively. The receptive field sizes of the second dilated convolution layer were 5×5, 13×13, and 21×21, respectively. Considering that the haze-related feature information obtained through these three feature extraction paths still comes from local areas, a self-correction module is designed to model the interdependencies between global and local features in the channel direction and spatial direction of the input feature mapping, respectively. In this self-correction module, the upper branch models the interdependencies between global and local features in the channel direction through global average pooling, 3×3 convolution operation, fully connected operation and Sigmoid operation in the channel direction. The lower branch models the dependencies between global and local features in the spatial direction through global average pooling, 1×1 convolution, 3×3 convolution, and Sigmoid operations in the height and width directions; finally, the output features of the upper and lower branches are concatenated and a 1×1 convolution operation is used to obtain the output features of the final self-correction module. The output features obtained from these three different path branches are concatenated, and then global average pooling, full connection, and Sigmoid operations are performed to obtain the weight matrix. Finally, the weight matrix and the output features of the three different path branches are multiplied to obtain the weighted output features of the three different path branches. These features are then fused to further obtain regionally discernible haze-related feature information. The fusion process is as follows: Where c represents the total feature dimension; F k This represents the output feature of any branch. This represents the feature dimension of the i-th channel in the k-th branch. The fusion features are represented by u and v, where u and v represent the height and width components of the features, and h and w represent the height and width of the output features. After concatenating the features obtained from the three different paths using global average pooling along the channel direction, the final fusion feature is obtained. Where FC represents the fully connected convolution operation in the network; g c This represents a 1×1 convolution operation; This is a multiplication operation; It is the value of the i-th channel of the weight w. i The matrix formed by expansion; This represents the final output feature of the haze positioning module, namely the haze-related features that indicate regional identifiability.

4. The image dehazing method based on haze localization and deentanglement as described in claim 1, characterized in that, The de-entanglement module specifically comprises: The de-entanglement module removes the dependency between essential features and irrelevant features by utilizing the random features in the Gaussian kernel Fourier domain and the characteristics of sample weighting. Let Y be the set of output features of the last layer of the encoder in the dehazing network. To make any pair of features y in Y... :,i y :,j If unrelated, then y :,i y :,j The cross-covariance between y :,i y :,j ) = 0; To reduce computational overhead, the Hibert-Schmidt norm of Euclidean space is converted to the Frobenius norm for solution, and the random characteristics in the Fourier domain of the Gaussian kernel function are used for approximate estimation. Therefore, the above y :,i y :,j The cross-covariance between y :,i y :,j Rephrased as: in, in, for n represents the feature values ​​sampled from the random Fourier feature function space; n is the number of samples. By introducing stochastic Fourier features and learning a weighted matrix based on them, global decorrelation and standardization are performed on the data to remove the dependency between haze-free background features and haze. Furthermore, the cross-covariance (Cov) of paired features is minimized to make them independent, thus achieving haze deentanglement. Represents sample weights and For any paired feature y :, i, y :, j, the weighted cross-covariance matrix The calculation is performed using the following formula: in This represents the sample weight of the i-th paired feature. Let represent the sample weight of the j-th paired feature; further, it is expressed by the following formula... Optimize to obtain the optimized weights. That is, the essential weight: in Represents the sample weights obtained through optimization. The minimum value in, and c represents the total feature dimension; This represents the 2-norm operation.

5. The image dehazing method based on haze localization and deentanglement as described in claim 4, characterized in that, The design loss function is specifically as follows: Use the optimized weights That is, the essential weights are used to weight the training samples, and the training of the dehazing network is guided by constructing the following adaptive loss: And with Where x i Represents a haze image, J i The label represents a clear image, and t represents the timestamp for each batch. This represents the predicted fog-free image output; initial weight values. y :,i y :,j This represents any pair of features in the feature set Y; Furthermore, a contrastive loss function is designed to constrain the haze-free image reconstructed by the network to be closer to the clear sample image and further away from the original haze image; its objective loss function is expressed in the following form: Among them, g k This represents the k-th hidden layer feature extracted from the VGG-19 model with fixed pre-trained weights. J represents a clear labeled image, and I represents the original haze image; Indicates a positive predicted value. Indicates a negative predicted value; This represents the predicted fog-free image output; further, the contrast loss is defined as follows: Where α represents the penalty parameter and λk represents the weight coefficient; The total loss function can be summarized in the following form: Loss=Loss a +Loss b 。 6. The image dehazing method based on haze localization and deentanglement as described in claim 1, characterized in that, The trained dehazing network based on haze localization and disentanglement is tested using a no-reference-graph quality evaluation metric. When the performance on the no-reference-graph quality evaluation metric is good, the network parameters are fixed to form the final dehazing network based on haze localization and disentanglement. Specifically: The test dataset comes from real-world haze data. During testing, haze images of any size are input into the dehazing network in batches. By loading the weights of the dehazing network trained on the LDHaze dataset, the haze images in the test dataset are successively input into the dehazing network to obtain the predicted haze-free image output; The quality of the predicted fog-free output image is evaluated using a no-reference-map quality evaluation index. If the performance is good on the no-reference-map quality evaluation index, the network parameters are fixed to form the final defogging network based on fog localization and disentanglement.

7. An image dehazing system based on haze localization and deentanglement, characterized in that, The system includes: The training data construction unit is used to construct the haze image dataset LDHaze as training data. A dehazing network construction unit is used to construct a dehazing network, which includes an encoder, a haze localization module, a deentanglement module, and a decoder. The encoder extracts features from the haze image, which are then input into the haze localization module and the deentanglement module. The haze localization module outputs haze-related feature information that indicates the region's identifiability. The deentanglement module obtains essential weights unrelated to haze, i.e., haze-free background feature weights, which are used to weight the training data and guide the training of the dehazing network by constructing an adaptive loss. The decoder decodes the output features of the haze localization module to obtain the predicted dehazing image. The dehazing network training unit is used to design the loss function and train the dehazing network using the haze image dataset LDHaze to form a trained dehazing network based on haze localization and disentanglement. The test application unit is used to test the trained haze-based dehazing network using a no-reference-map quality evaluation index. When the network performs well on the no-reference-map quality evaluation index, the network parameters are fixed to form the final haze-based dehazing network. Its input is an arbitrary haze image, and its output is the image dehazing result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images