Image defogging method and system based on haze positioning and deentanglement

Abstract

The invention discloses an image defogging method and system based on haze positioning and deentanglement. Constructing training data and a defogging network, wherein the defogging network comprises an encoder, a haze positioning module, a deentanglement module and a decoder; the haze positioning module is used for outputting regional discernible haze-related feature information, and the deentanglement module is used for guiding the training of a defogging network; designing a loss function and training the defogging network by using the training data to form a trained defogging network; and testing the trained defogging network by using a no-reference image quality evaluation index, and when the performance is good, fixing network parameters to form a final defogging network. The haze positioning module and the deentanglement module are introduced, various types of haze can be removed, compared with an existing image defogging method, the method has obvious advantages in the aspects of defogging quality evaluation, defogging stability, defogging accuracy and defogging rapidness, and a restored fogless image is more natural.

Description

Technical Field

[0001] The present invention relates to the fields of image processing and computer vision, and in particular to an image defogging method and system based on haze positioning and disentanglement. Background Art

[0002] Severe weather phenomena such as haze and smoke pollution generally cause reduced visibility, which seriously interferes with the imaging quality of the visual system. For visual tasks that rely on image information, such as autonomous driving assistance, near-shore ocean monitoring, and drone aerial photography, image quality degradation caused by haze pollution may pose serious safety risks. In order to eliminate haze interference, the academic community has conducted in-depth research on visual restoration methods for degraded images. Although many representative dehazing methods have been proposed, the removal of haze in real scenes still faces great challenges.

[0003] Affected by environmental factors such as atmospheric light intensity and wind speed changes, the distribution of haze in natural scenes often presents a non-uniform state. These non-uniformly distributed haze will interfere with the acquisition of edge features and texture information in the image to varying degrees, resulting in a decrease in image quality. In order to restore clear images from natural haze images, previous works use heuristically designed local or global priors to focus on the foggy areas in the image and model them to restore clear images. However, dehazing methods based on handcrafted priors often require ideal data assumptions and strong robust prior information constraints. Due to the non-uniformity of haze distribution in real scenes, when the actual prior information obtained deviates from the perfect data assumption, the dehazing effect is poor. Different from dehazing methods based on heuristically designed priors, many dehazing methods based on deep learning estimate physical model parameters or directly learn the mapping relationship between foggy images and clear images by designing deep and complex network structures. However, due to the complexity of haze distribution in the real world, these methods still have a lot of room for optimization when processing natural haze images with different densities and depths.

[0004] One of the current existing technologies is a real image dehazing method based on high-quality codebook priors proposed in the paper "RIDCP: Revitalizing Real Image Dehazing via High-Quality Codebook Priors", which proposes a new paradigm for real image dehazing from the perspective of synthesizing more realistic haze data and introducing more robust priors in the network. This technology first proposes a haze data generation pipeline containing multiple different degradation factors based on the traditional atmospheric light scattering model; secondly, this method pre-trains a VQGAN on a large-scale high-quality dataset to obtain a discrete codebook containing high-quality priors (HQPs). After replacing the negative effects of haze with HQPs, the decoder equipped with a novel normalized feature alignment module can effectively utilize high-quality features and produce clear results. The disadvantage of this technology is that due to the use of a relatively complex deep network structure, excellent dehazing performance indicators are obtained on synthetic datasets and real haze datasets in specific scenarios, but there are also large model parameters and high computational complexity, which cannot quickly remove haze in complex open scenes in the real world in real time.

[0005] The second existing technology is an integrated scene restoration framework based on two neural modules proposed in the paper "All-In-One Image Restoration for Unknown Corruption". This technology restores degraded images based on a contrasting degradation encoder and a degradation guidance module respectively. This method is not affected by the prior of the type and degree of degradation, and only uses the observed degraded images for inference. This advantage makes this method more flexible and more economical in real-world scenes. The disadvantage of this technology is that for haze types such as night, dust, and color interference, it is impossible to effectively defog due to the lack of more prior knowledge. In addition, although this technology has better performance and faster speed than many existing deep defogging methods on synthetic data sets, it still does not have the generalization and meet the real-time requirements when dealing with real complex haze. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the existing methods and propose an image defogging method and system based on haze positioning and disentanglement. The main problem solved by the present invention is to design a haze positioning module and a disentanglement strategy to form a lightweight network, so as to achieve lightweight model parameters and low computational complexity while ensuring the generalization and robustness of defogging.

[0007] In order to solve the above problems, the present invention proposes an image defogging method based on haze positioning and disentanglement, the method comprising:

[0008] Construct the haze image dataset LDHaze as training data;

[0009] Constructing a defogging network, the defogging network includes an encoder, a haze positioning module, a disentanglement module, and a decoder; wherein the encoder is used to extract features of a haze image, and the features of the haze image are input into the haze positioning module and the disentanglement module; the haze positioning module is used to output haze-related feature information of regional discernibility, and the disentanglement module is used to obtain essential weights unrelated to haze, that is, weights of haze-free background features, which are used to weight the training data and guide the training of the defogging network by constructing an adaptive loss; the decoder is used to decode the output features of the haze positioning module to obtain a predicted defogging image;

[0010] Designing a loss function, and using the haze image data set LDHaze to train the dehazing network, to form a trained dehazing network based on haze positioning and disentanglement;

[0011] The trained dehazing network based on haze positioning and disentanglement is tested using a no-reference image quality evaluation indicator. When the performance is good on the no-reference image quality evaluation indicator, the network parameters are fixed to form a final dehazing network based on haze positioning and disentanglement, whose input is an arbitrary haze image and output is an image dehazing result.

[0012] Preferably, the construction of the haze image dataset LDHaze as training data is specifically as follows:

[0013] The clear labels used to synthesize haze images come from 2061 clear images and corresponding depth image information in the RESIDE dataset. The construction process of the haze image dataset LDHaze is described by the following formula:

[0014]

[0015] The formula is a normal distribution in the interval [0.0001, 5]; I represents the haze image to be synthesized, and F filter (·) represents the blur kernel function; x, y represent the pixel coordinates, x′, y′ are the movement increments of the pixel coordinates; represents the generated initial haze image; γ is the constraint factor used to adjust the brightness of the generated haze data image, and is obtained by uniform sampling in the interval [0.2, 0.8]; A is the atmospheric light value, randomly selected from the interval [0.75, 0.9]; t represents the scene transmittance map, R represents the clear image; ε is the noise term, which obeys the mean of 0.

[0016] Preferably, the haze positioning module is specifically:

[0017] The overall process of the haze positioning module is as follows:

[0018] First, the output features of the last layer of the encoder in the defogging network are explicitly mapped by a random Fourier explicit mapping function; secondly, the explicit mapping results are input into three different path branches to obtain path output features, and the three path branches are composed of dilated convolutions with different dilated rates. The output features obtained from the three different path branches are spliced, and then global average pooling, full connection and Sigmoid operations are performed to obtain a weight matrix; finally, the weight matrix and the output features of the three path branches are multiplied to obtain the weighted output features of the three different path branches, and these output features are spliced ​​again and the 1×1 convolution operation is performed to obtain the output features of the final haze positioning module, that is, the haze-related features of regional discernibility;

[0019] The specific process of the haze positioning module is as follows:

[0020] First, the random mapping features in the Gaussian kernel Fourier domain are used to explicitly represent the high-order nonlinear feature information in the output features of the last layer of the encoder in a low-dimensional manner:

[0021] The output feature of the last layer of the encoder is recorded as A low-dimensional explicit mapping function z(y) is used to estimate a given Gaussian kernel G on each channel. y , the formula is as follows:

[0022]

[0023] Among them G y represents the Gaussian kernel, y a ,y b They represent the components of feature y; α represents the size of the Gaussian kernel; z represents the number of features used to estimate the given Gaussian kernel G(y a ,y b ) is a low-dimensional explicit mapping function; According to Bochner's theorem, if the Gaussian kernel G(y a ,y b ) in the original implicit space dimension Upper positive definite if and only if G y is the Fourier transform of a non-negative measure p(s); thus we get the following formula:

[0024]

[0025] Where p(s) represents a non-negative measure, i represents the imaginary part, and D represents the feature dimension; Bias parameter b d ∈[b1…b D ]From the uniform distribution U [0，2π] obtained in; From the non-negative measure The weight parameters obtained by sampling;

[0026] Based on the above formula, the random Fourier features of the output features of the last layer of the encoder are obtained The expression form:

[0027]

[0028] Secondly, three parallel paths are designed to capture the haze-related feature information of different haze densities and depths in different areas of the image; these three paths are built based on dilated convolutions, which can extract features from receptive fields of different sizes while maintaining image resolution and reducing the number of parameters; in the three paths, two layers of 3×3 dilated convolutions are used and the dilation rates are set to 1, 3, and 5 respectively; the receptive field sizes of the first dilated convolution layer are 3×3, 7×7, and 11×11 respectively; the receptive field sizes of the second dilated convolution layer are 5×5, 13×13, and 21×21 respectively;

[0029] Considering that the haze-related feature information obtained through these three feature extraction paths still comes from the local area, a self-correction module is designed to model the mutual dependence between the global and local features in the channel direction and the spatial direction of the input feature map respectively; in this self-correction module, the upper branch models the dependency between the global and local features in the channel direction through global average pooling, 3×3 convolution operation, full connection operation and Sigmoid operation in the channel direction; the lower branch models the dependency between the global and local features in the spatial direction through global average pooling, 1×1 convolution operation, 3×3 convolution operation and Sigmoid operation in the height and width direction; finally, the output features of the upper branch and the lower branch are spliced ​​and the output features of the final self-correction module are obtained through 1×1 convolution operation;

[0030] The output features obtained from the three different path branches are concatenated, and then global average pooling, full connection and Sigmoid operations are performed to obtain the weight matrix;

[0031] Finally, the weight matrix and the output features of the three different path branches are multiplied to obtain the weighted output features of the three different path branches, which are then fused to further obtain regionally discernible haze-related feature information; the fusion process is as follows:

[0032]

[0033] Among them, c represents the total feature dimension; F k represents the output feature of any branch, represents the i-th channel feature dimension of the k-th branch, represents the fusion feature, u, v represent the feature height and width components, h, w represent the height and width of the output feature; after the features obtained from the three different paths are spliced ​​using global average pooling in the channel direction, the final fusion feature is obtained:

[0034]

[0035] Among them, FC represents the network fully connected convolution operation; gc represents 1×1 convolution operation; is the multiplication operation; is the value w of the i-th channel with weight w i The matrix formed by expansion; It represents the final output features of the haze positioning module, that is, the haze-related features of regional discernibility.

[0036] Preferably, the detangling module is specifically:

[0037] The disentanglement module removes the dependencies between essential features and irrelevant features by utilizing the random features and sample weighting characteristics in the Gaussian kernel Fourier domain;

[0038] The set of output features of the last layer of the encoder in the defogging network is denoted as Y. To make any pair of features y in Y :，i ,y :，j Not relevant, then y :，i ,y :，j The cross-covariance cov(y :，i y :，j )=0; To reduce the computational overhead, the HibertSchmidt norm of the Euclidean space is converted to the Frobenius norm to solve, and the random features in the Fourier domain of the Gaussian kernel function are used for approximate estimation. Therefore, the above y :，i ,y :，j The cross-covariance cov(y :，i y :，j ) can be restated as:

[0039]

[0040] in,

[0041]

[0042] in, for h n (·), fx(·) is the eigenvalue sampled from the random Fourier eigenfunction space; the number of samples n = 5;

[0043] Random Fourier features are introduced and a weighting matrix is ​​learned based on them. The data is globally decorrelated and standardized to remove the dependency between the fog-free background features and the haze. The two are made independent by minimizing the mutual covariance Cov of the paired features, thereby achieving haze disentanglement. represents the sample weight and For any pair of features y ：，i ,y ：，j , the weighted cross-covariance matrix Calculate using the following formula:

[0044]

[0045] in represents the sample weight of the i-th paired feature, represents the sample weight of the jth paired feature; further, the following formula is used to Optimize and get the optimized weight That is, the essential weight:

[0046]

[0047] in Represents the sample weight obtained by optimization The minimum value in and c represents the total feature dimension; Represents the 2-norm operation.

[0048] Preferably, the design loss function is specifically:

[0049] Use the optimization weights That is, the essential weights are used to weight the training samples, and the training of the dehazing network is guided by constructing the following adaptive loss:

[0050]

[0051] And with

[0052]

[0053] where x i represents the haze image, J i represents the clear image label, t represents the timestamp of each batch, Represents the predicted haze-free image output; initial weight value y ：，i ,y ：，j represents any paired features in the feature set Y;

[0054] In addition, a contrast loss function is designed to constrain the network to restore the haze-free image closer to the clear sample image and further away from the original haze image; its target loss function is expressed as follows:

[0055]

[0056] Among them, g k represents the kth hidden layer feature extracted from the fixed pre-trained weight model VGG-19; J represents the clear label image, and I represents the original haze image; represents a positive prediction value, represents a negative predicted value; Represents the predicted haze-free image output; further, the contrast loss is defined as follows:

[0057]

[0058] Where α represents the penalty parameter, λ k represents the weight coefficient;

[0059] The total loss function is summarized as follows:

[0060] Loss=Loss a +Loss b .

[0061] Preferably, the trained defogging network based on haze positioning and detangling is tested using the no-reference image quality evaluation index. When the performance is good on the no-reference image quality evaluation index, the network parameters are fixed to form the final defogging network based on haze positioning and detangling, specifically:

[0062] The test data set comes from haze data in real scenes. During the test, haze images of any size are input into the dehazing network in batches.

[0063] Load the defogging network weights trained on the LDHaze dataset, and input the haze images in the test dataset into the defogging network one by one to obtain the predicted haze-free image output;

[0064] The quality of the predicted haze-free output image is evaluated using the no-reference image quality evaluation indicators PM2.5, FADE, and PIQE. If the performance is good on the above-mentioned no-reference image quality evaluation indicators, the network parameters are fixed to form the final defogging network based on haze positioning and disentanglement.

[0065] Accordingly, the present invention also proposes an image defogging system based on haze positioning and disentanglement, comprising:

[0066] A training data construction unit is used to construct a haze image dataset LDHaze as training data;

[0067] A defogging network construction unit is used to construct a defogging network, which includes an encoder, a haze positioning module, a disentanglement module, and a decoder; wherein the encoder is used to extract features of a haze image, and the features of the haze image are input into the haze positioning module and the disentanglement module; the haze positioning module is used to output haze-related feature information of regional discernibility, and the disentanglement module is used to obtain essential weights that are irrelevant to haze, that is, weights of haze-free background features, which are used to weight the training data and guide the training of the defogging network by constructing an adaptive loss; the decoder is used to decode the output features of the haze positioning module to obtain a predicted defogging image;

[0068] A defogging network training unit, used for designing a loss function and training the defogging network using the haze image data set LDHaze to form a trained defogging network based on haze positioning and disentanglement;

[0069] The test application unit is used to test the trained defogging network based on haze positioning and disentanglement using a no-reference image quality evaluation index. When the performance is good on the no-reference image quality evaluation index, the network parameters are fixed to form a final defogging network based on haze positioning and disentanglement, whose input is an arbitrary haze image and whose output is an image defogging result.

[0070] The implementation of the present invention has the following beneficial effects:

[0071] The present invention introduces a haze positioning module and adopts a weighted addition method to perform feature fusion. Unlike many defogging methods that simply splice features directly, the present invention has the following advantages: 1) Learning weights helps the module to adaptively adjust the importance of feature information extracted by three parallel paths, thereby enhancing the network's ability to distinguish and learn haze of different densities and depths; 2) Direct splicing operations will increase the number of feature channels exponentially, while using a positional addition operation will maintain the original number of channels and will not increase the amount of calculation.

[0072] The present invention introduces a haze disentanglement module, which directly decorrelates the dependency between haze-free background features and fog in feature space by learning weights of training samples, thereby helping the dehazing model to focus on learning the essential connection between discriminative features and clear labels.

[0073] The perceptual contrast loss constructed by the present invention can shorten the distance between the predicted haze-free output and the clear label in the implicit distance space, and increase the distance between the predicted haze-free output and the haze image, so that the overall network training is updated in the natural direction of dehazing.

[0074] The present invention can achieve the removal of various types of haze. Compared with the existing image defogging methods, it has obvious advantages in defogging quality evaluation, defogging stability, defogging accuracy, and defogging speed. In the invention, the input and output between the various parts of the network are interconnected, and different features are extracted and integrated, which improves efficiency and makes the restored haze-free image more natural. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 is an overall flow chart of an image defogging method based on haze positioning and disentanglement according to an embodiment of the present invention;

[0076] Figure 2 is a structural diagram of a haze positioning module according to an embodiment of the present invention;

[0077] Figure 3 : is a structural diagram of an extraction path for extracting regional identifiable haze-related feature information according to an embodiment of the present invention;

[0078] Figure 4 is a structural diagram of a self-correction module according to an embodiment of the present invention;

[0079] Figure 5 is a general structural diagram of a defogging network according to an embodiment of the present invention;

[0080] Figure 6 It is a structural diagram of an image defogging system based on haze positioning and disentanglement according to an embodiment of the present invention. DETAILED DESCRIPTION

[0081] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0082] Figure 1 is an overall flow chart of an image defogging method based on haze positioning and disentanglement according to an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0083] S1, construct the haze image dataset LDHaze as training data;

[0084] S2, constructing a defogging network, which includes an encoder, a haze positioning module, a disentanglement module, and a decoder; wherein the encoder is used to extract the features of the haze image, and the features of the haze image are input into the haze positioning module and the disentanglement module; the haze positioning module is used to output haze-related feature information of regional discernibility, and the disentanglement module is used to obtain essential weights that are irrelevant to haze, that is, the weights of haze-free background features, which are used to weight training data and guide the training of the defogging network by constructing an adaptive loss; the decoder is used to decode the output features of the haze positioning module to obtain a predicted defogging image;

[0085] S3, design the loss function and use the haze image dataset LDHaze to train the dehazing network to form a trained dehazing network based on haze positioning and disentanglement;

[0086] S4, use the no-reference image quality evaluation index to test the trained dehazing network based on haze positioning and disentanglement. When the performance on the no-reference image quality evaluation index is good, fix the network parameters to form the final dehazing network based on haze positioning and disentanglement. Its input is any haze image and its output is the image dehazing result.

[0087] Step S1 is as follows:

[0088] The clear labels used to synthesize haze images come from 2061 clear images and corresponding depth image information in the RESIDE dataset. The construction process of the haze image dataset LDHaze is described by the following formula:

[0089]

[0090] The formula is a normal distribution in the interval [0.0001, 5]; I represents the haze image to be synthesized, and F filter (·) represents the blur kernel function; x, y represent the pixel coordinates, x′, y′ are the movement increments of the pixel coordinates; represents the generated initial haze image; γ is the constraint factor used to adjust the brightness of the generated haze data image, and is obtained by uniform sampling in the interval [0.2, 0.8]; A is the atmospheric light value, randomly selected from the interval [0.75, 0.9]; t represents the scene transmittance map, R represents the clear image; ε is the noise term, which obeys the mean of 0.

[0091] Step S2 is as follows:

[0092] S2-1, construct haze positioning module:

[0093] like Figure 2 As shown in the figure, the overall process of the haze positioning module is as follows:

[0094] First, the output features of the last layer of the encoder in the dehazing network are explicitly mapped by a random Fourier explicit mapping function; secondly, the explicit mapping results are input into three different path branches to obtain the path output features. The three path branches are composed of dilated convolutions with different dilation rates. The output features obtained from the three different path branches are concatenated, and then global average pooling, full connection and Sigmoid operations are performed to obtain the weight matrix; finally, the weight matrix and the output features of the three path branches are multiplied to obtain the weighted output features of the three different path branches, and these output features are concatenated again and the 1×1 convolution operation is performed to obtain the output features of the final haze positioning module, that is, the haze-related features of regional discernibility;

[0095] The specific process of the haze positioning module is as follows:

[0096] First, the random mapping features in the Gaussian kernel Fourier domain are used to explicitly represent the high-order nonlinear feature information in the output features of the last layer of the encoder in a low-dimensional manner:

[0097] The output feature of the last layer of the encoder is A low-dimensional explicit mapping function z(y) is used to estimate a given Gaussian kernel G on each channel. y , the formula is as follows:

[0098]

[0099] Among them G y represents the Gaussian kernel, y a ,y b They represent the components of feature y respectively; α represents the size of the Gaussian kernel; z represents the number of features used to estimate the given Gaussian kernel G(y a ,y b ) is a low-dimensional explicit mapping function; According to Bochner's theorem, if the Gaussian kernel G(y a ,y b ) in the original implicit space dimension Upper positive definite if and only if G y is the Fourier transform of a non-negative measure p(s); thus we get the following formula:

[0100]

[0101] Where p(s) represents a non-negative measure, i represents the imaginary part, and D represents the feature dimension; Bias parameter b d ∈[b1…b D ]From the uniform distribution U [0，2π] obtained in; From the non-negative measure The weight parameters obtained by sampling;

[0102] Based on the above formula, the random Fourier features of the output features of the last layer of the encoder are obtained The expression form:

[0103]

[0104] Secondly, three parallel paths are designed to capture the haze-related feature information of different haze densities and depths in different regions of the image. These three paths are constructed based on dilated convolution, which can extract features from receptive fields of different sizes while maintaining image resolution and reducing the number of parameters. The structure of any extraction path for extracting regional identifiable haze-related feature information is as follows: Figure 3 As shown in the figure; in the three paths, two layers of 3×3 dilated convolutions are used and the dilation rates are set to 1, 3, and 5 respectively; the receptive field sizes of the first dilated convolution layer are 3×3, 7×7, and 11×11 respectively; the receptive field sizes of the second dilated convolution layer are 5×5, 13×13, and 21×21 respectively;

[0105] Considering that the haze-related feature information obtained through these three feature extraction paths still comes from the local area, a self-correction module is designed to model the interdependence between the global and local features in the channel direction and spatial direction of the input feature map respectively; Figure 4 As shown in the figure, in the self-correction module, the upper branch models the dependency relationship between global and local features in the channel direction through global average pooling, 3×3 convolution operation, full connection operation and Sigmoid operation in the channel direction; the lower branch models the dependency relationship between global and local features in the spatial direction through global average pooling, 1×1 convolution operation, 3×3 convolution operation and Sigmoid operation in the height and width direction; finally, the output features of the upper branch and the lower branch are concatenated and the output features of the final self-correction module are obtained through 1×1 convolution operation;

[0106] The output features obtained from the three different path branches are concatenated, and then global average pooling, full connection and Sigmoid operations are performed to obtain the weight matrix;

[0107] Finally, the weight matrix and the output features of the three different path branches are multiplied to obtain the weighted output features of the three different path branches, which are then fused to further obtain regionally discernible haze-related feature information; the fusion process is as follows:

[0108]

[0109] Among them, c represents the total feature dimension; F krepresents the output feature of any branch, represents the i-th channel feature dimension of the k-th branch, represents the fusion feature, u, v represent the feature height and width components, h, w represent the height and width of the output feature; after the features obtained from the three different paths are spliced ​​using global average pooling in the channel direction, the final fusion feature is obtained:

[0110]

[0111] Among them, FC represents the network fully connected convolution operation; g c Represents a 1×1 convolution operation; is the multiplication operation; is the value w of the i-th channel with weight w i The matrix formed by expansion; It represents the final output features of the haze positioning module, that is, the haze-related features of regional discernibility.

[0112] S2-2, construct the disentanglement module:

[0113] The disentanglement module removes the dependencies between essential features and irrelevant features by utilizing the random features and sample weighting characteristics in the Gaussian kernel Fourier domain;

[0114] The set of output features of the last layer of the encoder in the defogging network is called Y. To make any pair of features y in Y ：，i ,y ：，j Not relevant, then y ：，i ,y ：，j The cross-covariance cov(y ：，i y ：，j )=0; To reduce the computational overhead, the HibertSchmidt norm of the Euclidean space is converted to the Frobenius norm to solve, and the random features in the Fourier domain of the Gaussian kernel function are used for approximate estimation. Therefore, the above y ：，i ,y ：，j The cross-covariance cov(y ：，i y ：，j ) can be restated as:

[0115]

[0116] in,

[0117]

[0118] in, for h n (·), f n(·) is the eigenvalue sampled from the random Fourier eigenfunction space; the number of samples n = 5;

[0119] Random Fourier features are introduced and a weighting matrix is ​​learned based on them. The data is globally decorrelated and standardized to remove the dependency between the fog-free background features and the haze. The two are made independent by minimizing the mutual covariance Cov of the paired features, thereby achieving haze disentanglement. represents the sample weight and For any pair of features y ：，i ,y ：，j , the weighted cross-covariance matrix Calculate using the following formula:

[0120]

[0121] in represents the sample weight of the i-th paired feature, represents the sample weight of the jth paired feature; further, the following formula is used to Optimize and get the optimized weight That is, the essential weight:

[0122]

[0123] in Represents the sample weight obtained by optimization The minimum value in and c represents the total feature dimension; Represents the 2-norm operation.

[0124] S2-3, according to the above steps S1, S2-1 and S2-2, construct the overall defogging network, such as Figure 5 As shown in the figure. In this network, the encoder and decoder are any existing efficient lightweight codec networks. The haze positioning module is used to output discriminative haze-related feature information, and the disentanglement module is used to obtain essential weights that are not related to haze, and to constrain the training of the network by constructing an adaptive loss to achieve the disentanglement of haze and non-haze features. The negative sample is the input haze image, and the positive sample is the input corresponding clear label image. These two and the predicted haze-free output are input together into the encoder constructed by VGG-19 to construct the contrast loss.

[0125] Step S3 is as follows:

[0126] Use Optimize Weights That is, the training samples are weighted by the essential weights, and the training of the dehazing network is guided by constructing the following adaptive loss:

[0127]

[0128] And with

[0129]

[0130] where x i represents the haze image, J i represents the clear image label, t represents the timestamp of each batch, Represents the predicted haze-free image output; initial weight value y ：，i ,y ：，j Represents any pair of features in the feature set Y;

[0131] In addition, a contrast loss function is designed to constrain the network to restore the haze-free image closer to the clear sample image and further away from the original haze image; its target loss function is expressed as follows:

[0132]

[0133] Among them, g k represents the kth hidden layer feature extracted from the fixed pre-trained weight model VGG-19; J represents the clear label image, and I represents the original haze image; represents a positive prediction value, represents a negative predicted value; Represents the predicted haze-free image output; further, the contrast loss is defined as follows:

[0134]

[0135] Where α represents the penalty parameter, λ k represents the weight coefficient;

[0136] The total loss function is summarized as follows:

[0137] Loss=Loss a +Loss b .

[0138] Step S4 is as follows:

[0139] The test data set comes from haze data in real scenes. During the test, haze images of any size are input into the dehazing network in batches.

[0140] Load the dehazing network weights trained on the LDHaze dataset, and input the haze images in the test dataset into the dehazing network one by one to obtain the predicted haze-free image output;

[0141] The quality of the predicted fog-free output image is evaluated using the no-reference image quality evaluation indicators PM2.5, FADE, and PIQE. If the performance is good on the above-mentioned no-reference image quality evaluation indicators, the network parameters are fixed to form the final defogging network based on haze positioning and disentanglement. If the above-mentioned multiple no-reference image evaluation indicators do not meet satisfactory requirements, the model training weight parameters are replaced and retested until the network model performs well on multiple authoritative no-reference image indicators.

[0142] Accordingly, the present invention also provides an image defogging system based on haze positioning and disentanglement, such as Figure 6 As shown, including:

[0143] A training data construction unit 1 is used to construct a haze image dataset LDHaze as training data;

[0144] The defogging network construction unit 2 is used to construct a defogging network, which includes an encoder, a haze positioning module, a disentanglement module, and a decoder; wherein the encoder is used to extract the features of the haze image, and the features of the haze image are input into the haze positioning module and the disentanglement module; the haze positioning module is used to output haze-related feature information of regional discernibility, and the disentanglement module is used to obtain essential weights that are irrelevant to haze, that is, the weights of haze-free background features, which are used to weight training data and guide the training of the defogging network by constructing an adaptive loss; the decoder is used to decode the output features of the haze positioning module to obtain a predicted defogging image;

[0145] The defogging network training unit 3 is used to design a loss function and train the defogging network using the haze image dataset LDHaze to form a trained defogging network based on haze positioning and disentanglement;

[0146] The test application unit 4 is used to test the trained dehazing network based on haze positioning and disentanglement using the no-reference image quality evaluation index. When the performance is good on the no-reference image quality evaluation index, the network parameters are fixed to form the final dehazing network based on haze positioning and disentanglement, whose input is any haze image and output is the image dehazing result.

[0147] Therefore, the present invention introduces a haze positioning module and adopts a weighted addition method for feature fusion. Different from many defogging methods that simply splice features directly, the present invention has the following advantages: 1) Learning weights helps the module to adaptively adjust the importance of feature information extracted by three parallel paths, and enhances the network's ability to distinguish and learn haze of different densities and depths; 2) Direct splicing operations will increase the number of feature channels exponentially, while using a positional addition operation will maintain the original number of channels and will not increase the amount of calculation.

[0148] The present invention introduces a haze disentanglement module, which directly decorrelates the dependency between haze-free background features and fog in feature space by learning weights of training samples, thereby helping the dehazing model to focus on learning the essential connection between discriminative features and clear labels.

[0149] The perceptual contrast loss constructed by the present invention can shorten the distance between the predicted haze-free output and the clear label in the implicit distance space, and increase the distance between the predicted haze-free output and the haze image, so that the overall network training is updated in the natural direction of dehazing.

[0150] The present invention can achieve the removal of various types of haze. Compared with the existing image defogging methods, it has obvious advantages in defogging quality evaluation, defogging stability, defogging accuracy, and defogging speed. In the invention, the input and output between the various parts of the network are interconnected, and different features are extracted and integrated, which improves efficiency and makes the restored haze-free image more natural.

[0151] The above is a detailed introduction to an image defogging method and system based on haze positioning and disentanglement provided by an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An image defogging method based on haze positioning and disentanglement, characterized in that: The method comprises: Construct the haze image dataset LDHaze as training data; Constructing a defogging network, the defogging network includes an encoder, a haze positioning module, a disentanglement module, and a decoder; wherein the encoder is used to extract features of a haze image, and the features of the haze image are input into the haze positioning module and the disentanglement module; the haze positioning module is used to output haze-related feature information of regional discernibility, and the disentanglement module is used to obtain essential weights unrelated to haze, that is, weights of haze-free background features, which are used to weight the training data and guide the training of the defogging network by constructing an adaptive loss; the decoder is used to decode the output features of the haze positioning module to obtain a predicted defogging image; Designing a loss function, and using the haze image data set LDHaze to train the dehazing network, to form a trained dehazing network based on haze positioning and disentanglement; The trained dehazing network based on haze positioning and disentanglement is tested using a no-reference image quality evaluation indicator. When the performance is good on the no-reference image quality evaluation indicator, the network parameters are fixed to form a final dehazing network based on haze positioning and disentanglement, whose input is an arbitrary haze image and output is an image dehazing result.

2. The image defogging method based on haze positioning and disentanglement as claimed in claim 1, characterized in that: The construction of the haze image dataset LDHaze as training data is specifically as follows: The clear labels used to synthesize haze images come from the clear images and corresponding depth image information in the RESIDE dataset. The construction process of the haze image dataset LDHaze is described by the following formula: Where I represents the haze image to be synthesized, F filter (·) represents the blur kernel function; x, y represent the pixel coordinates, x′, y′ are the movement increments of the pixel coordinates; represents the generated initial haze image; γ is the constraint factor used to adjust the brightness of the generated haze data image; A is the atmospheric light value; t represents the scene transmittance map, R represents the clear image; ε is the noise term.

3. The image defogging method based on haze positioning and disentanglement as claimed in claim 1, characterized in that: The haze positioning module is specifically: The overall process of the haze positioning module is as follows: First, the output features of the last layer of the encoder in the defogging network are explicitly mapped by a random Fourier explicit mapping function; secondly, the explicit mapping results are input into three different path branches to obtain path output features, and the three path branches are composed of dilated convolutions with different dilated rates. The output features obtained from the three different path branches are spliced, and then global average pooling, full connection and Sigmoid operations are performed to obtain a weight matrix; finally, the weight matrix and the output features of the three path branches are multiplied to obtain the weighted output features of the three different path branches, and these output features are spliced ​​again and the 1×1 convolution operation is performed to obtain the output features of the final haze positioning module, that is, the haze-related features of regional discernibility; The specific process of the haze positioning module is as follows: First, the random mapping features in the Gaussian kernel Fourier domain are used to explicitly represent the high-order nonlinear feature information in the output features of the last layer of the encoder in a low-dimensional manner: The output feature of the last layer of the encoder is recorded as A low-dimensional explicit mapping function z(y) is used to estimate a given Gaussian kernel G on each channel. y , the formula is as follows: Among them G y represents the Gaussian kernel, y a ,y b They represent the components of feature y; α represents the size of the Gaussian kernel; z represents the number of features used to estimate the given Gaussian kernel G(y a ,y b ) is a low-dimensional explicit mapping function; According to Bochner's theorem, if the Gaussian kernel G(y a ,y b ) in the original implicit space dimension Upper positive definite if and only if G y is the Fourier transform of a non-negative measure p(s); thus we get the following formula: Where p(s) represents a non-negative measure, i represents the imaginary part, and D represents the feature dimension; Bias parameter b d ∈[b1…b D ]From the uniform distribution U [0,2π] obtained in; From the non-negative measure The weight parameters obtained by sampling; Based on the above formula, the random Fourier features of the output features of the last layer of the encoder are obtained The expression form: Secondly, three parallel paths are designed to capture the haze-related feature information of different haze densities and depths in different areas of the image; these three paths are built based on dilated convolutions, which can extract features from receptive fields of different sizes while maintaining image resolution and reducing the number of parameters; in the three paths, two layers of 3×3 dilated convolutions are used and the dilation rates are set to 1, 3, and 5 respectively; the receptive field sizes of the first dilated convolution layer are 3×3, 7×7, and 11×11 respectively; the receptive field sizes of the second dilated convolution layer are 5×5, 13×13, and 21×21 respectively; Considering that the haze-related feature information obtained through these three feature extraction paths still comes from the local area, a self-correction module is designed to model the mutual dependence between the global and local features in the channel direction and the spatial direction of the input feature map respectively; in this self-correction module, the upper branch models the dependency between the global and local features in the channel direction through global average pooling, 3×3 convolution operation, full connection operation and Sigmoid operation in the channel direction; The lower branch models the dependency between global and local features in the spatial direction through global average pooling, 1×1 convolution operation, 3×3 convolution operation, and Sigmoid operation in the height and width directions. Finally, the output features of the upper branch and the lower branch are concatenated and the output features of the final self-correction module are obtained through a 1×1 convolution operation. The output features obtained from the three different path branches are concatenated, and then global average pooling, full connection and Sigmoid operations are performed to obtain the weight matrix; Finally, the weight matrix and the output features of the three different path branches are multiplied to obtain the weighted output features of the three different path branches, which are then fused to further obtain regionally discernible haze-related feature information; The fusion process is as follows: Among them, c represents the total feature dimension; F k represents the output feature of any branch, represents the i-th channel feature dimension of the k-th branch, represents the fusion feature, u, v represent the feature height and width components, h, w represent the height and width of the output feature; after the features obtained from the three different paths are spliced ​​using the global average pooling in the channel direction, the final fusion feature is obtained: Among them, FC represents the network fully connected convolution operation; g c Represents a 1×1 convolution operation; is the multiplication operation; is the value w of the i-th channel with weight w i The matrix formed by expansion; It represents the final output features of the haze positioning module, that is, the haze-related features of regional discernibility.

4. The image defogging method based on haze positioning and disentanglement as claimed in claim 1, characterized in that: The de-entanglement module is specifically: The disentanglement module removes the dependencies between essential features and irrelevant features by utilizing the random features and sample weighting characteristics in the Gaussian kernel Fourier domain; The set of output features of the last layer of the encoder in the defogging network is denoted as Y. To make any pair of features y in Y :,i ,y :,j Not relevant, then y :,i ,y :,j The cross-covariance cov(y :,i y :,j )=0; To reduce the computational overhead, the HibertSchmidt norm of the Euclidean space is converted to the Frobenius norm to solve, and the random features in the Fourier domain of the Gaussian kernel function are used for approximate estimation. Therefore, the above y :,i ,y :,j The cross-covariance cov(y :,i y :,j ) can be restated as: in, in, for is the eigenvalue sampled from the random Fourier eigenfunction space; n is the number of samples; Random Fourier features are introduced and a weighting matrix is ​​learned based on them. The data is globally decorrelated and standardized to remove the dependency between the fog-free background features and the haze. The two are made independent by minimizing the mutual covariance Cov of the paired features, thereby achieving haze disentanglement. represents the sample weight and For any pair of features y :, i,y :, j, weighted cross-covariance matrix Calculate using the following formula: in represents the sample weight of the i-th paired feature, represents the sample weight of the jth paired feature; further, the following formula is used to Optimize and get the optimized weight That is, the essential weight: in Represents the sample weight obtained by optimization The minimum value in and c represents the total feature dimension; Represents the 2-norm operation.

5. The image defogging method based on haze positioning and disentanglement as claimed in claim 4, characterized in that: The design loss function is specifically: Use the optimization weights That is, the essential weights are used to weight the training samples, and the training of the dehazing network is guided by constructing the following adaptive loss: And with where x i represents the haze image, J i represents the clear image label, t represents the timestamp of each batch, Represents the predicted haze-free image output; initial weight value y :,i ,y :,j represents any paired features in the feature set Y; In addition, a contrast loss function is designed to constrain the network to restore the haze-free image closer to the clear sample image and further away from the original haze image; its target loss function is expressed as follows: Among them, g k represents the kth hidden layer feature extracted from the fixed pre-trained weight model VGG-19; J represents the clear label image, and I represents the original haze image; represents a positive prediction value, represents a negative predicted value; Represents the predicted haze-free image output; further, the contrast loss is defined as follows: Where α represents the penalty parameter and λk represents the weight coefficient; The total loss function is summarized as follows: Loss=Loss a +Loss b 。 6. The image defogging method based on haze positioning and disentanglement as claimed in claim 1, characterized in that: The trained defogging network based on haze positioning and detangling is tested using the no-reference image quality evaluation index. When the performance is good on the no-reference image quality evaluation index, the network parameters are fixed to form the final defogging network based on haze positioning and detangling, specifically: The test data set comes from haze data in real scenes. During the test, haze images of any size are input into the dehazing network in batches. Load the defogging network weights trained on the LDHaze dataset, and input the haze images in the test dataset into the defogging network one by one to obtain the predicted haze-free image output; The quality of the predicted haze-free output image is evaluated using a no-reference image quality evaluation index. If the performance on the no-reference image quality evaluation index is good, the network parameters are fixed to form a final dehazing network based on haze positioning and disentanglement.

7. An image defogging system based on haze positioning and disentanglement, characterized in that: The system comprises: A training data construction unit is used to construct a haze image dataset LDHaze as training data; A defogging network construction unit is used to construct a defogging network, which includes an encoder, a haze positioning module, a disentanglement module, and a decoder; wherein the encoder is used to extract features of a haze image, and the features of the haze image are input into the haze positioning module and the disentanglement module; the haze positioning module is used to output haze-related feature information of regional discernibility, and the disentanglement module is used to obtain essential weights that are irrelevant to haze, that is, weights of haze-free background features, which are used to weight the training data and guide the training of the defogging network by constructing an adaptive loss; the decoder is used to decode the output features of the haze positioning module to obtain a predicted defogging image; A defogging network training unit, used for designing a loss function and training the defogging network using the haze image data set LDHaze to form a trained defogging network based on haze positioning and disentanglement; The test application unit is used to test the trained defogging network based on haze positioning and disentanglement using a no-reference image quality evaluation index. When the performance is good on the no-reference image quality evaluation index, the network parameters are fixed to form a final defogging network based on haze positioning and disentanglement, whose input is an arbitrary haze image and whose output is an image defogging result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images