Double-branch defogging method based on Laplacian pyramid

Abstract

The invention relates to the technical field of computer image processing, and particularly discloses a double-branch defogging method based on a Laplacian pyramid, and the method comprises the following steps: S1, building a training data set A containing an original clear image and a corresponding foggy image; s2, performing cutting and data enhancement processing on the input image; s3, a defogging network LPSDNet based on the Laplacian pyramid is constructed, the defogging network LPSDNet comprises a multi-feature fusion focusing branch and a Laplacian pyramid supplementary branch, and after feature extraction of the two branches is completed, features learned by the two branches are fused through a learnable fusion module; s4, training the defogging network LPSDNet by adopting the training data set A until a preset loss function is converged; obtaining a trained defogging network LPSDNet in combination with an objective function; and S5, inputting an image to be defogged into the trained defogging network LPSDNet for testing and verification to obtain a defogging result. The global structure information of the image can be fully captured, and the distortion phenomenon is prevented.

Description

Technical Field

[0001] The invention relates to the technical field of computer image processing, and specifically provides a double-branch defogging method based on a Laplacian pyramid. Background Art

[0002] Due to the presence of a large number of suspended particles in the atmosphere under severe weather conditions such as haze, images are often greatly affected, resulting in problems such as degraded visual quality, blurring, color shift, reduced contrast, and buried details. This phenomenon can seriously affect the performance of computer vision systems. Therefore, image dehazing technology has become an important research direction in the field of image enhancement.

[0003] Image dehazing is an important low-level vision task, which aims to restore haze-free images from haze images. In haze environment, due to the presence of suspended particles in the air, the captured images will have color distortion and detail loss. Such low-quality images will significantly reduce algorithm performance when applied to high-level vision tasks such as target detection, semantic segmentation or autonomous driving. Image dehazing uses image processing technology to restore foggy images to haze-free images, thereby effectively improving the execution accuracy of subsequent vision tasks. In recent years, accurate and clear dehazed images have also attracted more and more attention.

[0004] With the widespread application of deep learning in image processing tasks, many scholars have proposed many defogging methods based on deep learning. The defogging methods based on deep learning use their powerful nonlinear fitting capabilities to automatically learn prior knowledge from a large amount of image data and directly map foggy images to fog-free images. Existing defogging methods based on deep learning mainly use convolutional neural networks (CNNs). However, since convolutional neural networks use continuous downsampling, they are prone to lose some structural information, resulting in serious distortion of the reconstruction results in some complex areas, thus affecting the defogging effect.

[0005] In order to overcome the limitations of convolutional neural networks in extracting structural information, the existing technologies include: 1. SDBAD-NET (Spatial Dual-Branch Attention Dehazing Network) in the existing technology is a technology widely used in the fields of deep learning and computer vision. It aims to improve the performance of the model by dynamically adjusting and optimizing feature representation, and compensates for the structural loss of convolutional neural networks by dynamically fusing shallow structural features with deep contextual semantic information.

[0006] 2. The Dynamic Feature Enhancement Module (DFE) in the existing technology is a dehazing network based on the Meta-Former paradigm, which aims to solve the image dehazing problem in complex scenes. It introduces deformable convolution to dynamically expand the receptive field and integrate more structured information.

[0007] The above methods take into account the importance of structural information in image restoration and retain the structural details of the original image to a certain extent. However, limited by the receptive field of the convolution operation, these methods lack the ability of global modeling and find it difficult to fully capture the global structural information of the image. Therefore, the haze-free images generated by these methods may be distorted in local areas. Summary of the invention

[0008] In order to solve the above problems, the present invention provides a dual-branch defogging method based on Laplacian pyramid, which can effectively restore the detail information of the image and generate a defogging image close to the real image.

[0009] The present invention provides a dual-branch defogging method based on Laplace pyramid, comprising the following steps: S1. Establish a training data set A including original clear images and corresponding foggy images; S2, performing data enhancement processing on the training data set A, including cropping and data enhancement processing on the input image; S3. Construct a dehazing network LPSDNet based on Laplacian pyramid; The dehazing network LPSDNet includes a multi-feature fusion focusing branch and a Laplacian pyramid supplementing branch. After the multi-feature fusion focusing branch and the Laplacian pyramid supplementing branch complete feature extraction, the features learned by the multi-feature fusion focusing branch and the Laplacian pyramid supplementing branch are further fused through the learnable fusion module. The multi-feature fusion focusing branch includes an encoder, a decoder, and a jump connection; the encoder includes multiple 3×3 convolutional layers, a supplementary fusion module, and a hybrid focus attention module, through which the encoder extracts the main feature information, reduces the size of the feature map, and increases the number of feature channels; the decoder restores the feature map output by the encoder to the same size as the original image through an upsampling layer, an SK fusion module, a hybrid focus attention module, and multiple 3×3 convolutional layers; a jump connection of a 1×1 convolutional layer is set between the encoder and the decoder to dynamically fuse shallow and deep features; S4. Use the training data set A to train the defogging network LPSDNet until the preset loss function converges; combine the L1 loss function and the structural loss function as the objective function of the network training to obtain the trained defogging network LPSDNet; S5. Input the image to be dehazed into the trained dehazing network LPSDNet for testing and verification to obtain the dehazing result.

[0010] Furthermore, the supplementary fusion module in step S3 includes two inputs and one output, and fuses the structural information obtained by decomposing the supplementary branch of the Laplacian pyramid with the features extracted by the current branch; the operation steps of the supplementary fusion module include: S31a, feature F extracted by multi-feature fusion focusing branch d The image feature F obtained by decomposing the Laplacian pyramid supplementary branches p , first input feature F p Perform a 1×1 convolution operation to expand the number of channels of the feature map; then perform a 3×3 convolution operation to obtain the feature map ; S32a, in the channel dimension and F d Perform feature splicing to obtain splicing feature F cat ; At the same time, in the pixel dimension, and F d Perform addition operation to obtain feature F add ; S33a, for the concatenated feature F catt Perform convolution operation to realize features and F d Fusion of; S34a, using a spatial attention module to generate a feature attention weight map from the fused features; wherein the spatial attention module is composed of a 1×1 convolution, a GELU activation function, a 1×1 convolution, and a Sigmoid function in sequence; S35a, the obtained attention weight map and F add Multiply to get feature F frame , highlighting feature F add Spatial structure information in S36a, feature F frame With feature F d Add together to get the output feature F of the supplementary fusion module output .

[0011] Furthermore, the learnable fusion module in step S3 includes two inputs and one output, and fuses the structural information obtained by decomposing the Laplacian pyramid supplementary branch with the features extracted by the current branch; the operation steps of the learnable fusion module include: S31b, feature F extracted by focusing on the multi-feature fusion branch d The image feature F obtained by decomposing the Laplacian pyramid supplementary branches p , first input feature F pPerform a 1×1 convolution operation to expand the number of channels of the feature map; then perform a 3×3 convolution operation to obtain the feature map; S32b, and F in the channel dimension d Perform feature splicing to obtain splicing feature F cat ; At the same time, in the pixel dimension, and F d Perform addition operation to obtain feature F add ; S33b, after concatenation, feature F cat Perform convolution operation to achieve feature sum F d Fusion of; S34b, using a spatial attention module to generate a feature attention weight map from the fused features; wherein the spatial attention module is composed of a 1×1 convolution, a GELU activation function, a 1×1 convolution, and a Sigmoid function in sequence; S35b, the obtained attention weight map and F add Multiply to get feature F frame , highlighting feature F add Spatial structure information in S36b, introduce a learnable parameter a, and combine the parameter a with the feature F frame After multiplication, add it to the original feature F d Add.

[0012] Furthermore, in step S3, the network introduces the multi-scale structural information extracted by the Laplacian pyramid supplementary branch into the multi-feature fusion focusing branch. The Laplacian pyramid supplementary branch uses the Laplacian pyramid decomposition module to perform multi-scale decomposition on the input foggy image to obtain features of four different scales. The output of the Laplacian pyramid supplementary branch is obtained by the following steps: S31c, among the four features of different scales, the first three multi-scale features are passed to the supplementary fusion module corresponding to the multi-feature fusion focusing branch, and the last multi-scale feature is doubled in size after the pixel padding operation and then passed to the supplementary fusion module corresponding to the multi-feature fusion focusing branch; S32c, using a supplementary fusion module and a frequency enhancement module to enhance the high-frequency features and the low-frequency features; S33c, reconstruct the enhanced high-frequency features and low-frequency features using the Laplacian pyramid reconstruction module to obtain the output of the Laplacian pyramid supplementary branch.

[0013] Furthermore, the high-frequency and low-frequency feature information is extracted through the Laplacian pyramid decomposition module and the Laplacian pyramid reconstruction module. Specifically, for the input image I, the Laplacian pyramid decomposition module decomposes the input image I into a low-frequency component and several high-frequency components. The calculation process is as follows: The calculation formula of the low-frequency component L is: L=G N (I), where Represents the result of Gaussian blurring image I N times; The calculation formula of the high-frequency component Hi of the i-th layer is as follows:

[0014] Among them, 1≤i <N; represents the Gaussian pyramid image at the i-th level, Indicates that a 2D Gaussian kernel of size 5×5 is used for blurring. Indicates upsampling the image to the original size; Gaussian Pyramid The calculation formula is:

[0015] in, Indicates that the image is downsampled to half the size of the original image; The Laplacian pyramid reconstruction is the inverse process of the above decomposition.

[0016] Furthermore, the frequency enhancement module in step S32c includes two residual blocks, which are used to extract global structural information and focus on key features; specifically: the first residual block uses a normalization layer to normalize the input feature map x; three convolution layers with convolution kernels of 3×3, 5×5, and 7×7 are used to extract multi-scale features of the image in parallel, and the obtained multi-scale features are spliced ​​to obtain a feature map x2; the input feature x2 is added to the input feature x to obtain the output feature y of the first residual block, and convolution operations with different convolution kernel sizes are used to extract high-frequency and low-frequency feature information in the image; the second residual block contains a spatial attention layer, which is used to enhance the captured features and make full use of frequency features.

[0017] Furthermore, the hybrid focus attention module in step S3 includes two residual blocks, and the operation steps of the hybrid focus attention module include: The first residual block: S31d, for the input feature map x, the first residual block normalizes it using a normalization layer; S32d, using three convolution layers with convolution kernels of 3×3, 5×5, and 7×7 to extract multi-scale features of the image in parallel, and concatenating the obtained multi-scale features to obtain a feature map x2; S33d, adding the input feature x2 to the input feature x to obtain the output feature y of the first residual block; The second residual block: S34d, the second residual block uses multiple attention layers to further extract information from the feature map y; S35d, using normalization to normalize the feature map y to obtain feature y1; S36d, input feature y1 into the tangent attention layer, square attention layer, and re-attention layer respectively to obtain different focus features F1, F2, and F3; S37d. Concatenate the focused features F1, F2, and F3, and use the channel attention layer to further extract the key features in the image to obtain feature y3. The channel attention layer is composed of global average pooling, 1×1 convolution, GELU activation function, 1×1 convolution, and Sigmoid function in sequence. Among them, the first 1×1 convolution is to change the number of channels of the feature map, and the second 1×1 convolution is to extract features.

[0018] S38d and y3 are added to the feature y after passing through the multi-layer perceptron to obtain the final output O of the hybrid focus attention module.

[0019] Furthermore, in step S36d, the tangent attention layer consists of a residual connection and two consecutive 1×1 convolutional layers, the first convolutional layer is followed by a GELU activation function, and the second convolutional layer is followed by a Tanh activation function. The formula of the tangent attention layer is expressed as follows: F1= y1⊙Tanh(PWConv(GELU(PWConv(X)))) Among them, Tanh represents the hyperbolic tangent function, ⊙ represents pixel-by-pixel multiplication; the hyperbolic tangent function can map the attention weight to a range between -1 and 1, where a weight value less than zero indicates that the data at the corresponding input position is unimportant or harmful to the current task.

[0020] Furthermore, in step S36d, the formula of the square attention layer is expressed as follows: F2= y1⊙Sigmoid(PWConv(GELU(PWConv(X)))) 2 Among them, Sigmoid represents the Sigmoid activation function. In the square attention layer, the Sigmoid activation function can map the feature value to an attention weight from 0 to 1.

[0021] Furthermore, in step S36d, the attention layer uses continuous spatial attention to gradually focus the network on important features, which specifically includes the following steps: S36d-1: The attention layer applies spatial attention to get preliminary attention weights; S36d-2: Multiply the initial attention weight with the input feature to obtain the intermediate feature; S36d-3: Perform spatial attention operations on intermediate features to obtain refined attention weights; S36d-4: Multiply the obtained attention weight by the input to obtain the aggregate feature F3; the formula of the aggregate feature F3 is as follows: X^= y1⊙Sigmoid(PWConv(GELU(PWConv(X)))) F3= y1⊙Sigmoid(PWConv(GELU(PWConv(X^)))) Among them, X^ represents the intermediate features generated by the re-attention layer.

[0022] Compared with the prior art, the present invention can achieve the following beneficial effects: the image defogging method of the present invention can respectively extract the main features and high- and low-frequency illumination features of the image through a multi-feature fusion focusing branch and a Laplacian pyramid supplementary branch, and then realize the adaptive fusion of the main features and the high- and low-frequency features through a learnable feature fusion module, and finally obtain the final defogging image by using reconstruction technology, which has the ability of global modeling and can fully capture the global structural information of the image, and the generated fog-free image will not be distorted in the local area. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a schematic diagram of the structure of a Laplace pyramid double-branch network provided according to an embodiment of the present invention; Figure 2 is a schematic diagram of the structure of a supplementary fusion module provided according to an embodiment of the present invention; Figure 3 is a schematic diagram of the structure of a hybrid focus attention module provided according to an embodiment of the present invention; Figure 4 is a structural schematic diagram of a frequency enhancement module provided according to an embodiment of the present invention; Figure 5 1 is an example of a dehazing result provided according to an embodiment of the present invention; wherein (a) is a foggy image, (b), (c), and (d) represent images dehazed by the DCP, FFANet, and GridDehaze methods, respectively, (e) is an image dehazed by the present invention, and (f) is an original non-hazy image; Figure 6 is an overall flow chart of the dual-branch defogging method provided by an embodiment of the present invention; Figure 7 yes Figure 5 (a) is an enlarged view of the upper image in FIG. Figure 8 yes Figure 5 (b) is an enlarged view of the upper image in FIG. Fig. 9 yes Figure 5 (c) is an enlarged view of the upper image; Fig.10yes Figure 5 (d) is an enlarged view of the upper image; Fig.11 yes Figure 5 (e) is an enlarged view of the upper image; Fig.12 yes Figure 5 (f) is an enlarged view of the upper image; Fig.13 yes Figure 5 (a) is an enlarged view of the middle image in FIG. Fig.14 yes Figure 5 (b) is an enlarged view of the middle image in FIG. Fig.15 yes Figure 5 (c) is an enlarged view of the middle image in FIG. Fig.16 yes Figure 5 (d) is an enlarged view of the middle image in FIG. Fig.17 yes Figure 5 (e) is an enlarged view of the middle image in FIG. Fig.18 yes Figure 5 (f) is an enlarged view of the middle image in FIG. Fig.19 yes Figure 5 (a) is an enlarged view of the lower figure in the figure; Fig. 20 yes Figure 5 (b) is an enlarged view of the lower figure; Fig.21 yes Figure 5 (c) is an enlarged view of the lower figure; Fig. 22 yes Figure 5 (d) is an enlarged view of the lower image; Fig.23 yes Figure 5 (e) is an enlarged view of the lower figure; Fig.24 yes Figure 5 (f) is an enlarged view of the lower figure; Fig.25 yes Figure 1 A zoomed-in image of the foggy image; Fig.26 yes Figure 1 A zoomed-in image of the dehazed image; Fig. 27 yes Figure 1 Enlarged view of the H×W×3 figure in the middle; Fig.28 yes Figure 1 Enlarged view of the H / 2×W / 2×3 diagram; Fig.29yes Figure 1 Enlarged view of the H / 4×W / 4×3 diagram; Fig.30 yes Figure 1 Enlarged view of the H / 8×W / 8×3 diagram; Fig.31 yes Figure 2 A magnified view of Fp in the middle; Fig.32 yes Figure 2 Enlarged view of Fd in the middle; Fig.33 yes Figure 2 A magnified image of the output in ; Fig.34 yes Figure 1 A zoomed-in view of the supplementary branch of the Laplacian pyramid. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the following Figure 1-34 It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0025] A dual-branch dehazing method based on Laplacian pyramid, such as Figure 6 As shown, the following steps are included: S1. Establish a training dataset A containing original clear images and corresponding foggy images.

[0026] S2. Perform data enhancement processing on the training data set A; including cropping and data enhancement processing on the input image, specifically including: for a size of The input image is randomly cropped into 256×256 patches and data augmented by randomly rotating it by 90°, 180°, or 270°.

[0027] S3. Construct a defogging network LPSDNet based on the Laplace pyramid. The specific structure is as follows Figure 1 shown.

[0028] The dehazing network LPSDNet includes a multi-feature fusion focusing branch and a Laplacian pyramid supplementing branch. After the multi-feature fusion focusing branch and the Laplacian pyramid supplementing branch complete feature extraction, the features learned by the multi-feature fusion focusing branch and the Laplacian pyramid supplementing branch are further fused through a learnable fusion module.

[0029] The multi-feature fusion focusing branch includes an encoder, a decoder, and a skip connection. The encoder includes multiple 3×3 convolutional layers, a supplementary fusion module (SFM), and a hybrid focused attention module (MFAM). The encoder extracts the main feature information, reduces the size of the feature map, and increases the number of feature channels. The decoder restores the feature map output by the encoder to the same size as the original image through an upsampling layer, an SK fusion module, a hybrid focused attention module, and multiple 3×3 convolutional layers. In order to retain spatial detail information, a skip connection of a 1×1 convolutional layer is set between the encoder and the decoder to dynamically fuse shallow and deep features.

[0030] The supplementary fusion module in step S3 is as follows Figure 2 As shown, it contains two inputs and one output, and fuses the structural information obtained by decomposing the Laplacian pyramid supplementary branch with the features extracted by the current branch; the operation steps of the supplementary fusion module include: S31a, feature F extracted by multi-feature fusion focusing branch d The image feature F obtained by decomposing the Laplacian pyramid supplementary branches p , first input feature F p Perform a 1×1 convolution operation to expand the number of channels of the feature map; then perform a 3×3 convolution operation to obtain the feature map; S32a, and F in the channel dimension d Perform feature splicing to obtain splicing feature F cat ; At the same time, in the pixel dimension, and F d Perform addition operation to obtain feature F add ; S33a, for the concatenated feature F cat Perform convolution operation to achieve feature sum F d effective integration of S34a, using a spatial attention module to generate a feature attention weight map from the fused features; wherein the spatial attention module is composed of a 1×1 convolution, a GELU activation function, a 1×1 convolution, and a Sigmoid function in sequence; S35a, the obtained attention weight map and F add Multiply to get feature F frame , highlighting feature F add Spatial structure information in S36a, feature F frame With feature F d Add together to get the output feature F of the supplementary fusion module output .

[0031] To enhance the structural features, the network introduces the multi-scale structural information extracted by the Laplacian pyramid supplementary branch into the multi-feature fusion focusing branch. The Laplacian pyramid supplementary branch uses the Laplacian pyramid decomposition module to perform multi-scale decomposition on the input foggy image to obtain features of four different scales. The output of the Laplacian pyramid supplementary branch is obtained by the following steps: S31c, among the four features of different scales, the first three multi-scale features are directly passed to the supplementary fusion module corresponding to the multi-feature fusion focusing branch, and the last multi-scale feature is doubled in size after the pixel padding operation and then passed to the supplementary fusion module corresponding to the multi-feature fusion focusing branch; S32c, using a supplementary fusion module and a frequency enhancement module (FEM) to enhance the high-frequency features and the low-frequency features; S33c, reconstruct the enhanced high-frequency features and low-frequency features using the Laplace pyramid reconstruction module to obtain the output of the Laplace pyramid supplementary branch.

[0032] After the multi-feature fusion focus branch and the Laplacian pyramid supplementary branch complete feature extraction, the features learned by the two branches are further fused through the learnable fusion module in step S3. The learnable fusion module includes two inputs and one output, and fuses the structural information decomposed by the Laplacian pyramid supplementary branch with the features extracted by the current branch. The operation steps of the learnable fusion module include: S31b, feature F extracted by focusing on the multi-feature fusion branch d The image feature F obtained by decomposing the Laplacian pyramid supplementary branch p , first input feature F p Perform a 1×1 convolution operation to expand the number of channels of the feature map; then perform a 3×3 convolution operation to obtain the feature map ; S32b, in the channel dimension and F d Perform feature splicing to obtain splicing feature F cat ; At the same time, in the pixel dimension Add it to Fd to get the feature F add ; S33b, after concatenation, feature F cat Perform convolution operation to realize features and F d Fusion of; S34b, using a spatial attention module to generate a feature attention weight map from the fused features; wherein the spatial attention module is composed of a 1×1 convolution, a GELU activation function, a 1×1 convolution, and a Sigmoid function in sequence; S35b, the obtained attention weight map and Fadd Multiply to get feature F frame , highlighting feature F add Spatial structure information in S36b, introduce a learnable parameter a, and combine the parameter a with the feature F frame After multiplication, add it to the original feature F d The introduction of the learnable parameter a allows the model to dynamically adjust the relative importance of the two branch outputs, achieving a more flexible and effective fusion of the main features and high- and low-frequency features, and producing clearer and more accurate dehazed images.

[0033] The high-frequency and low-frequency feature information is extracted through the Laplacian pyramid decomposition module and the Laplacian pyramid reconstruction module. Specifically, for the input image I, the Laplacian pyramid decomposition module decomposes the input image I into a low-frequency component and several high-frequency components. The calculation process is as follows: The calculation formula of the low-frequency component L is: L=G N (I), where Represents the result of Gaussian blurring image I N times; The calculation formula of the high-frequency component Hi of the i-th layer is as follows:

[0034] Among them, 1≤i <N; represents the Gaussian pyramid image at the i-th level, Indicates that a 2D Gaussian kernel of size 5×5 is used for blurring. Indicates upsampling the image to the original size.

[0035] Gaussian Pyramid The calculation formula is:

[0036] in, Indicates that the image is downsampled to half the size of the original image; The Laplacian pyramid reconstruction is the inverse process of the above decomposition.

[0037] The hybrid focus attention module in step S3 is as follows Figure 3 As shown in Figure 1, it includes two residual blocks. The operation steps of the hybrid focus attention module include: The first residual block: S31d, for the input feature map x, the first residual block normalizes it using a normalization layer; S32d, using three convolution layers with convolution kernels of 3×3, 5×5, and 7×7 to extract multi-scale features of the image in parallel, and concatenating the obtained multi-scale features to obtain a feature map x2; S33d, adding the input feature x2 to the input feature x to obtain the output feature y of the first residual block; S34d, the second residual block uses multiple attention layers to further extract information from the feature map y; The second residual block: S35d, using normalization to normalize the feature map y to obtain feature y1; S36d, input feature y1 into the tangent attention layer, square attention layer, and re-attention layer respectively to obtain different focus features F1, F2, and F3; S37d. Concatenate the focused features F1, F2, and F3, and use the channel attention layer to further extract the key features in the image to obtain feature y3. The channel attention layer is composed of global average pooling, 1×1 convolution, GELU activation function, 1×1 convolution, and Sigmoid function in sequence. Among them, the first 1×1 convolution is to change the number of channels of the feature map, and the second 1×1 convolution is to extract features.

[0038] S38d and y3 are added to the feature y after passing through the multi-layer perceptron to obtain the final output O of the hybrid focus attention module.

[0039] In step S36d, the tangent attention layer consists of a residual connection and two consecutive 1×1 convolutional layers. The first convolutional layer is followed by a GELU activation function, and the second convolutional layer is followed by a Tanh activation function. The formula of the tangent attention layer is expressed as follows: F1= y1⊙Tanh(PWConv(GELU(PWConv(X)))) Among them, Tanh represents the hyperbolic tangent function, and ⊙ represents pixel-by-pixel multiplication. The hyperbolic tangent function can map the attention weight to a range between -1 and 1, where a weight value less than zero indicates that the data at the corresponding input position is unimportant or harmful to the current task. Therefore, the attention weight map generated by the Tanh activation function can use negative weight values ​​to effectively suppress irrelevant or harmful signals, and has a stronger feature distinction ability.

[0040] In step S36d, the formula of the square attention layer is expressed as follows: F2= y1⊙Sigmoid(PWConv(GELU(PWConv(X)))) 2 Among them, Sigmoid represents the Sigmoid activation function. In the square attention layer, the Sigmoid activation function can map the feature value to the attention weight from 0 to 1. The square attention layer will generate the attention weight map and perform a point-by-point square operation. Although the point-by-point square operation will make all the values ​​in the attention weight map smaller, the values ​​close to 0 will become smaller than the values ​​close to 1. This is equivalent to increasing the larger values ​​in the attention weight map and weakening the smaller values ​​in the attention weight map, thereby directing the model's attention more focused on key areas.

[0041] In step S36d, the attention layer uses continuous spatial attention to gradually focus the network on important features, which specifically includes the following steps: S36d-1: The attention layer applies spatial attention to get preliminary attention weights; S36d-2: Multiply the initial attention weight with the input feature to obtain the intermediate feature; S36d-3: Perform spatial attention operations on intermediate features to obtain refined attention weights; S36d-4: Multiply the obtained attention weight by the input to obtain the aggregate feature F3; the formula of the aggregate feature F3 is as follows: X^= y1⊙Sigmoid(PWConv(GELU(PWConv(X)))) F3= y1⊙Sigmoid(PWConv(GELU(PWConv(X^)))) Among them, X^ represents the intermediate feature generated by the re-attention layer. After X^ passes through the attention mechanism again to generate the attention weight map, it is multiplied point by point with the input feature y1. This allows the network to suppress useless information in the feature map again and focus on key information. The use of continuous attention can effectively improve the degree of attention to key areas of the image, enhance the expression of useful information, and suppress the interference of useless information.

[0042] The frequency enhancement module in step S32C is as follows Figure 4 As shown in the figure, specifically: the frequency enhancement module includes two residual blocks, which are used to extract global structural information and focus on key features; specifically: the first residual block uses a normalization layer to normalize the input feature map x; three convolution layers with convolution kernels of 3×3, 5×5, and 7×7 are used in parallel to extract multi-scale features of the image, and the obtained multi-scale features are concatenated to obtain a feature map x2; the input feature x2 is added to the input feature x to obtain the output feature y of the first residual block, and convolution operations with different convolution kernel sizes are used to extract high-frequency and low-frequency feature information in the image; the second residual block contains a spatial attention layer, which is used to strengthen the captured features and make full use of the frequency features.

[0043] S4. Use the training data set A to train the defogging network LPSDNet. Specifically, put the training data set A into the defogging network LPSDNet for training until the preset loss function converges; combine the L1 loss function and the structural loss function as the objective function of the network training to obtain the trained defogging network LPSDNet.

[0044] S5. Input the image to be defogged into the trained defogging network LPSDNet for testing and verification, obtain the defogging result, and output the defogged image.

[0045] The above method is used to experiment on the RESIDE-IN (ITS) public dataset. ITS includes 13990 pairs of clean and foggy images of indoor scenes. The foggy images are generated from the clean images according to the atmospheric scattering model, where the medium extinction coefficient β is selected in the range of [0.6, 1.8] and the global atmospheric light A is selected in the range of [0.7, 1.0]. We train the proposed method for 500 rounds and test the trained network on the SOTS dataset.

[0046] Figure 5 The results of dehazing some images on the test dataset are shown, where (a) is a foggy image, (b), (c), and (d) are images after dehazing by the DCP, FFANet, and GridDehaze methods, respectively, (e) is the image after dehazing by the present invention, and (f) is the original fog-free image. It can be seen from the figure that the dehazed images generated by other methods contain artifacts left over from incomplete dehazing, while the method proposed in the present invention can effectively restore the detailed information of the image and generate a dehazed image that is close to the real image.

[0047] Figure 5 The enlarged view of each picture is as follows Figure 7-Figure 24 As shown, Figure 1 The enlarged image of the image is as follows Figure 25-Figure 30 As shown, Figure 2 The enlarged image of the image is as follows Figure 31-Figure 33 shown.

[0048] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. Those skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A dual-branch dehazing method based on Laplacian pyramid, characterized in that: The steps include: S1. Establish a training data set A including original clear images and corresponding foggy images; S2, performing data enhancement processing on the training data set A, including cropping and data enhancement processing on the input image; S3. Construct a dehazing network LPSDNet based on Laplacian pyramid; The dehazing network LPSDNet includes a multi-feature fusion focusing branch and a Laplacian pyramid supplementing branch. After the multi-feature fusion focusing branch and the Laplacian pyramid supplementing branch complete feature extraction, the features learned by the multi-feature fusion focusing branch and the Laplacian pyramid supplementing branch are further fused through the learnable fusion module. The multi-feature fusion focusing branch includes an encoder, a decoder, and a jump connection; the encoder includes multiple 3×3 convolutional layers, a supplementary fusion module, and a hybrid focus attention module, through which the encoder extracts the main feature information, reduces the size of the feature map, and increases the number of feature channels; the decoder restores the feature map output by the encoder to the same size as the original image through an upsampling layer, an SK fusion module, a hybrid focus attention module, and multiple 3×3 convolutional layers; a jump connection of a 1×1 convolutional layer is set between the encoder and the decoder to dynamically fuse shallow and deep features; S4. Use the training data set A to train the defogging network LPSDNet until the preset loss function converges; combine the L1 loss function and the structural loss function as the objective function of the network training to obtain the trained defogging network LPSDNet; S5. Input the image to be dehazed into the trained dehazing network LPSDNet for testing and verification to obtain the dehazing result.

2. The double-branch defogging method based on Laplacian pyramid according to claim 1, characterized in that: The supplementary fusion module in step S3 includes two inputs and one output, and fuses the structural information obtained by decomposing the supplementary branch of the Laplacian pyramid with the features extracted by the current branch; the operation steps of the supplementary fusion module include: S31a, for the feature Fd extracted by the multi-feature fusion focus branch and the image feature Fp obtained by the Laplace pyramid supplementary branch decomposition, first perform a 1×1 convolution operation on the input feature Fp to expand the number of channels of the feature map; then perform a 3×3 convolution operation to obtain the feature map ; S32a, in the channel dimension and Fd to obtain the concatenated feature Fcat; at the same time, Add it to Fd to get the feature Fadd; S33a, perform convolution operation on the concatenated feature Fcat to achieve feature Fusion with Fd; S34a, using a spatial attention module to generate a feature attention weight map from the fused features; wherein the spatial attention module is composed of a 1×1 convolution, a GELU activation function, a 1×1 convolution, and a Sigmoid function in sequence; S35a, multiplying the obtained attention weight map by Fadd to obtain the feature Fframe, highlighting the spatial structure information in the feature Fadd; S36a, adding the feature Fframe to the feature Fd to obtain the output feature Foutput of the supplementary fusion module.

3. The double-branch defogging method based on Laplacian pyramid according to claim 1, characterized in that: The learnable fusion module in step S3 includes two inputs and one output, and fuses the structural information obtained by decomposing the supplementary branch of the Laplacian pyramid with the features extracted by the current branch; The operation steps of the learnable fusion module include: S31b, for the feature Fd extracted by the multi-feature fusion focus branch and the image feature Fp obtained by the Laplacian pyramid supplementary branch decomposition, first perform a 1×1 convolution operation on the input feature Fp to expand the number of channels of the feature map; then perform a 3×3 convolution operation to obtain the feature map ; S32b, in the channel dimension and Fd to obtain the concatenated feature Fcat; at the same time, Add it to Fd to get the feature Fadd; S33b, perform convolution operation on the concatenated feature Fcat to achieve feature Fusion with Fd; S34b, using a spatial attention module to generate a feature attention weight map from the fused features; wherein the spatial attention module is composed of a 1×1 convolution, a GELU activation function, a 1×1 convolution, and a Sigmoid function in sequence; S35b, multiplying the obtained attention weight map by Fadd to obtain the feature Fframe, highlighting the spatial structure information in the feature Fadd; S36b. Introduce a learnable parameter a, multiply the learned parameter a with the feature Fframe and then add it to the original feature Fd.

4. The double-branch defogging method based on Laplacian pyramid according to claim 1, characterized in that: In step S3, the network introduces the multi-scale structural information extracted by the Laplacian pyramid supplementary branch into the multi-feature fusion focusing branch. The Laplacian pyramid supplementary branch uses the Laplacian pyramid decomposition module to perform multi-scale decomposition on the input foggy image to obtain features of four different scales. The output of the Laplacian pyramid supplementary branch is obtained by the following steps: S31c, among the four features of different scales, the first three multi-scale features are passed to the supplementary fusion module corresponding to the multi-feature fusion focusing branch, and the last multi-scale feature is doubled in size after the pixel padding operation and then passed to the supplementary fusion module corresponding to the multi-feature fusion focusing branch; S32c, using a supplementary fusion module and a frequency enhancement module to enhance the high-frequency features and the low-frequency features; S33c, reconstruct the enhanced high-frequency features and low-frequency features using the Laplace pyramid reconstruction module to obtain the output of the Laplace pyramid supplementary branch.

5. The double-branch defogging method based on Laplacian pyramid according to claim 4, characterized in that: The high-frequency and low-frequency feature information is extracted through the Laplacian pyramid decomposition module and the Laplacian pyramid reconstruction module. Specifically, for the input image I, the Laplacian pyramid decomposition module decomposes the input image I into a low-frequency component and several high-frequency components. The calculation process is as follows: The calculation formula of the low-frequency component L is: L=GN(I), where, Represents the result of Gaussian blurring image I N times; The calculation formula of the high-frequency component Hi of the i-th layer is as follows: ； Among them, 1≤i <N; represents the Gaussian pyramid image at the i-th level, Indicates that a 2D Gaussian kernel of size 5×5 is used for blurring. Indicates upsampling the image to the original size; Gaussian Pyramid The calculation formula is: ； in, Indicates that the image is downsampled to half the size of the original image; The Laplacian pyramid reconstruction is the inverse process of the above decomposition.

6. The double-branch defogging method based on Laplacian pyramid according to claim 4, characterized in that: The frequency enhancement module in step S32c includes two residual blocks for extracting global structural information and focusing on key features; Specifically: the first residual block uses a normalization layer to normalize the input feature map x; three convolution layers with convolution kernels of 3×3, 5×5, and 7×7 are used in parallel to extract multi-scale features of the image, and the obtained multi-scale features are concatenated to obtain the feature map x2; the input feature x2 is added to the input feature x to obtain the output feature y of the first residual block, and convolution operations with different convolution kernel sizes are used to extract high-frequency and low-frequency feature information in the image; the second residual block contains a spatial attention layer to enhance the captured features and make full use of frequency features.

7. The double-branch defogging method based on Laplacian pyramid according to claim 1, characterized in that: The hybrid focus attention module in step S3 includes two residual blocks, and the operation steps of the hybrid focus attention module include: The first residual block: S31d, for the input feature map x, the first residual block normalizes it using a normalization layer; S32d, using three convolution layers with convolution kernels of 3×3, 5×5, and 7×7 to extract multi-scale features of the image in parallel, and concatenating the obtained multi-scale features to obtain a feature map x2; S33d, adding the input feature x2 to the input feature x to obtain the output feature y of the first residual block; The second residual block: S34d, the second residual block uses multiple attention layers to further extract information from the feature map y; S35d, using normalization to normalize the feature map y to obtain feature y1; S36d, input feature y1 into the tangent attention layer, square attention layer, and re-attention layer respectively to obtain different focus features F1, F2, and F3; S37d, concatenate the focused features F1, F2, and F3, and use the channel attention layer to further extract the key features in the image to obtain feature y3; the channel attention layer is composed of global average pooling, 1×1 convolution, GELU activation function, 1×1 convolution, and Sigmoid function in sequence; S38d and y3 are added to the feature y after passing through the multi-layer perceptron to obtain the final output O of the hybrid focus attention module.

8. The double-branch defogging method based on Laplacian pyramid according to claim 7, characterized in that: In step S36d, the tangent attention layer consists of a residual connection and two consecutive 1×1 convolutional layers. The first convolutional layer is followed by a GELU activation function, and the second convolutional layer is followed by a Tanh activation function. The formula of the tangent attention layer is expressed as follows: F1= y1⊙Tanh(PWConv(GELU(PWConv(X)))) Among them, Tanh represents the hyperbolic tangent function, ⊙ represents pixel-by-pixel multiplication; the hyperbolic tangent function can map the attention weight to a range between -1 and 1, where a weight value less than zero indicates that the data at the corresponding input position is unimportant or harmful to the current task.

9. The double-branch defogging method based on Laplacian pyramid according to claim 7, characterized in that: In step S36d, the formula of the square attention layer is expressed as follows: F2= y1⊙Sigmoid(PWConv(GELU(PWConv(X)))) 2 Among them, Sigmoid represents the Sigmoid activation function. In the square attention layer, the Sigmoid activation function can map the feature value to an attention weight from 0 to 1.

10. The double-branch defogging method based on Laplacian pyramid according to claim 7, characterized in that: In step S36d, the attention layer uses continuous spatial attention to gradually focus the network on important features, which specifically includes the following steps: S36d-1: The attention layer applies spatial attention to get preliminary attention weights; S36d-2: Multiply the initial attention weight with the input feature to obtain the intermediate feature; S36d-3: Perform spatial attention operations on intermediate features to obtain refined attention weights; S36d-4: Multiply the obtained attention weight by the input to obtain the aggregate feature F3; the formula of the aggregate feature F3 is as follows: X^= y1⊙Sigmoid(PWConv(GELU(PWConv(X)))) F3= y1⊙Sigmoid(PWConv(GELU(PWConv(X^)))) Among them, X^ represents the intermediate features generated by the re-attention layer.

Images