Non-uniform haze degraded image restoration method and related device

Abstract

The invention provides a non-uniform haze degraded image restoration method and a related device, and belongs to the technical field of image processing. The method comprises the following steps: estimating noise randomly sampled from standard normal distribution by using a trained noise estimator to obtain estimated noise, and removing the estimated noise from the noise randomly sampled from the standard normal distribution to obtain a noisy image; constructing a visual state space mixed scale expansion convolution module, and constructing a conditional diffusion U-shaped denoising model based on the visual state space mixed scale expansion convolution module; and based on the guide image, estimating and denoising the noisy image by using a conditional diffusion U-shaped denoising model in an iterative calculation mode to obtain a restored non-uniform haze degraded image. According to the method, the problem that the restoration sharpness degree of the degraded image is poor when the haze concentration change range is large or the non-uniform change is severe is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for restoring a non-uniform haze image and a related device. Background Art

[0002] Haze weather usually causes blurry target scenes, low contrast, and short viewing distance due to light scattering. In particular, non-uniform haze causes inconsistent haze concentration distribution in the target scene, which can lead to loss of detailed information of the observed target at best and even submergence of the overall target information at worst, seriously affecting the observation of the target scene and the application of computer vision tasks.

[0003] For example, non-uniform haze obscures the target scene and shortens the viewing distance, easily causing the driver to misjudge the target scene. Furthermore, because the degraded image presents less scene information, the autonomous driving function can easily reduce the detection and recognition rates of traffic signs, pedestrians, vehicles, and other targets. The blurring effect of non-uniform haze on the scene seriously reduces the intelligence level of application scenarios such as smart security and smart transportation, and increases the difficulty and cost of developing such computer vision tasks. Therefore, it is urgent to develop methods for restoring degraded images with blur, low contrast, and short viewing distance in non-uniform haze scattering weather conditions, laying the foundation for clear perception of target scenes in non-uniform haze scattering weather conditions.

[0004] Currently, most methods for restoring haze-degraded images use the following two methods: 1) haze-degraded image restoration methods based on atmospheric scattering models or mathematical modeling and inversion calculation of prior information; 2) haze-degraded image restoration methods based on deep learning feature extraction.

[0005] When restoring haze-degraded images using inversion calculations based on atmospheric scattering models or mathematical modeling of prior information, it is necessary to find prior information and use the atmospheric scattering model to estimate unknown parameters such as atmospheric transmittance and atmospheric light value. The accuracy of the estimation of these unknown parameters seriously affects the restoration effect of the degraded image, but accurate estimation of unknown parameters is difficult and time-consuming.

[0006] The haze-degraded image restoration method uses deep learning feature extraction. By constructing a neural network for feature extraction, it can learn the mapping relationship between haze-degraded images and clear images driven by sample data. This nonlinear mapping relationship is usually difficult to express through mathematical formulas, and has a certain degree of robustness for different haze distribution states and target scenes. It has now developed into the mainstream method for haze-degraded image restoration.

[0007] Haze-degraded image restoration methods based on deep learning feature extraction include CNN (Convolutional Neural Network)-based methods, Transformer-based methods, and generative image restoration methods based on diffusion models.

[0008] Due to the ever-changing distribution of haze concentration in haze scenes, end-to-end image restoration methods based on deep learning (such as CNN) find it difficult to accurately map the relationship between degraded images and clear images, which may result in poor clarity in the boundary areas where haze concentration changes.

[0009] The generative image restoration method based on the diffusion model learns the inherent properties of the target scene to be clarified by constraining the distribution state between adding noise and removing noise, and then generates a clarified target image from a noise image that obeys a Gaussian distribution. It performs well in restoring degraded images when the haze distribution state changes. However, this method only performs well in restoring haze-degraded images with a small range of haze concentration changes. When the haze concentration changes over a large range or the non-uniform changes are drastically, the clarity of the restored degraded image is poor. Summary of the Invention The purpose of the present invention is to provide a method and related device for restoring a non-uniform haze-degraded image, which is used to solve the problem in the prior art that the clarity of the degraded image restoration is poor when the haze concentration varies widely or the non-uniform changes are drastic.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a method for restoring a non-uniform haze-degraded image, comprising the following steps: Obtaining a non-uniform haze-degraded image and a ground-truth image corresponding to the non-uniform haze-degraded image; Construct a U-shaped feature extraction network and use it to extract deep feature information of non-uniform haze-degraded images; The depth feature information is added to the residual of the non-uniform haze-degraded image to obtain the non-uniform haze-degraded image enhancement result; Construct a dynamic fusion network module, use the dynamic fusion network module to obtain the weights corresponding to the non-uniform haze degraded image and the non-uniform haze degraded image enhancement results, and fuse the non-uniform haze degraded image and the non-uniform haze degraded image enhancement results based on the weights to obtain a guidance image; Forward diffuse the true value image corresponding to the non-uniform haze-degraded image into the standard normal distribution noise to obtain the trained noise estimator; The noise randomly sampled from the standard normal distribution is estimated using the trained noise estimator to obtain the estimated noise, and the estimated noise is removed from the noise randomly sampled from the standard normal distribution to obtain a noisy image; Construct a visual state space mixed-scale dilated convolution module, and build a conditional diffusion U-shaped denoising model based on the visual state space mixed-scale dilated convolution module; Based on the guide image, the conditional diffusion U-shaped denoising model is used to estimate and denoise the noisy image through iterative calculation to obtain the restored non-uniform haze-degraded image.

[0011] A further improvement of the present invention is that, in the step of constructing a U-shaped feature extraction network and using the U-shaped feature extraction network to extract depth feature information of the non-uniform haze-degraded image, the U-shaped feature extraction network is specifically constructed based on a dynamic depth-by-depth convolution group, and the dynamic depth-by-depth convolution group includes several dynamic depth-by-depth convolution blocks, and each dynamic depth-by-depth convolution block includes several residual blocks.

[0012] A further improvement of the present invention is that the calculation formula of the guide image is:

[0013] in, x c represents the guide image, I x represents a non-uniform haze-degraded image, w x represents the weight of the non-uniform haze-degraded image, I p represents the enhancement result of non-uniform haze degradation image, w p represents the weight of the non-uniform haze degradation image enhancement result, C 1×1 (·) represents a convolution operation with a convolution kernel of 1×1. Relu (·) represents the ReLU nonlinear activation function, Sigmoid (·) represents the Sigmoid nonlinear activation function.

[0014] A further improvement of the present invention lies in that, in the steps of constructing a visual state space mixed-scale dilated convolution module and constructing a conditional diffusion U-type denoising model based on the visual state space mixed-scale dilated convolution module, a visual state space mixed-scale dilated convolution module is specifically constructed based on the visual state space module and mixed-scale adaptive channel attention, the visual state space module is used to extract global features, and the mixed-scale adaptive channel attention is used to extract local features.

[0015] A further improvement of the present invention is that the specific calculation process of the visual state space module is:

[0016] in, X Emb represents the input of the visual state space module, Linear (·) represents a linear operation, DConv 3×3 (·) represents a depthwise convolution operation with a convolution kernel of 3×3, 2 DSSM (·) represents a 2D selective sweep state space operation, LNorm (·) represents the layer regularization operation, F XEmb Indicates that the visual state space module inputs the feature expression after linear mapping and depth convolution, M XEmb Input the visual state space module into the feature matrix after 2D selective scanning, Fvssm Represents the feature output expression after the visual state space module.

[0017] A further improvement of the present invention is that the specific calculation process of the mixed-scale adaptive channel attention is:

[0018] in, DConv n×n (·) indicates the convolution kernel size is n × n The depth convolution operation, represents the element-by-element addition operation, Indicates channel-based splicing operation. F inca represents the input of mixed-scale adaptive channel attention, F msf Represents the multi-scale feature extraction fusion result, F outmca represents the output of mixed-scale adaptive channel attention, CA (·) represents the channel attention calculation process, C 1×1 (·) represents a convolution operation with a 1×1 kernel.

[0019] A further improvement of the present invention is that the specific calculation process for restoring the non-uniform haze-degraded image is as follows:

[0020] in, P θ (·) represents the Markov chain representation in the reverse diffusion process of the conditional diffusion model, express t The noisy image at time t, express t -1 moment after removing noise, x c represents the guide image, σ t represents the time-dependent variance, t Indicates the time stamp during the iteration process, Indicates that the parameter is θ The mean learned from the conditional diffusion U-shaped denoising model; The calculation formula is:

[0021] in, represents the noise image predicted by the conditional diffusion U-shaped denoising model, Indicates that at time step t The noise variance added to the true value image corresponding to the non-uniform haze degraded image is, Indicates that at time step t The original signal ratio is retained, , express The product of multiple accumulations; It is obtained through the loss function, and the calculation formula of the loss function is:

[0022] in, L ( θ ) represents the loss function of the conditional diffusion U-shaped denoising model, represents the noise randomly sampled from the standard normal distribution, represents the L2 constraint, Represents the guide image The corresponding reference image, Express expectations.

[0023] In a second aspect, the present invention provides a non-uniform haze-degraded image restoration system, comprising a data acquisition module, a U-shaped feature extraction network construction module, a residual addition module, a guided image acquisition module, a forward diffusion module, a noisy image acquisition module, a U-shaped denoising model construction module, and a degraded image restoration module; The data acquisition module is used to acquire a non-uniform haze-degraded image and a true value image corresponding to the non-uniform haze-degraded image; The U-shaped feature extraction network construction module is used to construct a U-shaped feature extraction network, and use the U-shaped feature extraction network to extract depth feature information of the non-uniform haze-degraded image; The residual addition module is used to perform residual addition on the depth feature information and the non-uniform haze-degraded image to obtain an enhanced result of the non-uniform haze-degraded image; The guide image acquisition module is used to construct a dynamic fusion network module, use the dynamic fusion network module to obtain weights corresponding to the non-uniform haze degraded image and the non-uniform haze degraded image enhancement results, and fuse the non-uniform haze degraded image and the non-uniform haze degraded image enhancement results based on the weights to obtain a guide image; The forward diffusion module is used to forward diffuse the true value image corresponding to the non-uniform haze-degraded image into standard normal distribution noise to obtain a trained noise estimator; The noisy image acquisition module is used to estimate the noise randomly sampled from the standard normal distribution using a trained noise estimator to obtain estimated noise, and remove the estimated noise from the noise randomly sampled from the standard normal distribution to obtain a noisy image; The U-shaped denoising model construction module is used to construct a visual state space mixed-scale dilated convolution module, and to construct a conditional diffusion U-shaped denoising model based on the visual state space mixed-scale dilated convolution module; The degraded image restoration module is used to estimate and denoise the noisy image based on the guide image using the conditional diffusion U-type denoising model through iterative calculation to obtain a restored non-uniform haze-degraded image.

[0024] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-described method for restoring non-uniform haze-degraded images when executing the computer program.

[0025] In a fourth aspect, the present invention provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method for restoring a non-uniform haze-degraded image.

[0026] Compared with the prior art, the present invention has the following beneficial effects: The present invention is an improved invention. Compared with existing methods for restoring non-uniform haze-degraded images, on the one hand, the present invention utilizes a U-shaped feature extraction network to extract depth feature information of the non-uniform haze-degraded image, and adds the depth feature information to the non-uniform haze-degraded image to obtain a non-uniform haze-degraded image enhancement result. This operation can enhance and restore the feature information in the non-uniform haze image, so that it contains more effective feature information and reduces erroneous feature information. It further guides the reverse sampling process of the diffusion model in the conditional diffusion model, thereby effectively improving the speed and accuracy of the conditional diffusion model in restoring the non-uniform haze-degraded image. On the other hand, based on the guided image, the present invention utilizes the conditional diffusion U-shaped denoising model to estimate and denoise the noisy image through iterative calculation to obtain a restored non-uniform haze-degraded image. This operation can improve the contrast and clarity of the non-uniform haze-degraded image, thereby effectively solving the problem in the prior art of poor clarity in degraded image restoration when the haze concentration varies widely or the non-uniform changes are severe. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of the method for restoring a non-uniform haze-degraded image according to the present invention; Figure 2 Schematic diagram of the non-uniform haze-degraded image restoration system of the present invention; Figure 3 This is a schematic diagram of the principle of the method for restoring a non-uniform haze-degraded image in Example 3 of the present invention; Figure 4 This is a network structure diagram of the dynamic depth-by-depth convolution group of the present invention; Figure 5 This is a network structure diagram of the visual state space mixed-scale dilated convolution module of the present invention; Figure 6 The figure shows the comparison results of the qualitative evaluation of the present invention and the mainstream image restoration methods on the NH-Haze dataset; Figure 7 The comparison results of the quantitative evaluation of the present invention and mainstream image restoration methods on the NH-Haze dataset are shown in the figure; Figure 8 This is a comparison diagram of feature point matching results before and after restoration of a non-uniform haze-degraded image using the method of the present invention; Figure 9 This is a comparison diagram of edge detection results before and after restoration of a non-uniform haze-degraded image using the method of the present invention; Figure 10 Schematic diagram of the structure of the electronic device of the present invention. DETAILED DESCRIPTION

[0028] In order to further understand the content of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the embodiments are only for explaining the present invention and are not intended to limit it.

[0029] Example 1: The flowchart of the method for restoring non-uniform haze-degraded images of the present invention is as follows: Figure 1 As shown, the method for restoring a non-uniform haze-degraded image of the present invention comprises the following steps: S1. Obtain a non-uniform haze-degraded image and a ground-truth image corresponding to the non-uniform haze-degraded image.

[0030] S2. Construct a U-shaped feature extraction network and use it to extract deep feature information of non-uniform haze-degraded images.

[0031] S3. Add the residual of the depth feature information to the non-uniform haze-degraded image to obtain the non-uniform haze-degraded image enhancement result.

[0032] S4. Construct a dynamic fusion network module, use the dynamic fusion network module to obtain the weights corresponding to the non-uniform haze-degraded image and the non-uniform haze-degraded image enhancement results, and fuse the non-uniform haze-degraded image and the non-uniform haze-degraded image enhancement results based on the weights to obtain a guidance image.

[0033] S5. Forward diffuse the true value image corresponding to the non-uniform haze-degraded image into standard normal distribution noise to obtain a trained noise estimator.

[0034] S6. Use the trained noise estimator to estimate the noise randomly sampled from the standard normal distribution to obtain estimated noise, and remove the estimated noise from the noise randomly sampled from the standard normal distribution to obtain a noisy image.

[0035] S7. Construct a visual state space mixed-scale dilated convolution module, and construct a conditional diffusion U-shaped denoising model based on the visual state space mixed-scale dilated convolution module.

[0036] S8. Based on the guidance image, the conditional diffusion U-shaped denoising model is used to estimate and denoise the noisy image through iterative calculation to obtain the restored non-uniform haze-degraded image.

[0037] Example 2: The schematic diagram of the non-uniform haze degradation image restoration system of the present invention is as follows: Figure 2 As shown, the non-uniform haze-degraded image restoration system of the present invention includes a data acquisition module, a U-shaped feature extraction network construction module, a residual addition module, a guided image acquisition module, a forward diffusion module, a noisy image acquisition module, a U-shaped denoising model construction module and a degraded image restoration module.

[0038] The data acquisition module is used to obtain the non-uniform haze-degraded image and the true value image corresponding to the non-uniform haze-degraded image.

[0039] The U-shaped feature extraction network construction module is used to construct a U-shaped feature extraction network, and use the U-shaped feature extraction network to extract the deep feature information of non-uniform haze-degraded images.

[0040] The residual addition module is used to perform residual addition on the depth feature information and the non-uniform haze-degraded image to obtain the non-uniform haze-degraded image enhancement result.

[0041] The guidance image acquisition module is used to construct a dynamic fusion network module, which uses the dynamic fusion network module to obtain the weights corresponding to the non-uniform haze degraded image and the non-uniform haze degraded image enhancement results. Based on the weights, the non-uniform haze degraded image and the non-uniform haze degraded image enhancement results are fused to obtain the guidance image.

[0042] The forward diffusion module is used to forward diffuse the true value image corresponding to the non-uniform haze-degraded image into the standard normal distribution noise to obtain a trained noise estimator.

[0043] The noisy image acquisition module is used to estimate the noise randomly sampled from the standard normal distribution using a trained noise estimator to obtain estimated noise, and remove the estimated noise from the noise randomly sampled from the standard normal distribution to obtain a noisy image.

[0044] The U-shaped denoising model construction module is used to construct a visual state space mixed-scale dilated convolution module, and a conditional diffusion U-shaped denoising model is constructed based on the visual state space mixed-scale dilated convolution module; The image restoration module is used to estimate and denoise the noisy image based on the guide image and use the conditional diffusion U-type denoising model through iterative calculation to obtain the restored non-uniform haze-degraded image.

[0045] Example 3: The principle diagram of the method for restoring non-uniform haze-degraded images of the present invention is as follows: Figure 3 As shown, the method for restoring a non-uniform haze-degraded image of the present invention comprises the following steps: S1. Obtain a non-uniform haze-degraded image and a ground-truth image corresponding to the non-uniform haze-degraded image.

[0046] S2. Construct a U-shaped feature extraction network and use it to extract deep feature information of non-uniform haze-degraded images.

[0047] In this step, a U-shaped feature extraction network is constructed based on a dynamic depthwise convolution group (DDCG). The dynamic depthwise convolution group includes several dynamic depthwise convolution blocks (two dynamic depthwise convolution blocks in this embodiment, the full name of the dynamic depthwise convolution block is Dynamic Depthwise Convolution Block, abbreviated as DDCB). Each dynamic depthwise convolution block includes several residual blocks (two residual blocks in this embodiment). The network structure of the dynamic depthwise convolution group is shown in the following figure: Figure 4 shown.

[0048] The first residual block is used to extract spatial features, which is implemented by the Norm regularization operation, DynamicConv dynamic convolution operation and DropPath operation. The second residual block is used to extract channel features, which is implemented by the Norm regularization operation and the channel attention module.

[0049] The following is a detailed description of the DynamicConv dynamic convolution operation and channel attention module: DynamicConv dynamic convolution operation optimizes dynamic weights through learnable parameters to obtain dynamic convolution kernels, and then obtains spatial extraction features through depth-by-depth convolution with input embedded features. The specific structure of dynamic convolution is as follows: Figure 4 As shown. The learnable parameters are implemented as follows: global average pooling of the input embedding features, after 1×1 convolution and BatchNorm batch normalization, activated by the GELU nonlinear function, and then divided by channel after 1×1 convolution and activated by the Softmax nonlinear function to obtain the learnable parameters for optimizing dynamic weights. The dynamic weights are implemented as follows: adaptive average pooling of the input embedding features, adjusting the dimension size through the Reshape operation, after 1×1 convolution and BatchNorm batch normalization, activated by the GELU nonlinear function, and then divided by channel after 1×1 convolution and activated by the Softmax nonlinear function to obtain dynamic weights. The specific operation process of DynamicConv dynamic convolution is:

[0050] in, I FE Represents the spatial extraction features after the DynamicConv dynamic convolution operation, K DC represents the dynamic convolution kernel, I IE represents the input embedding features, P L represents the learnable parameters, WD represents the dynamic weight, Softmax (·) represents the softmax nonlinear activation function, Split (·) indicates the operation is divided by channel, GELU (·) is the GELU nonlinear activation function, BN (·) represents the batch normalization operation, GMP (·) represents global average pooling, AAP (·) represents adaptive average pooling, R (·) indicates the Reshape operation.

[0051] CAB (Channel Attention Block) is activated by 3×3 convolution and GELU nonlinear function, and then implemented by 3×3 convolution and Channel Attention operation. The specific structure of the channel attention module is as follows: Figure 4 As shown. The specific implementation process of the channel attention matrix is: adaptive average pooling is performed on the input features, activated by 1×1 convolution and RELU nonlinear function, and then activated by 1×1 convolution and Sigmoid nonlinear function to obtain the channel attention matrix. The implementation process of the channel attention module can be expressed as:

[0052] in, I CABI represents the input of the channel attention module, C 3×3 (·) represents a convolution operation with a convolution kernel of 3×3. I CAI represents the input of the channel attention operation of the channel attention module, I CABO is the output of the channel attention module.

[0053] The following is a detailed description of the process of extracting deep feature information of non-uniform haze-degraded images using the U-shaped feature extraction network: The input of the U-shaped feature extraction network is the non-uniform haze-degraded image I x Vector after Embedding I emb, the size of the feature map does not change after DDCG dynamic convolution processing. The downsampling operation of the U-shaped feature extraction network is "CNN+PixelUnshuffle", which first reduces the number of channels of the input feature map by half through convolution, and then uses PixelUnshuffle to reduce the two-dimensional size of the input feature map by half, while increasing the number of channels of the convolved feature map by 4 times. Therefore, after the downsampling operation, the input feature map is Figure 2 The upsampling operation on the U-shaped feature extraction network is "CNN+PixelShuffle", which first increases the number of channels of the input feature map by 2 times through convolution, and then uses PixelUnshuffle to increase the two-dimensional size of the input feature map by 2 times, while reducing the number of channels of the convolved feature map by 4 times. Therefore, after the upsampling operation, the input feature map is Figure 2 The dimensional size is doubled while the number of channels is halved. The upsampling process uses cross-layer connections to splice the downsampled feature information of the same level with the upsampled feature information by channel, and then uses convolution to halve the number of channels before sending it to the next upsampling DDCG. After the four-layer U-shaped feature extraction network, the deep feature information of the non-uniform haze degraded image is obtained. I dd .

[0054] S3. Add the residual of the depth feature information to the non-uniform haze-degraded image to obtain the non-uniform haze-degraded image enhancement result.

[0055] Specifically, the depth feature information of non-uniform haze-degraded images I dd After convolution operation, spatial projection is performed to obtain I hd , I hd The residual is added element by element with the non-uniform haze degraded image to obtain the enhanced result of the non-uniform haze degraded image. I p .

[0056] S4. Construct a dynamic fusion network module, use the dynamic fusion network module to obtain the weights corresponding to the non-uniform haze-degraded image and the non-uniform haze-degraded image enhancement results, and fuse the non-uniform haze-degraded image and the non-uniform haze-degraded image enhancement results based on the weights to obtain a guidance image.

[0057] The dynamic fusion network module is activated by 1×1 convolution and Relu nonlinearity, and then by 1×1 convolution and Sigmoid nonlinearity.

[0058] The calculation formula of the guide image in this step is:

[0059] in, x c represents the guide image, I x represents a non-uniform haze-degraded image, w x represents the weight of the non-uniform haze-degraded image, I p represents the enhancement result of non-uniform haze degradation image, w p represents the weight of the non-uniform haze degradation image enhancement result, C 1×1 (·) represents a convolution operation with a convolution kernel of 1×1. Relu (·) represents the ReLU nonlinear activation function, Sigmoid (·) represents the Sigmoid nonlinear activation function.

[0060] S5. Forward diffuse the true value image corresponding to the non-uniform haze-degraded image into standard normal distribution noise to obtain a trained noise estimator.

[0061] The following describes step S5 in detail: The diffusion model consists of forward diffusion and backward sampling processes. The forward diffusion process follows the Markov chain and uses the true value image (the true value image corresponding to the non-uniform haze degradation image) as input for diffusion. By adding noise that obeys the Gaussian normal distribution to the true value image, T After the first diffusion step, the ground-truth image is diffused into a noise pattern that follows a Gaussian normal distribution. This forward diffusion process indirectly introduces labeled data into the network training process by adding noise. The forward diffusion process's diffusion pattern depends on the computing device's video memory and the input image resolution. If the computing device has limited video memory and a high-resolution image, the forward diffusion process can be performed by extracting small blocks of a certain size from the high-resolution image and iterating the process multiple times.

[0062] In the forward diffusion process, the image is guided x c As a condition, from t -1 time has arrived t The diffusion process calculation formula at the moment is:

[0063] in, represents a hyperparameter that determines the variance of the noise distribution added during each iteration. , represents the variance, x t-1 and x t Represents the time stept- 1 and time step t The diffusion result is , T represents the total number of diffusion steps, q represents the Karkov chain forward diffusion of the diffusion model, N represents a normal distribution, I Represents the identity matrix.

[0064] According to the Markov chain rule, the image to be diffused at the initial moment can be x 0, after T After diffusion, we get T Always obey the Gaussian normal distribution x T , the diffusion process can be expressed as:

[0065] in, x 0 represents the image to be diffused input at the initial moment (the true value image corresponding to the non-uniform haze-degraded image), .through T After the diffusion step, x T represents the noise that obeys the standard normal distribution (standard normal distribution noise). T During the diffusion process, the noise added in each diffusion step ε ~ N (0,1), the added T The noises will form a noise sequence, which is used as a label for network constraints when training the noise estimator, and then a trained noise estimator is obtained.

[0066] S6. Use the trained noise estimator to estimate the noise randomly sampled from the standard normal distribution to obtain estimated noise, and remove the estimated noise from the noise randomly sampled from the standard normal distribution to obtain a noisy image.

[0067] S7. Construct a visual state space mixed-scale dilated convolution module, and construct a conditional diffusion U-shaped denoising model based on the visual state space mixed-scale dilated convolution module.

[0068] In this step, a visual state-space mixed-scale dilated convolution block (VSMDCB, also called visual state-space mixed-scale dilated convolution block) is constructed based on the Visual State Space Module (VSSM) and Mixed-Scale Adaptive Channel Attention (MSACA). The visual state space module is used to extract global features, and the mixed-scale adaptive channel attention extracts local features through multi-scale deep convolution operations and channel attention mechanism. The network structure diagram of the visual state space mixed-scale dilated convolution module is shown in the figure below. Figure 5 shown.

[0069] The input of the visual state space mixed scale dilated convolution module is the time step t Embedding and 6 channels after the guide image and the sampled noise image are spliced X Embedding amount, time steps t The embedding amount is decomposed into 6 quantities after the multi-layer perceptron MLP, namely scale1, shift1 and gate1 and scale2, shift2 and gate2. Scale1 and shift1 are input to Modulate for layer regularization. X Perform linear mapping, and then linearly map the X After the tensor is expressed by VSSM global features, X The embedding amount is adaptively gated through gate1; similarly, scale2 and shift2 are input to Modulate for linear mapping of the adaptive gated output after layer regularization. The feature tensor that has undergone linear mapping again is expressed by MSACA multi-scale adaptive attention features and then adaptively gated with the first adaptive gated output again through gate2 to achieve two-dimensional global and local feature expression and channel local feature extraction.

[0070] The Visual State Space Module is a visual representation learning model that combines the advantages of CNNs and ViTs. It achieves linear complexity without sacrificing the global receptive field and significantly improves computational efficiency. VSSM achieves the representation and update of visual states through its unique network structure. First, the patches of the input image are converted into high-dimensional feature vectors through a linear embedding layer, depth-wise convolution, and nonlinear mapping operations; then, these feature vectors are processed through a series of VSS (Visual State Space) blocks. Each SSM block contains a 2D selective scanning (2D-SSM) module. In this 2D-SSM module, the feature vectors are flattened into a 1D sequence in four directions and interact through the selective scanning space state sequence model operation to achieve state update; finally, the updated state is operated through a normalization layer and matrix multiplication is performed with the patches that have undergone linear mapping and depth-wise convolution, and then the final visual representation is obtained after linear mapping. The specific calculation process of the visual state space module is as follows:

[0071] in, X Emb represents the input of the visual state space module, Linear (·) represents a linear operation, DConv 3×3 (·) represents a depthwise convolution operation with a convolution kernel of 3×3, 2 DSSM (·) represents a 2D selective sweep state space operation, LNorm (·) represents the layer regularization operation, F XEmb Indicates that the visual state space module inputs the feature expression after linear mapping and depth convolution, M XEmb Input the visual state space module into the feature matrix after 2D selective scanning, Fvssm Represents the feature output expression after the visual state space module.

[0072] Hybrid-scale adaptive channel attention combines multi-scale feature extraction (multi-scale deep convolution operations) with the channel attention mechanism to extract local features in the channel. It uses deep convolution operations with kernel sizes of 3×3, 5×5, and 7×7. The feature information extracted by the 5×5 convolution kernel is fused with the large receptive field and small receptive field convolution features respectively. After reducing the number of channels through 1×1 convolution, it is input into the channel attention module. The output of the channel attention module is further reduced through 1×1 convolution to extract local features in the channel. The specific calculation process of hybrid-scale adaptive channel attention is as follows:

[0073] in, DConv n×n (·) indicates the convolution kernel size is n × n (n is 3, 5 and 7) depthwise convolution operation, represents the element-by-element addition operation, Indicates channel-based splicing operation. F inca represents the input of mixed-scale adaptive channel attention, F msf Represents the multi-scale feature extraction fusion result, F outmca represents the output of mixed-scale adaptive channel attention, CA (·) represents the channel attention calculation process, C 1×1 (·) represents a convolution operation with a 1×1 kernel.

[0074] S8. Based on the guidance image, the conditional diffusion U-shaped denoising model is used to estimate and denoise the noisy image through iterative calculation to obtain the restored non-uniform haze-degraded image.

[0075] The specific calculation process for restoring the non-uniform haze-degraded image in this step is:

[0076] in, P θ (·) represents the Markov chain representation in the reverse diffusion process of the conditional diffusion model, express t The noisy image at time t, express t -1 moment after removing noise, x c represents the guide image, σ t represents the time-dependent variance, t Indicates the time stamp during the iteration process, Indicates that the parameter is θ The mean learned from the conditional diffusion U-shaped denoising model; The calculation formula is:

[0077] in, represents the noise image predicted by the conditional diffusion U-shaped denoising model, Indicates that at time step t The noise variance added to the true value image corresponding to the non-uniform haze degraded image is, Indicates that at time stept The original signal ratio is retained, , express The product of multiple accumulations.

[0078] It is obtained through the loss function, and the calculation formula of the loss function is:

[0079] in, L ( θ ) represents the loss function of the conditional diffusion U-shaped denoising model, represents the noise randomly sampled from the standard normal distribution, represents the L2 constraint, Represents the guide image The corresponding reference image, Express expectations.

[0080] The following is a detailed description of the conditional diffusion U-shaped denoising model: The encoding and decoding parts of the conditional diffusion U-type denoising model are composed of 4 layers of symmetrical visual state space mixed-scale dilated convolution modules. The visual state space mixed-scale dilated convolution module is used for feature extraction, and the size of the input and output feature maps does not change. The dimensional size of the temporal embedding is different at different levels in the encoding and decoding network, but it is consistent with the dimensional size of the image embedding input to the visual state space mixed-scale dilated convolution module of each layer. During decoding, in order to avoid the loss of detail information during the encoding process, cross-layer connections are used to connect the encoding layer features of the same level to the decoding layer in a channel-by-channel splicing manner, and a 1×1 convolution operation is used to reduce the number of channels in the decoding layer. The information obtained from the decoding layer of the conditional diffusion U-type denoising model is again represented in the high-dimensional feature space by the visual state space mixed-scale dilated convolution module, and finally a 3-channel two-dimensional image is obtained after the channel is compressed by a 3×3 convolution layer. ( t -1 moment after removing noise).

[0081] To verify the effectiveness of the proposed method for restoring non-uniform haze-degraded images, this example used high-resolution non-uniform haze-degraded images from the NH-Haze dataset for network training. The samples in this dataset are paired samples, with 157 training images and 13 test images. Due to the high image resolution, the randomly captured block size during training and testing was 128×128. During network training, the total diffusion step size was 1000, the initial learning rate was 2e-5, the final learning rate was 2e-6, the learning rate decay strategy was linear, the optimizer was Adam, the total number of iterations was 100,000, and the skip sampling step size was 25. The hardware environment for network training was an Intel Xeon Silver 4210R @ 2.4GHz, 128GB of memory, and three NVIDIA GeForce RTX 3090 24GB GPUs.

[0082] This example restores 13 paired non-uniform haze-degraded images and compares them with nine current mainstream deep learning-based methods (GridDehaze, MSBDN, FFA-Net, 4Kdehazing, Dehamer, DehazeFormer, DiffUIR, MixDehaze, and DEA-Net).

[0083] The qualitative comparison results of the proposed method with the mainstream image restoration methods on the NH-Haze dataset are as follows: Figure 6 As shown, Figure 6 Inhomogeneous refers to an ordinary non-uniform haze-degraded image, where the haze concentration is only high in a local area, emphasizing the non-uniform characteristics. Dense refers to a dense fog-inhomogeneous haze-degraded image, where the haze concentration is high overall, emphasizing the non-uniform characteristics in extreme cases. Figure 6 It can be seen that the non-uniform haze-degraded image restored by the method of the present invention ( Figure 6 The clarity of the restored non-uniform haze images (the image column marked Ours) has been improved, especially the overall clarity has been consistent. The contrast of the restored non-uniform haze images is high, the viewing distance of the restored non-uniform haze images has been increased, and the edge details are clear, without any blurred, overly bright, or overly dark areas.

[0084] To further quantitatively evaluate the quality of restored images degraded by non-uniform haze, this example performs a statistical average analysis on 13 paired images degraded by non-uniform haze, using both a reference evaluation metric based on Peak Signal-to-Noise Ratio (PSNR) and a no-reference evaluation metric based on the Natural Image Quality Evaluator (NIQE). PSNR is calculated by calculating the mean squared error between the restored image (the restored non-uniform haze-degraded image) and the true image. Higher PSNR values ​​indicate less distortion and better quality. NIQE evaluates image quality without comparison with the true image; lower NIQE values ​​indicate a higher overall naturalness of the image (the restored non-uniform haze-degraded image).

[0085] The comparison results of the quantitative evaluation of the present invention and the mainstream image restoration methods on the NH-Haze dataset are as follows: Figure 7 As shown, from Figure 7 As can be seen, the PSNR evaluation value of the present invention is the highest, indicating that the present invention has the best effect on the clarity, contrast, and edge detail of the non-uniform haze-degraded image. The NIQE evaluation value of the present invention is the lowest, indicating that the restored image (the restored non-uniform haze-degraded image) of the present invention has the highest overall naturalness.

[0086] In order to further verify the effect of the method of the present invention in computer vision applications, this embodiment experimentally tests the effectiveness of the non-uniform haze-degraded image restoration method proposed in the present invention in feature point detection and edge detection.

[0087] This embodiment uses the SIFT feature point detection algorithm to evaluate the detection effect of feature points before and after restoration of non-uniform haze-degraded images. Figure 8 As shown, Figure 8 The number in the lower left corner indicates the number of matched feature points detected by the SIFT feature point detection algorithm on the non-uniform haze-degraded image. The larger the number, the better the detection effect. Figure 8 The image marked (a) shows the feature point matching between the non-uniform haze degraded image and the reference image. Figure 8 The image marked (b) shows the feature point matching between the restored non-uniform haze image and the reference image. Figure 8 It can be seen that after the method of the present invention is used to restore the non-uniform haze-degraded image, the number of paired feature points detected by the SIFT feature point detection algorithm increases significantly, indicating that the clarity and contrast of the restored non-uniform haze-degraded image are improved.

[0088] This embodiment uses the Canny edge detection algorithm to evaluate the impact of non-uniform haze degradation image restoration on edge detection. Figure 9 As shown, Figure 9 The image marked (a) shows the non-uniform haze degradation image and its edge detection. Figure 9 The image marked (b) shows the non-uniform haze-degraded image restored by the method of the present invention and its edge detection. Figure 9 It can be seen from the figure that after the non-uniform haze-degraded image is restored using the method of the present invention, the target edges detected by the Canny algorithm are significantly increased, indicating that the image detail information after the non-uniform haze-degraded image is restored using the method of the present invention is improved.

[0089] In summary, the non-uniform haze-degraded image restoration method proposed in the present invention can realize non-uniform haze-degraded image restoration, which not only improves the contrast and clarity of non-uniform haze-degraded images, but also compensates for the detail information of non-uniform haze-degraded images, and further improves the target scene perception performance in haze environments.

[0090] Example 4: See also Figure 10 As shown, the present invention also provides an electronic device 100 for a method for restoring a non-uniform haze-degraded image; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0091] The memory 101 can be used to store the computer program 103. The processor 102 implements the steps of the method for restoring a non-uniform haze-degraded image described in Example 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 may mainly include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data (such as audio data) generated based on the use of the electronic device 100. In addition, the memory 101 may include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device.

[0092] The at least one processor 102 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor, etc. The processor 102 is the control center of the electronic device 100 and connects various parts of the entire electronic device 100 using various interfaces and lines.

[0093] The memory 101 in the electronic device 100 stores a plurality of instructions to implement a method for restoring a non-uniform haze-degraded image. The processor 102 may execute the plurality of instructions to implement: Obtaining a non-uniform haze-degraded image and a ground-truth image corresponding to the non-uniform haze-degraded image; Construct a U-shaped feature extraction network and use it to extract deep feature information of non-uniform haze-degraded images; The depth feature information is added to the residual of the non-uniform haze-degraded image to obtain the non-uniform haze-degraded image enhancement result; Construct a dynamic fusion network module, use the dynamic fusion network module to obtain the weights corresponding to the non-uniform haze degraded image and the non-uniform haze degraded image enhancement results, and fuse the non-uniform haze degraded image and the non-uniform haze degraded image enhancement results based on the weights to obtain a guidance image; Forward diffuse the true value image corresponding to the non-uniform haze-degraded image into the standard normal distribution noise to obtain the trained noise estimator; The noise randomly sampled from the standard normal distribution is estimated using the trained noise estimator to obtain the estimated noise, and the estimated noise is removed from the noise randomly sampled from the standard normal distribution to obtain a noisy image; Construct a visual state space mixed-scale dilated convolution module, and build a conditional diffusion U-shaped denoising model based on the visual state space mixed-scale dilated convolution module; Based on the guide image, the conditional diffusion U-shaped denoising model is used to estimate and denoise the noisy image through iterative calculation to obtain the restored non-uniform haze-degraded image.

[0094] Example 5: If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory and read-only memory (ROM, Read-Only Memory).

[0095] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0097] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0098] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A method for restoring a non-uniform haze-degraded image, characterized in that: The following steps are involved: Obtaining a non-uniform haze-degraded image and a ground-truth image corresponding to the non-uniform haze-degraded image; Construct a U-shaped feature extraction network and use it to extract deep feature information of non-uniform haze-degraded images; The depth feature information is added to the residual of the non-uniform haze-degraded image to obtain the non-uniform haze-degraded image enhancement result; Construct a dynamic fusion network module, use the dynamic fusion network module to obtain the weights corresponding to the non-uniform haze degraded image and the non-uniform haze degraded image enhancement results, and fuse the non-uniform haze degraded image and the non-uniform haze degraded image enhancement results based on the weights to obtain a guidance image; Forward diffuse the true value image corresponding to the non-uniform haze-degraded image into the standard normal distribution noise to obtain the trained noise estimator; The noise randomly sampled from the standard normal distribution is estimated using the trained noise estimator to obtain the estimated noise, and the estimated noise is removed from the noise randomly sampled from the standard normal distribution to obtain a noisy image; Construct a visual state space mixed-scale dilated convolution module, and build a conditional diffusion U-shaped denoising model based on the visual state space mixed-scale dilated convolution module; Based on the guide image, the conditional diffusion U-shaped denoising model is used to estimate and denoise the noisy image through iterative calculation to obtain the restored non-uniform haze-degraded image.

2. The method for restoring a non-uniform haze-degraded image according to claim 1, characterized in that: In the step of constructing a U-shaped feature extraction network and using the U-shaped feature extraction network to extract depth feature information of the non-uniform haze-degraded image, the U-shaped feature extraction network is specifically constructed based on a dynamic depth-by-depth convolution group, and the dynamic depth-by-depth convolution group includes several dynamic depth-by-depth convolution blocks, and each dynamic depth-by-depth convolution block includes several residual blocks.

3. The method for restoring a non-uniform haze-degraded image according to claim 1, wherein: The calculation formula of the guidance image is: in, x c represents the guide image, I x represents a non-uniform haze-degraded image, w x represents the weight of the non-uniform haze-degraded image, I p represents the enhancement result of non-uniform haze degradation image, w p represents the weight of the non-uniform haze degradation image enhancement result, C 1×1 (·) represents a convolution operation with a convolution kernel of 1×1. Relu (·) represents the ReLU nonlinear activation function, Sigmoid (·) represents the Sigmoid nonlinear activation function.

4. The method for restoring a non-uniform haze-degraded image according to claim 1, wherein: In the steps of constructing a visual state space mixed-scale dilated convolution module and constructing a conditional diffusion U-type denoising model based on the visual state space mixed-scale dilated convolution module, the visual state space mixed-scale dilated convolution module is specifically constructed based on the visual state space module and mixed-scale adaptive channel attention, the visual state space module is used to extract global features, and the mixed-scale adaptive channel attention is used to extract local features.

5. The method for restoring a non-uniform haze-degraded image according to claim 4, characterized in that: The specific calculation process of the visual state space module is: in, X Emb represents the input of the visual state space module, Linear (·) represents a linear operation, DConv 3×3 (·) represents a depthwise convolution operation with a convolution kernel of 3×3, 2 DSSM (·) represents a 2D selective sweep state space operation, LNorm (·) represents the layer regularization operation, F XEmb Indicates that the visual state space module inputs the feature expression after linear mapping and depth convolution, M XEmb Input the visual state space module into the feature matrix after 2D selective scanning, Fvssm Represents the feature output expression after the visual state space module.

6. The method for restoring a non-uniform haze-degraded image according to claim 4, wherein: The specific calculation process of the hybrid scale adaptive channel attention is: in, DConv n×n (·) indicates the convolution kernel size is n × n The depth convolution operation, represents element-wise addition operation, Indicates channel-by-channel splicing operation. F inca represents the input of mixed-scale adaptive channel attention, F msf Represents the multi-scale feature extraction fusion result, F outmca represents the output of mixed-scale adaptive channel attention, CA (·) represents the channel attention calculation process, C 1×1 (·) represents a convolution operation with a 1×1 kernel.

7. The method for restoring a non-uniform haze-degraded image according to claim 1, wherein: The specific calculation process for restoring the non-uniform haze-degraded image is as follows: in, P θ (·) represents the Markov chain representation in the reverse diffusion process of the conditional diffusion model, express t The noisy image at time t, express t -1 moment after removing noise, x c represents the guide image, σ t represents the time-dependent variance, t Indicates the time stamp during the iteration process, Indicates that the parameter is θ The mean learned from the conditional diffusion U-shaped denoising model; The calculation formula is: in, represents the noise image predicted by the conditional diffusion U-shaped denoising model, Indicates that at time step t The noise variance added to the true value image corresponding to the non-uniform haze degraded image is, Indicates that at time step t The original signal ratio is retained, , express The product of multiple accumulations; It is obtained through the loss function, and the calculation formula of the loss function is: in, L ( θ ) represents the loss function of the conditional diffusion U-shaped denoising model, represents the noise randomly sampled from the standard normal distribution, represents the L2 constraint, Represents a guide image The corresponding reference image, Express expectations.

8. A non-uniform haze-degraded image restoration system, characterized in that: It includes data acquisition module, U-shaped feature extraction network construction module, residual addition module, guided image acquisition module, forward diffusion module, noisy image acquisition module, U-shaped denoising model construction module and degraded image restoration module; The data acquisition module is used to acquire a non-uniform haze-degraded image and a true value image corresponding to the non-uniform haze-degraded image; The U-shaped feature extraction network construction module is used to construct a U-shaped feature extraction network, and use the U-shaped feature extraction network to extract depth feature information of the non-uniform haze-degraded image; The residual addition module is used to perform residual addition on the depth feature information and the non-uniform haze-degraded image to obtain an enhanced result of the non-uniform haze-degraded image; The guide image acquisition module is used to construct a dynamic fusion network module, use the dynamic fusion network module to obtain weights corresponding to the non-uniform haze degraded image and the non-uniform haze degraded image enhancement results, and fuse the non-uniform haze degraded image and the non-uniform haze degraded image enhancement results based on the weights to obtain a guide image; The forward diffusion module is used to forward diffuse the true value image corresponding to the non-uniform haze-degraded image into standard normal distribution noise to obtain a trained noise estimator; The noisy image acquisition module is used to estimate the noise randomly sampled from the standard normal distribution using a trained noise estimator to obtain estimated noise, and remove the estimated noise from the noise randomly sampled from the standard normal distribution to obtain a noisy image; The U-shaped denoising model construction module is used to construct a visual state space mixed-scale dilated convolution module, and to construct a conditional diffusion U-shaped denoising model based on the visual state space mixed-scale dilated convolution module; The degraded image restoration module is used to estimate and denoise the noisy image based on the guide image using the conditional diffusion U-type denoising model through iterative calculation to obtain a restored non-uniform haze-degraded image.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for restoring a non-uniform haze-degraded image according to any one of claims 1 to 7 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for restoring a non-uniform haze-degraded image according to any one of claims 1 to 7 are implemented.

Images