Method and system for recovering multi-class degraded images

Abstract

The invention relates to the technical field of computer vision, and particularly discloses a multi-class degraded image recovery method and system, and the method comprises the steps: constructing and training a multi-class degraded image recovery network, and enabling the network to decompose a damaged image into a reflection layer R, a light effect layer G and a shadow layer L; the network can perform specific processing according to different physical characteristics of the image; hierarchy embedding is added to each layer of features so as to sense different features of each layer; extracting key information from different image layers by using a multi-head reverse attention network (CSMTAN); the features are further processed in a refined mode through a gated deconvolution network (GCSADN); then, simulating an image degradation and recovery process through a diffusion model, and effectively recovering the original texture and details of the image; and finally carrying out decoding operation to obtain an output image. According to the method, various image recovery tasks such as low-light enhancement, denoising, rain removal, defogging and deblurring can be processed at the same time, so that the image quality is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for restoring multiple categories of degraded images. Background Art

[0002] Image restoration technology is a key research direction in the field of image processing, which aims to restore clear and usable information from damaged or degraded images. With the popularity of smartphones and digital cameras, a large amount of image data is generated in people's daily lives. These images are often affected by factors such as lighting, motion, and lens limitations, and require post-processing to improve quality. In professional fields such as medical imaging, satellite image analysis, and historical document restoration, high-quality images are essential to ensure the accuracy and reliability of information. In addition, in machine learning and computer vision tasks such as object detection and image classification, high-quality input images can significantly improve the recognition effect and accuracy of the algorithm.

[0003] Early image restoration techniques mainly rely on filtering techniques, such as median filtering and Gaussian filtering. These methods often lose image details while removing noise. In addition, traditional image processing models are often designed for a single task, such as only denoising or only deblurring. The flexibility and efficiency of such single-task models in practical applications are limited. These models usually lack the ability to handle multiple types of degradation, which limits their practicality. In addition, traditional methods are also deficient in information integration, especially when dealing with complex scenes that require information to be extracted from multiple image layers, and often fail to effectively restore image details. For example, when dealing with images with high dynamic range or severe degradation, these methods often have difficulty restoring true details and colors. Therefore, there is an urgent need for a multifunctional image restoration network that can simultaneously handle multiple degradation factors and improve image quality. Summary of the invention

[0004] The present invention provides a method and system for restoring multiple categories of degraded images, and solves the technical problem of how to restore images of multiple categories of degraded images with high quality.

[0005] In order to solve the above technical problems, the present invention provides a method for restoring multiple types of degraded images, comprising:

[0006] Construct a multi-category degraded image restoration network;

[0007] Training the multi-category degraded image restoration network;

[0008] The multi-category degraded images are input into the trained multi-category degraded image restoration network for restoration to obtain the restored images;

[0009] The constructed multi-category degraded image restoration network includes an image decomposition module, a hierarchical embedding module, a multi-head transposition attention module, a gated deconvolution module, a diffusion module, and a decoder;

[0010] The image decomposition module is used to decompose the input multi-category degraded image to obtain a reflection layer R, a light effect layer G and a shadow layer L;

[0011] The hierarchical embedding module is used to hierarchically embed the reflection layer R, the light effect layer G and the shadow layer L respectively to obtain the corresponding reflection layer R1, light effect layer G1 and shadow layer L1;

[0012] The multi-head transposition attention module is used to apply the channel attention mechanism, the spatial attention mechanism and the multi-head transposition attention mechanism to extract features of the reflection layer R1, the light effect layer G1 and the shadow layer L1 to obtain the corresponding reflection layer R3, the light effect layer G3 and the shadow layer L3;

[0013] The gated deconvolution module is used to perform a deconvolution operation on the reflection layer R3, the light effect layer G3 and the shadow layer L3 to obtain a corresponding reflection layer R5, a light effect layer G5 and a shadow layer L5;

[0014] The diffusion module is used to fuse the reflection layer R5, the light effect layer G5 and the shadow layer L5 to obtain a comprehensive feature map F combined , and then the comprehensive feature map F combined Generate an initial noise image x0, and then denoise the initial noise image x0 to obtain a low-noise image

[0015] The decoder module is used to decode the low noise image Decode and get the restored image.

[0016] Furthermore, the operation of the multi-head transposed attention module includes the steps of:

[0017] S31, normalizing the reflection layer R1, the light effect layer G1, and the shadow layer L1 through a normalization layer respectively;

[0018] S32, generating respective Q, K, and V values ​​for the normalized reflection layer R1, light effect layer G1, and shadow layer L1;

[0019] S33, multiply the transposed Q value and K value corresponding to each layer of the reflection layer R1, the light effect layer G1, and the shadow layer L1, and process them through the Softmax function to generate the transposed attention map of the layer; then multiply the transposed attention map corresponding to the layer with the V value of the layer and perform feature weighting, and then transpose and perform 1×1 convolution to generate a weighted feature map;

[0020] S34, performing context enhancement on the weighted feature maps corresponding to the reflection layer R1, the light effect layer G1, and the shadow layer L1 through a first context enhancement module to obtain the corresponding reflection layer R2, the light effect layer G2, and the shadow layer L2;

[0021] S35, make a jump connection between the reflection layer R2, the light effect layer G2, the shadow layer L2 and their corresponding reflection layer R1, the light effect layer G1, the shadow layer L1 to obtain the reflection layer R3, the light effect layer G3, the shadow layer L3.

[0022] Furthermore, the operation performed by the first context enhancement module includes the steps of:

[0023] 341. Use the channel attention mechanism to generate feature map R1 for the weighted feature map of the reflection layer R1 CA ; Use the channel attention mechanism to generate the feature map G1L1 for the weighted feature map of the light effect layer G1 and the shadow layer L1 CA ; Use the spatial attention mechanism to generate feature map L1 for the weighted feature map of the shadow layer L1 SA ;

[0024] S342, feature map R1 CA And feature map G1L1 CA Perform addition operation to obtain the reflection layer R2; transform the feature map G1L1 CA And feature map L1 SA Perform addition operation to obtain the light effect layer G2; add the feature map L1 SA Subtract the weighted feature map of the shadow layer L1 to obtain the shadow layer L2.

[0025] Furthermore, the operation performed by the gated deconvolution module includes the steps of:

[0026] S41, normalizing the reflection layer R3, the light effect layer G3, and the shadow layer L3 through a standardization layer respectively;

[0027] S42, process each of the normalized reflection layer R3, light effect layer G3, and shadow layer L3 with a 1×1 convolution layer and a 3×3 deconvolution layer in turn to obtain an intermediate feature map; process each of the normalized reflection layer R3, light effect layer G3, and shadow layer L3 with a 1×1 convolution layer, a 3×3 deconvolution layer, a GELU activation function, and layer normalization in turn to obtain a gating value; perform 1×1 convolution on the intermediate feature map and the gating value of each layer in the normalized reflection layer R3, light effect layer G3, and shadow layer L3 after weighting and superposition, and obtain an enhanced feature map of each layer;

[0028] S43, performing context enhancement on the enhancement feature maps corresponding to the reflection layer R3, the light effect layer G3, and the shadow layer L3 through a second context enhancement module to obtain the corresponding reflection layer R4, the light effect layer G4, and the shadow layer L4;

[0029] S44, make a jump connection between the enhanced feature maps corresponding to the reflection layer R3, light effect layer G3, and shadow layer L3 and their corresponding reflection layer R4, light effect layer G4, and shadow layer L4 to obtain the reflection layer R5, light effect layer G5, and shadow layer L5.

[0030] Furthermore, the operation performed by the second context enhancement module specifically includes the steps of:

[0031] S341, use the channel attention mechanism to generate feature map R3G3 for the enhanced feature map corresponding to the reflection layer R3 and the light effect layer G3 CA ; Use the channel attention mechanism to generate feature map G3L3 for the enhanced feature map of the light effect layer G3 and the shadow layer L3 CA ; Use the spatial attention mechanism to generate feature map L3 for the enhanced feature map of the shadow layer L3 SA ;

[0032] S342, feature map R3G3 CA And feature map G3L3 CA Perform addition operation to obtain the reflection layer R4; convert the feature map G3L3 CA And feature map L3 SA Perform addition operation to obtain the light effect layer G4; add the feature map L1 SA Subtract the enhanced feature map of the shadow layer L3 to obtain the shadow layer L4.

[0033] Furthermore, the operation performed by the diffusion module includes the steps of:

[0034] S51, merge the light effect layer G5, the reflection layer R5 and the shadow layer L5 into a comprehensive feature map F combined ;

[0035] S52, comprehensive feature map F combined Generate an initial noise image x0;

[0036] S53, adding noise to the initial noise image x0 through noise diffusion and parameter renormalization to generate a noisy image x T ;

[0037] S54, using the noise estimation network ∈ θ For a noisy image x T Perform reverse denoising to generate a low-noise image

[0038] Furthermore, in step S53, noise diffusion and parameter renormalization are used to generate a noise image x at time step t. t , where time step t∈[0,T], T represents the total number of diffusion steps; time step t is input through an iterative loop, starting from 0 and ending at T. In the diffusion stage, each step adds new noise to the image obtained by the previous step diffusion, and then the image x obtained by parameter renormalization and diffusion formula is t Perform equal weight merging, and after T time steps, the final noisy image x is obtained T .

[0039] Furthermore, noise diffusion is expressed by the formula:

[0040]

[0041]

[0042] Among them, x t represents the noisy image at time step t, x t-1 represents the noisy image at time step t-1, β t represents the variance added to the noise at time step t, represents a normal distribution, is x t-1 The result of scaling;

[0043] Use parameter renormalization to generate a noisy image x at each time step t The formula is:

[0044]

[0045] Among them, α t is a coefficient that decreases with time, controlling the initial noise image x0 at x t The weights in ∈ t is the noise sampled from a standard normal distribution.

[0046] Furthermore, during training, the noise estimation network ∈ θ The training loss is used to optimize the parameters θ of the noise estimation network;

[0047] Noise estimation network ∈ θ The training loss is expressed as:

[0048]

[0049] in, Indicates that context and additional information are taken into account Next, the noise estimation network ∈ θ For the noisy image x at time step tt The estimated noise, ‖‖ represents the norm.

[0050] The present invention also provides a restoration system for multi-category degraded images, the key of which is that an intelligent agent is provided to implement the restoration method for multi-category degraded images.

[0051] The invention provides a method and system for restoring a multi-category degraded image. The method and system construct and train a multi-category degraded image restoration network. The multi-category degraded image restoration network first decomposes the damaged image into a reflection layer R, a light effect layer G and a shadow layer L, so that the network can perform specific processing on different physical properties of the image. Then, hierarchical embedding is added to each layer of features, so that the network can perceive different properties of each layer. Then, a cross-channel and spatial multi-head reverse attention network (CSMTAN) is used to extract key information from different image layers. The CSMTAN can remove redundant or interfering information while retaining necessary information, thereby optimizing the efficiency and effect of subsequent processing steps. Then, a gated deconvolutional network (GCSADN) is used to further refine these features. The gating mechanism controls the flow of information through element-level multiplication, so that the network can better learn the relationship between features, enhance image details and reduce noise. Subsequently, a diffusion model is used to simulate the image degradation and restoration process, so as to effectively restore the original texture and details of the image. Finally, a decoder is entered for decoding operation to obtain an output image.

[0052] The present invention combines the architecture of CSMTAN, GCSADN and diffusion model to bring significant improvements to the field of image restoration. This integrated technical framework effectively processes and integrates information from different image layers through multi-head attention and gating mechanisms, providing a comprehensive restoration strategy. In addition, through the strategy of multi-task learning, the model can handle multiple image degradation problems at the same time, greatly improving practicality and efficiency. The dynamic diffusion model provides a natural and effective way to gradually improve image quality by simulating the image degradation and restoration process. The multi-category degraded image restoration network can simultaneously handle multiple image restoration tasks such as low-light enhancement, denoising, rain removal, defogging and deblurring, thereby improving image quality.

[0053] Low-light enhancement: Enhances image brightness and contrast through the information of the light effect layer G, with special attention to detail recovery in low-light environments;

[0054] Image deraining: Use the information of the reflection layer R to identify and remove the influence of raindrops in the image, because the reflection layer R can help the model better understand the surface and boundaries of the object;

[0055] Image deblurring: Combine the information of the reflection layer R and the light effect layer G to process the blur caused by perturbation or focus problems and restore the clarity of the image;

[0056] Image denoising: It mainly relies on the information of the reflection layer R and the shadow layer L to reduce the noise in the image, especially the noise commonly found in dark areas and shadows. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a structural diagram of a multi-category degraded image restoration network provided by an embodiment of the present invention;

[0058] Figure 2 is a structural diagram of a multi-head transposed attention module provided by an embodiment of the present invention;

[0059] Figure 3 is a structural diagram of a gated deconvolution module provided in an embodiment of the present invention;

[0060] Figure 4 It is a structural diagram of a diffusion module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following specifically illustrates the implementation mode of the present invention in conjunction with the accompanying drawings. The embodiments are provided for illustrative purposes only and cannot be understood as limiting the present invention. The accompanying drawings are provided for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0062] The embodiment of the present invention provides a method for restoring multiple types of degraded images, such as Figure 1 As shown, including:

[0063] Construct a multi-category degraded image restoration network;

[0064] Train a multi-class degraded image restoration network;

[0065] The multi-category degraded images are input into the trained multi-category degraded image restoration network for restoration to obtain restored images.

[0066] like Figure 1 As shown in the figure, the constructed multi-category degraded image restoration network includes an image decomposition module, a hierarchical embedding module, a multi-head transposition attention module (referred to as a CSMTAN network in this example), a gated deconvolution module (referred to as a GCSADN network in this example), a diffusion module (model), and a decoder.

[0067] The image decomposition module is used to decompose the input multi-category degraded image I (Input) to obtain the reflection layer R, the light effect layer G and the shadow layer L;

[0068] The hierarchical embedding module is used to hierarchically embed the reflection layer R, the light effect layer G and the shadow layer L respectively to obtain the corresponding reflection layer R1, the light effect layer G1 and the shadow layer L1;

[0069] The multi-head transposition attention module is used to apply the channel attention mechanism (CA), the spatial attention mechanism (SA) and the multi-head transposition attention mechanism to extract features of the reflection layer R1, the light effect layer G1 and the shadow layer L1, and obtain the corresponding reflection layer R3, the light effect layer G3 and the shadow layer L3;

[0070] The gated deconvolution module is used to perform a deconvolution operation on the reflection layer R3, the light effect layer G3 and the shadow layer L3 to obtain the corresponding reflection layer R5, the light effect layer G5 and the shadow layer L5;

[0071] The diffusion module is used to fuse the reflection layer R5, the light effect layer G5 and the shadow layer L5 to obtain a comprehensive feature map F combined , and then the comprehensive feature map F combined Generate an initial noisy image x0, and then denoise the initial noisy image x0 to obtain a low-noise image

[0072] The decoder module is used to decode low-noise images Decode and obtain the restored image O (Output).

[0073] (1) Image decomposition module

[0074] Specifically, the multi-category degraded image I (Input) is damaged due to insufficient lighting, noise interference, motion blur or environmental factors such as raindrops. The image decomposition module decomposes the multi-category degraded image I (Input) to obtain the reflection layer R, the light effect layer G and the shadow layer L, so that each layer can focus on processing specific features, so that image enhancement and restoration can be performed more accurately, and each layer can be optimized through different network structures. The decomposition process of the image decomposition module is expressed as follows:

[0075] I=R⊙L+G

[0076] Wherein, ⊙ represents the Hadamard product.

[0077] Through multiple feature extraction modules (each feature extraction module includes a convolutional layer and a residual module), the inherent color and texture information of the multi-category degraded image I (Input) is extracted, the details and structure of the image are retained, and the reflection layer R is obtained. Through the residual module, the feature extraction capability of the network can be enhanced.

[0078] The highlight features of the multi-category degraded image I (Input) are extracted through the convolutional network to capture light effects such as light spots and reflections, and the light effect layer (G) is obtained. The convolution layer extracts local features through filters. Different depths of the network extract features at different levels. The shallow layer captures basic edges and textures, and the deep layer captures high-level abstract features.

[0079] The shadow and illumination distribution of the multi-category degraded image I (Input) is extracted through a convolutional network with skip-layer connections, reflecting the distribution of light sources and shadows in the image, and a shadow layer (L) is obtained. Skip-layer connections allow information to be directly transferred from shallow layers to deep layers, helping to restore high-resolution features and details.

[0080] (2) Hierarchical Embedding Module

[0081] The hierarchical embedding module adds hierarchical embedding to the decomposed reflection layer (R), light effect layer (G) and shadow layer (L) to obtain the reflection layer (R1), light effect layer (G1) and shadow layer (L1), so that the network can perceive the different characteristics of each layer.

[0082] Specifically, the hierarchical embedding module includes a reflection layer embedding unit, a light effect layer embedding unit, and a shadow layer embedding unit. The reflection layer embedding unit extracts the reflection layer information of the image, focusing on the structure of the image and the object boundary. The light effect layer embedding unit extracts the light effect layer information of the image, mainly focusing on illumination and brightness. The shadow layer embedding unit extracts the shadow layer information of the image, which is used to analyze dark areas and shadows in the image.

[0083] (3) Multi-head transposition attention module

[0084] Figure 2 The figure shows the structure of the multi-head transposed attention module, which reflects the characteristics of the cross-channel attention mechanism (CA), spatial attention mechanism (SA) and multi-head transposed attention mechanism. The multi-head transposed attention mechanism generates a transposed attention map by calculating the dot product between the query (Query), key (Key) and value (Value) of the reflection layer R1, the light effect layer G1 and the shadow layer L1, and captures the long-range dependencies in the feature map. Through the multi-head transposed attention mechanism, CSMTAN can enhance the useful information in the input feature map and suppress noise and irrelevant information.

[0085] The channel attention mechanism (CA) is used for the light effect layer G1 and the reflection layer R1 processed by the multi-head transposed attention mechanism, and the preliminary feature map and the enhanced feature map are combined for addition operation to generate the enhanced light effect layer G2 and the reflection layer R2. The spatial attention mechanism (SA) is used for the shadow layer L2, and the preliminary feature map and the enhanced feature map are combined for subtraction operation to reduce the interference of unnecessary features and improve the accuracy and efficiency of feature extraction. Finally, a jump connection is made with each corresponding original layer of the input CSMTAN network, and then the output is performed. The output layers are the light effect layer G3, the reflection layer R3, and the shadow layer L3.

[0086] More specifically, Figure 2 As shown, the operation of the multi-head transposed attention module includes the following steps:

[0087] S31, normalizing the reflection layer R1, the light effect layer G1, and the shadow layer L1 through a normalization layer (Norm) to improve the stability and convergence speed of model training;

[0088] S32. Generate the query (Q), key (K), and value (V) for the normalized reflection layer R1, light effect layer G1, and shadow layer L1. The Q and K values ​​are used to generate the attention map, compressing the dimension to reduce the computational complexity; the V value is used to generate weighted features and maintain the original feature depth. The method for generating any query (Q), key (K), and value (V) is to transform the input image through a 1×1 convolution layer (Conv) and two 3×3 (Dconv).

[0089] S33, transpose the Q value and K value corresponding to each layer of the reflection layer R1, the light effect layer G1, and the shadow layer L1 After multiplication, the Softmax function is used to generate the transposed attention map (A) of the layer. This step generates a square matrix to describe the relationship and importance between channels. Then the transposed attention map (A) corresponding to the layer is multiplied by the V value of the layer and the feature is weighted. V is reweighted according to the attention distribution, and then transposed and 1×1 convolution is performed to generate a weighted feature map.

[0090] S34, performing context enhancement on the weighted feature maps corresponding to the reflection layer R1, the light effect layer G1, and the shadow layer L1 through a first context enhancement module to obtain the corresponding reflection layer R2, the light effect layer G2, and the shadow layer L2;

[0091] S35, make a jump connection between the reflection layer R2, the light effect layer G2, the shadow layer L2 and their corresponding reflection layer R1, the light effect layer G1, the shadow layer L1 to obtain the reflection layer R3, the light effect layer G3, the shadow layer L3.

[0092] In step S34, the operation performed by the first context enhancement module includes the steps of:

[0093] 341. Use the channel attention mechanism (CA) on the weighted feature map of the reflection layer R1 to generate feature map R1 CA ; Use the channel attention mechanism (CA) on the weighted feature maps of the light effect layer G1 and the shadow layer L1 to generate the feature map G1L1 CA ; Use the spatial attention mechanism (SA) on the weighted feature map of the shadow layer L1 to generate the feature map L1 SA ;

[0094] S342, feature map R1 CA And feature map G1L1 CA Perform addition operation to obtain the reflection layer R2; transform the feature map G1L1 CAAnd feature map L1 SA Perform addition operation to obtain the light effect layer G2; add the feature map L1 SA The shadow layer L2 is obtained by subtracting the weighted feature map of the shadow layer L1, thereby reducing the interference of unnecessary features and improving the accuracy and efficiency of feature extraction.

[0095] (4) Gated Deconvolution Module

[0096] Figure 3 The figure shows the structure of the gated deconvolution module. The light effect layer G3, reflection layer R3, and shadow layer L3 processed by the CSMTAN network are input into the GCSADN network to obtain the corresponding light effect layer G5, reflection layer R5, and shadow layer L5. The GCSADN network reflects the characteristics of the cross-channel attention mechanism (CA), spatial attention mechanism (SA), gating mechanism, and deconvolution. The GCSADN network performs feature map conversion and enhancement through a series of convolution operations and gating mechanisms. The gating mechanism controls the flow of information through element-level multiplication, enabling the network to better learn the relationship between features. The channel attention mechanism (CA) is used for the light effect layer G3 and the reflection layer R3 processed by the gating mechanism and deconvolution, and the preliminary feature map and the enhanced feature map are combined for addition operation to generate the enhanced light effect layer G4 and the reflection layer R4. The spatial attention mechanism (SA) is used for the shadow layer L3, and the preliminary feature map and the enhanced feature map are combined for subtraction operation to reduce the interference of unnecessary features and improve the accuracy and efficiency of feature extraction. Finally, a jump connection is made with each corresponding original layer of the input GCSADN network, and then the output is performed. The output layers are light effect layer G5, reflection layer R5, and shadow layer L5.

[0097] refer to Figure 3 , the operation performed by the gated deconvolution module includes the steps:

[0098] S41, normalizing the reflection layer R3, the light effect layer G3, and the shadow layer L3 through a normalization layer (Norm) to improve the stability and convergence speed of model training;

[0099] S42, process each of the normalized reflection layer R3, light effect layer G3, and shadow layer L3 with a 1×1 convolution layer (conv) and a 3×3 deconvolution layer (Dconv) to obtain an intermediate feature map; process each of the normalized reflection layer R3, light effect layer G3, and shadow layer L3 with a 1×1 convolution layer (conv), a 3×3 deconvolution layer (Dconv), a GELU activation function, and layer normalization to obtain a gating value; weight and superimpose the intermediate feature map and the gating value of each layer in the normalized reflection layer R3, light effect layer G3, and shadow layer L3, and then perform a 1×1 convolution to obtain an enhanced feature map of each layer;

[0100] S43, further refining the features of the enhanced feature maps corresponding to the reflection layer R3, the light effect layer G3, and the shadow layer L3 through the second context enhancement module to obtain the corresponding reflection layer R4, the light effect layer G4, and the shadow layer L4;

[0101] S44, make a jump connection between the enhanced feature maps corresponding to the reflection layer R3, light effect layer G3, and shadow layer L3 and their corresponding reflection layer R4, light effect layer G4, and shadow layer L4 to obtain the reflection layer R5, light effect layer G5, and shadow layer L5.

[0102] like Figure 3 As shown, the operation performed by the second context enhancement module is consistent with that of the first context enhancement module, and specifically includes the steps of:

[0103] S341. Use the channel attention mechanism (CA) for the enhanced feature maps corresponding to the reflection layer R3 and the light effect layer G3 to generate the feature map R3G3. CA ; Use the channel attention mechanism (CA) for the enhanced feature maps of the light effect layer G3 and the shadow layer L3 to generate the feature map G3L3 CA ; Use the spatial attention mechanism (SA) on the enhanced feature map of the shadow layer L3 to generate feature map L3 SA ;

[0104] S342, feature map R3G3 CA And feature map G3L3 CA Perform addition operation to obtain the reflection layer R4; convert the feature map G3L3 CA And feature map L3 SA Perform addition operation to obtain the light effect layer G4; add the feature map L1 SA The shadow layer L4 is obtained by subtracting the enhanced feature map of the shadow layer L3, thereby reducing the interference of unnecessary features and improving the accuracy and efficiency of feature extraction.

[0105] (5) Diffusion module / model

[0106] Figure 4 is the structural diagram of the diffusion module. Figure 4 As shown in the figure, the light effect layer G5, reflection layer R5, and shadow layer L5 processed by the GCSADN network are added to the diffusion model, and the feature maps of the three layers are first fused into a comprehensive feature map F combined , using q(x t |x t-1 ) for the comprehensive feature map F combinedGenerate an initial noise image x0, then perform forward diffusion, add noise for gradient descent optimization, and then perform reverse diffusion, starting from Gaussian noise, and gradually restore to the original low-light feature representation through multiple sampling steps. During the training process, the model parameters are further optimized through the mean square error loss function.

[0107] More specifically, the operations performed by the diffusion module include the following steps:

[0108] S51, merge the light effect layer G5, the reflection layer R5 and the shadow layer L5 into a comprehensive feature map F combined ;

[0109] S52, comprehensive feature map F combined Generate an initial noise image x0;

[0110] S53, adding noise to the initial noise image x0 through noise diffusion and parameter renormalization to generate a noisy image x T ;

[0111] S54, using the noise estimation network ∈ θ For a noisy image x T Perform reverse denoising to generate a low-noise image

[0112] In step S51, the fusion of feature maps is expressed as:

[0113] F combined =concat(L5,R5,G5),

[0114] concat means connecting in the channel dimension, that is, concatenating the feature maps of the three layers together to form a comprehensive feature map F combined , the comprehensive feature map F combined Input into the noise generation network to generate the initial noise image x0.

[0115] In step S53, the diffusion formula and the parameter renormalization step are used to generate the noise image x at each time step. t , where time step t: time step t∈[0,T] (T represents the total number of diffusion steps), is used to represent each step in the diffusion process. Time step t is used to control the increase of image noise, where the noise is calculated according to a predetermined variance plan β t The time step t is input through an iterative loop, starting from 0 and ending at T. In the diffusion stage, each step adds a new noise of a specific level to the image obtained by the previous diffusion step.

[0116] The diffusion formula is expressed as:

[0117]

[0118] Among them, x t represents the noisy image at time step t, x t-1 represents the noisy image at time step t-1, β t is a predefined variance sequence that controls the noise level, represents a normal distribution, is x t-1 The result of scaling, which keeps the variance of the data under control and prevents the data from becoming too blurred during continuous diffusion, β t represents the variance added to the noise at time step t, helping to model real-world uncertainty and image loss. t-1 Scale and add to it a value of β t is a Gaussian noise with variance , to generate the image x at the current time step t t .

[0119] The parameter renormalization formula is expressed as:

[0120]

[0121] Among them, α t is a coefficient that decreases with time, controlling the original data x0 at x t The weight in ∈ t is the noise sampled from a standard normal distribution. Indicates the degree of preservation of the original image x0, represents the noise added, whose amount increases with time to simulate the gradual degradation of the data. This formula is used to generate the simulated data x at each time step t , by combining the original data x t A scaled version of and an appropriate amount of noise are used to simulate the gradual degradation of the data.

[0122] Then the image x obtained by parameter renormalization and diffusion formula is t Merge with equal weights, and after T time steps, get the final image x T .

[0123] In the reverse denoising of step S54, it is necessary to determine the sampling step length S, which refers to the number of time steps skipped in each iteration of the reverse diffusion (reverse denoising) process. It is used to determine the amount of noise that should be reduced in each step of the reverse diffusion process. The sampling step length S is calculated by t next Calculated, where t next The formula is:

[0124] t next =(i-2)·T / S+1,

[0125] Where i is the current iteration index. next This shows that S controls the update amplitude at each iteration t, i.e., the noise step size reduced each time.

[0126] The reverse denoising formula is expressed as:

[0127]

[0128] in, and are the mean and variance respectively, defined as:

[0129]

[0130]

[0131] and Represent the low-noise images at time t and t-1 respectively, Represents context and additional information (determined by the diffusion model itself). Denotes the noise estimation network ∈ at time t θ For the current low-noise image The inverse denoising process is used to recover cleaner data from a noisy image by Adjust and correct based on the noise estimation network ∈ θ The estimated noise is used to predict the low-noise image at an earlier step

[0132] In this embodiment, the noise estimation network ∈ θ A convolutional neural network is used. In the training phase: optimize the parameters θ of the noise estimation network to minimize the noise estimation network ∈ θ The training loss is:

[0133]

[0134] in, Denotes the noise estimation network ∈ θ For the current noisy image x t The estimated noise, ‖‖ represents the norm.

[0135] In view of the above method, the present invention further provides a multi-category degraded image restoration system, which is provided with an intelligent agent for implementing the multi-category degraded image restoration method.

[0136] In summary, the multi-category degraded image restoration method and system provided by the embodiment of the present invention construct and train a multi-category degraded image restoration network, which first decomposes the damaged image into a reflection layer R, a light effect layer G and a shadow layer L, so that the network can perform specific processing for different physical properties of the image; then add hierarchical embedding to each layer of features, so that the network can perceive the different characteristics of each layer; then use a cross-channel and spatial multi-head reverse attention network (CSMTAN) to extract key information from different image layers. CSMTAN can remove redundant or interfering information while retaining necessary information, thereby optimizing the efficiency and effect of subsequent processing steps; then use a gated deconvolution network (GCSADN) to further refine these features, and the gating mechanism controls the flow of information through element-level multiplication, so that the network can better learn the relationship between features, enhance image details and reduce noise; then, the image degradation and restoration process is simulated by a diffusion model, so as to effectively restore the original texture and details of the image; finally, enter the decoder for decoding operation to obtain the output image.

[0137] The present invention combines the architecture of CSMTAN, GCSADN and diffusion model to bring significant improvements to the field of image restoration. This integrated technical framework effectively processes and integrates information from different image layers through multi-head attention and gating mechanisms, providing a comprehensive restoration strategy. In addition, through the strategy of multi-task learning, the model can handle multiple image degradation problems at the same time, greatly improving practicality and efficiency. The dynamic diffusion model provides a natural and effective way to gradually improve image quality by simulating the image degradation and restoration process. The network can simultaneously handle multiple image restoration tasks such as low-light enhancement, denoising, rain removal, defogging and deblurring, thereby improving image quality.

[0138] Low-light enhancement: Enhances image brightness and contrast through the information of the light effect layer G, with special attention to detail recovery in low-light environments;

[0139] Image deraining: Use the information of the reflection layer R to identify and remove the influence of raindrops in the image, because the reflection layer R can help the model better understand the surface and boundaries of the object;

[0140] Image deblurring: Combine the information of the reflection layer R and the light effect layer G to process the blur caused by perturbation or focus problems and restore the clarity of the image;

[0141] Image denoising: It mainly relies on the information of the reflection layer R and the shadow layer L to reduce the noise in the image, especially the noise commonly found in dark areas and shadows.

[0142] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for restoring multi-category degraded images, characterized in that: include: Construct a multi-category degraded image restoration network; Training the multi-category degraded image restoration network; The multi-category degraded images are input into the trained multi-category degraded image restoration network for restoration to obtain the restored images; The constructed multi-category degraded image restoration network includes an image decomposition module, a hierarchical embedding module, a multi-head transposition attention module, a gated deconvolution module, a diffusion module, and a decoder; The image decomposition module is used to decompose the input multi-category degraded image to obtain a reflection layer R, a light effect layer G and a shadow layer L; The hierarchical embedding module is used to hierarchically embed the reflection layer R, the light effect layer G and the shadow layer L respectively to obtain the corresponding reflection layer R1, light effect layer G1 and shadow layer L1; The multi-head transposition attention module is used to apply the channel attention mechanism, the spatial attention mechanism and the multi-head transposition attention mechanism to extract features of the reflection layer R1, the light effect layer G1 and the shadow layer L1 to obtain the corresponding reflection layer R3, the light effect layer G3 and the shadow layer L3; The gated deconvolution module is used to perform a deconvolution operation on the reflection layer R3, the light effect layer G3 and the shadow layer L3 to obtain a corresponding reflection layer R5, a light effect layer G5 and a shadow layer L5; The diffusion module is used to fuse the reflection layer R5, the light effect layer G5 and the shadow layer L5 to obtain a comprehensive feature map F combined , and then the comprehensive feature map F combined Generate an initial noise image x0, and then denoise the initial noise image x0 to obtain a low-noise image The decoder module is used to decode the low noise image Decode and get the restored image.

2. The method for restoring multi-category degraded images according to claim 1, characterized in that: The operation of the multi-head transposed attention module includes the steps of: S31, normalizing the reflection layer R1, the light effect layer G1, and the shadow layer L1 through a normalization layer respectively; S32, generating respective Q, K, and V values ​​for the normalized reflection layer R1, light effect layer G1, and shadow layer L1; S33, multiply the transposed Q value and K value corresponding to each layer of the reflection layer R1, the light effect layer G1, and the shadow layer L1, and process them through the Softmax function to generate the transposed attention map of the layer; then multiply the transposed attention map corresponding to the layer with the V value of the layer and perform feature weighting, and then transpose and perform 1×1 convolution to generate a weighted feature map; S34, performing context enhancement on the weighted feature maps corresponding to the reflection layer R1, the light effect layer G1, and the shadow layer L1 through a first context enhancement module to obtain the corresponding reflection layer R2, the light effect layer G2, and the shadow layer L2; S35, make a jump connection between the reflection layer R2, the light effect layer G2, the shadow layer L2 and their corresponding reflection layer R1, the light effect layer G1, the shadow layer L1 to obtain the reflection layer R3, the light effect layer G3, the shadow layer L3.

3. The method for restoring multi-category degraded images according to claim 2, characterized in that: The operation performed by the first context enhancement module comprises the steps of: S341, use the channel attention mechanism to generate feature map R1 for the weighted feature map of the reflection layer R1 CA ; Use the channel attention mechanism to generate the feature map G1L1 for the weighted feature map of the light effect layer G1 and the shadow layer L1 CA ; Use the spatial attention mechanism to generate feature map L1 for the weighted feature map of the shadow layer L1 SA ; S342, feature map R1 CA And feature map G1L1 CA Perform addition operation to obtain the reflection layer R2; transform the feature map G1L1 CA And feature map L1 SA Perform addition operation to obtain the light effect layer G2; add the feature map L1 SA Subtract the weighted feature map of the shadow layer L1 to obtain the shadow layer L2.

4. The method for restoring multi-category degraded images according to claim 3, characterized in that: The operation performed by the gated deconvolution module includes the steps of: S41, normalizing the reflection layer R3, the light effect layer G3, and the shadow layer L3 through a standardization layer respectively; S42, process each of the normalized reflection layer R3, light effect layer G3, and shadow layer L3 with a 1×1 convolution layer and a 3×3 deconvolution layer in turn to obtain an intermediate feature map; process each of the normalized reflection layer R3, light effect layer G3, and shadow layer L3 with a 1×1 convolution layer, a 3×3 deconvolution layer, a GELU activation function, and layer normalization in turn to obtain a gating value; perform 1×1 convolution on the intermediate feature map and the gating value of each layer in the normalized reflection layer R3, light effect layer G3, and shadow layer L3 after weighting and superposition, and obtain an enhanced feature map of each layer; S43, performing context enhancement on the enhancement feature maps corresponding to the reflection layer R3, the light effect layer G3, and the shadow layer L3 through a second context enhancement module to obtain the corresponding reflection layer R4, the light effect layer G4, and the shadow layer L4; S44, make a jump connection between the enhanced feature maps corresponding to the reflection layer R3, light effect layer G3, and shadow layer L3 and their corresponding reflection layer R4, light effect layer G4, and shadow layer L4 to obtain the reflection layer R5, light effect layer G5, and shadow layer L5.

5. The method for restoring multi-category degraded images according to claim 4, characterized in that: The operation performed by the second context enhancement module specifically includes the following steps: S341, use the channel attention mechanism to generate feature map R3G3 for the enhanced feature map corresponding to the reflection layer R3 and the light effect layer G3 CA ; Use the channel attention mechanism to generate feature map G3L3 for the enhanced feature map of the light effect layer G3 and the shadow layer L3 CA ; Use the spatial attention mechanism to generate feature map L3 for the enhanced feature map of the shadow layer L3 SA ; S342, feature map R3G3 CA And feature map G3L3 CA Perform addition operation to obtain the reflection layer R4; convert the feature map G3L3 CA And feature map L3 SA Perform addition operation to obtain the light effect layer G4; add the feature map L1 SA Subtract the enhanced feature map of the shadow layer L3 to obtain the shadow layer L4.

6. The method for restoring multi-category degraded images according to claim 5, characterized in that: The operation performed by the diffusion module includes the steps of: S51, merge the light effect layer G5, the reflection layer R5 and the shadow layer L5 into a comprehensive feature map F combined ; S52, comprehensive feature map F combined Generate an initial noise image x0; S53, adding noise to the initial noise image x0 through noise diffusion and parameter renormalization to generate a noisy image x T ; S54, using the noise estimation network ∈ θ For a noisy image x T Perform reverse denoising to generate a low-noise image 7. The multi-category degraded image restoration system according to claim 6, characterized in that: In step S53, noise diffusion and parameter renormalization are used to generate a noise image x at time step t. t , where time step t∈[0,T], T represents the total number of diffusion steps; time step t is input through an iterative loop, starting from 0 and ending at T. In the diffusion stage, each step adds new noise to the image obtained by the previous diffusion step, and then renormalizes the parameters and obtains the image x obtained by the diffusion formula t Perform equal weight merging, and after T time steps, the final noisy image x is obtained T .

8. The multi-category degraded image restoration system according to claim 7, characterized in that: Noise diffusion is expressed by the formula: Among them, x t represents the noisy image at time step t, x t-1 represents the noisy image at time step t-1, β t represents the variance added to the noise at time step t, represents a normal distribution, is x t-1 The result of scaling; Use parameter renormalization to generate a noisy image x at each time step t The formula is: Among them, α t is a coefficient that decreases with time, controlling the initial noise image x0 at x t The weights in ∈ t is the noise sampled from a standard normal distribution.

9. The multi-category degraded image restoration system according to claim 8, characterized in that: During training, the noise estimation network ∈ θ The training loss is used to optimize the parameters θ of the noise estimation network; Noise estimation network ∈ θ The training loss is expressed as: in, Indicates that context and additional information are taken into account Next, the noise estimation network ∈ θ For the noisy image x at time step t t The estimated noise, ‖‖ represents the norm.

10. A multi-category degraded image restoration system, characterized by: An intelligent agent is provided for implementing the method for restoring multi-category degraded images as described in any one of claims 1 to 9.

Images