Method and system for restoring multi-category degraded images

Abstract

The application relates to the technical field of computer vision, and particularly discloses a multi-class degenerated image recovery method and system, which constructs and trains a multi-class degenerated image recovery network. The network first decomposes a damaged image into a reflection layer R, a light effect layer G and a shadow layer L, so that the network can perform specific processing on different physical characteristics of the image; then, a hierarchical embedding is added to each layer feature to perceive different characteristics of each layer; a multi-head reverse attention network (CSMTAN) is used to extract key information from different image layers; a gated deconvolution network (GCSADN) is used to further finely process the features; subsequently, a diffusion model is used to simulate the image degradation and recovery process, effectively recovering the original texture and details of the image; and finally, a decoding operation is performed to obtain an output image. The application can simultaneously process multiple image recovery tasks such as low-light enhancement, denoising, rain removal, defogging and deblurring, thereby improving the image quality.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for restoring multi-category degraded images. Background Technology

[0002] Image restoration technology is a key research direction in the field of image processing, aiming to recover clear and usable information from damaged or degraded images. With the widespread use of smartphones and digital cameras, people generate a large amount of image data in their daily lives. These images are often affected by factors such as lighting, motion, and lens limitations, requiring post-processing to improve quality. In professional fields such as medical imaging, satellite image analysis, and historical document restoration, high-quality images are crucial for ensuring the accuracy and reliability of information. Furthermore, in machine learning and computer vision tasks, such as object detection and image classification, high-quality input images can significantly improve the recognition performance and accuracy of algorithms.

[0003] Early image restoration techniques primarily relied on filtering techniques, such as median filtering and Gaussian filtering. While these methods effectively remove noise, they often lose image details. Furthermore, traditional image processing models are often designed for a single task, such as denoising or deblurring only. Such single-task models limit their flexibility and efficiency in practical applications. These models typically lack the ability to handle multiple types of degradation, limiting their practicality. In addition, traditional methods are insufficient in information integration, especially when dealing with complex scenes requiring information extraction from multiple image layers, often failing to effectively restore image details. For example, when processing images with high dynamic range or severe degradation, these methods often struggle to recover true details and colors. Therefore, there is an urgent need for a multifunctional image restoration network capable of simultaneously handling multiple degradation factors to improve image quality. Summary of the Invention

[0004] This invention provides a method and system for restoring images of multiple categories of degradation, and solves the technical problem of how to restore images of multiple degradation categories with high quality.

[0005] To address the above technical problems, this invention provides a method for restoring multi-category degraded images, including:

[0006] Construct a multi-category degraded image restoration network;

[0007] The multi-category degraded image restoration network is trained;

[0008] Multi-class degraded images are input into a multi-class degraded image restoration network that has been trained to restore the images, resulting in restored images.

[0009] The constructed multi-class degraded image restoration network includes an image decomposition module, a hierarchical embedding module, a multi-head transpose attention module, a gated deconvolution module, a diffusion module, and a decoder;

[0010] The image decomposition module is used to decompose the input multi-category degraded image to obtain a reflection layer R, a light effect layer G, and a shadow layer L;

[0011] The hierarchical embedding module is used to hierarchically embed the reflective layer R, the light effect layer G, and the shadow layer L respectively to obtain the corresponding reflective layer R1, light effect layer G1, and shadow layer L1;

[0012] The multi-head transposed attention module is used to extract features from the reflection layer R1, the light effect layer G1 and the shadow layer L1 by applying the channel attention mechanism, the spatial attention mechanism and the multi-head transposed attention mechanism, so as to obtain the corresponding reflection layer R3, light effect layer G3 and shadow layer L3.

[0013] The gated deconvolution module is used to perform deconvolution operations on the reflection layer R3, the light effect layer G3 and the shadow layer L3 to obtain the corresponding reflection layer R5, light effect layer G5 and shadow layer L5;

[0014] The diffusion module is used to fuse the reflective layer R5, the light effect layer G5, and the shadow layer L5 to obtain a comprehensive feature map. Then, the comprehensive feature map Generate initial noisy image Then, the initial noise image Denoising is performed to obtain a low-noise image. ;

[0015] The decoder module is used to process the low-noise image. Decode the image to obtain the recovered image.

[0016] Furthermore, the operation of the multi-head transposed attention module includes the following steps:

[0017] S31. The reflective layer R1, the light effect layer G1, and the shadow layer L1 are each normalized by a normalization layer.

[0018] S32. Generate the Q, K, and V values ​​for the normalized reflection layer R1, light effect layer G1, and shadow layer L1 respectively;

[0019] S33. Multiply the transposes of the Q and K values ​​of each layer (R1, G1, L1) and process them using the Softmax function to generate the transposed attention map of that layer. Then multiply the transposed attention map of that layer with the V value of that layer, perform feature weighting, transpose it, and then perform a 1×1 convolution to generate a weighted feature map.

[0020] S34. The weighted feature maps corresponding to the reflection layer R1, the light effect layer G1, and the shadow layer L1 are enhanced by the first context enhancement module to obtain the corresponding reflection layer R2, the light effect layer G2, and the shadow layer L2.

[0021] S35. Make a skip connection between the reflection layer R2, the light effect layer G2, the shadow layer L2 and their corresponding reflection layer R1, the light effect layer G1, and the shadow layer L1 to obtain the reflection layer R3, the light effect layer G3, and the shadow layer L3.

[0022] Furthermore, the operations performed by the first context enhancement module include the following steps:

[0023] 341. Generate feature map R1 using a channel attention mechanism on the weighted feature map of the reflection layer R1. CA The weighted feature maps G1 and L1 of the light effect layer G1 and the shadow layer L1 are used to generate feature map G1L1. CA Spatial attention mechanism is used to generate feature map L1 from the weighted feature map of the shadow layer L1. SA ;

[0024] S342, Transfer feature map R1 CA and feature map G1L1 CA Perform addition to obtain the reflective layer R2; then perform feature map G1L1. CA and feature map L1 SA Perform addition to obtain the optical effect layer G2; then perform feature map L1. SA Subtract the weighted feature map of the shadow layer L1 to obtain the shadow layer L2.

[0025] Furthermore, the operations performed by the gated deconvolution module include the following steps:

[0026] S41. The reflective layer R3, the light effect layer G3, and the shadow layer L3 are each normalized by a normalization layer.

[0027] S42. Each of the normalized reflection layer R3, light effect layer G3, and shadow layer L3 is processed sequentially with a 1×1 convolutional layer and a 3×3 deconvolutional layer to obtain intermediate feature maps. Each of the normalized reflection layer R3, light effect layer G3, and shadow layer L3 is processed sequentially with a 1×1 convolutional layer, a 3×3 deconvolutional layer, a GELU activation function, and layer normalization to obtain a gating value. The intermediate feature maps and gating values ​​of each of the normalized reflection layer R3, light effect layer G3, and shadow layer L3 are weighted and superimposed, and then subjected to a 1×1 convolution to obtain the enhanced feature map of each layer.

[0028] S43. The enhanced feature maps corresponding to the reflection layer R3, light effect layer G3, and shadow layer L3 are enhanced by the second context enhancement module to obtain the corresponding reflection layer R4, light effect layer G4, and shadow layer L4.

[0029] S44. Perform a skip connection between the enhanced feature maps corresponding to the reflection layer R3, light effect layer G3, and shadow layer L3 and their corresponding reflection layer R4, light effect layer G4, and shadow layer L4 to obtain the reflection layer R5, light effect layer G5, and shadow layer L5.

[0030] Furthermore, the operations performed by the second context enhancement module specifically include the following steps:

[0031] S341. Use the channel attention mechanism to generate feature map R3G3 from the enhanced feature maps corresponding to the reflection layer R3 and the optical effect layer G3. CA The enhanced feature maps G3L3 of the light effect layer G3 and the shadow layer L3 are generated using a channel attention mechanism. CA Spatial attention mechanism is used to generate feature map L3 for the enhanced feature map of the shadow layer L3. SA ;

[0032] S342, Transfer feature map R3G3 CA and feature map G3L3 CA Perform addition to obtain the reflective layer R4; then perform feature map G3L3. CA and feature map L3 SA Perform addition to obtain the optical effect layer G4; then perform feature map L1. SA Subtract the enhanced feature map of shadow layer L3 to obtain shadow layer L4.

[0033] Furthermore, the operations performed by the diffusion module include the following steps:

[0034] S51. Merge the light effect layer G5, the reflection layer R5, and the shadow layer L5 into a comprehensive feature map. ;

[0035] S52, Comprehensive Feature Map Generate initial noisy image ;

[0036] S53. The initial noisy image is processed through noise diffusion and parameter renormalization. Add noise to generate a noisy image. ;

[0037] S54, Using a noise estimation network For noisy images Perform reverse denoising to generate a low-noise image. .

[0038] Furthermore, in step S53, noise diffusion and parameter renormalization are used to generate a noise image at time step t. Among them, time step T represents the total number of diffusion steps; time step Input is obtained through an iterative loop, starting from 0... Finally, in the diffusion phase, new noise is added to the image obtained from the previous diffusion step at each step, and then the parameters are renormalized and the image obtained from the diffusion formula is... By performing weighted merging, the final noisy image is obtained after T time steps. .

[0039] Furthermore, noise diffusion can be expressed by the formula:

[0040] ,

[0041] ,

[0042] in, Indicates at time step Noisy images, Indicates at time step Noisy images, This represents the variance added to the noise at time step t. Indicates a normal distribution. yes The result of scaling;

[0043] Use parameter renormalization to generate noisy images at each time step. Expressed as a formula:

[0044] ,

[0045] in, It is a coefficient that decreases over time, controlling the initial noisy image. exist The weights in It is noise sampled from a standard normal distribution.

[0046] Furthermore, during training, the network is estimated by minimizing noise. The training loss is used to optimize the parameters of the noise estimation network. ;

[0047] Noise estimation network The training loss is expressed as:

[0048] ,

[0049] in, This indicates that context and additional information are being considered. Below, noise estimation network Noisy image at time step t Estimated noise, Represents the norm.

[0050] The present invention also provides a system for restoring multi-category degraded images, the key feature of which is: an intelligent agent is provided to implement the method for restoring multi-category degraded images.

[0051] This invention provides a method and system for restoring multi-class degraded images. It constructs and trains a multi-class degraded image restoration network. This network first decomposes the damaged image into a reflection layer R, a light effect layer G, and a shadow layer L, enabling the network to perform specific processing for different physical properties of the image. Then, hierarchical embedding is added to the features of each layer, allowing the network to perceive the different characteristics of each layer. Next, a cross-channel and spatial multi-head inverse attention network (CSMTAN) is used to extract key information from different image layers. CSMTAN can retain necessary information while removing redundant or interfering information, thereby optimizing the efficiency and effectiveness of subsequent processing steps. Then, a gated deconvolutional network (GCSADN) is used to further refine these features. The gating mechanism controls the flow of information through element-wise multiplication, enabling the network to better learn the relationships between features, enhance image details, and reduce noise. Subsequently, a diffusion model is used to simulate the image degradation and restoration process, effectively restoring the original texture and details of the image. Finally, the image is decoded to obtain the output image.

[0052] This invention brings significant improvements to the field of image restoration by combining the architectures of CSMTAN, GCSADN, and diffusion models. This integrated framework effectively processes and integrates information from different image layers through multi-head attention and gating mechanisms, providing comprehensive restoration strategies. Furthermore, through a multi-task learning strategy, the model can simultaneously handle various image degradation problems, greatly improving practicality and efficiency. The dynamic diffusion model provides a natural and effective way to gradually improve image quality by simulating the image degradation and restoration process. This multi-class degradation image restoration network can simultaneously handle various image restoration tasks such as low-light enhancement, denoising, deraining, dehazing, and deblurring, thereby improving image quality.

[0053] Low-light enhancement: Enhances image brightness and contrast using information from the light effect layer G, with a particular focus on detail recovery in low-light environments;

[0054] Image deraining: Utilizing information from the reflective layer R to identify and remove the effects of raindrops in the image, as the reflective layer R helps the model better understand object surfaces and boundaries;

[0055] Image deblurring: Combining information from the reflection layer R and the light effect layer G to process blur caused by perturbation or focus issues, restoring image sharpness;

[0056] Image denoising: mainly relies on information from the reflection layer R and the shadow layer L to reduce noise in the image, especially noise that is common in dark areas and shadows. Attached Figure Description

[0057] Figure 1 This is a structural diagram of the multi-category degraded image restoration network provided in the embodiments of the present invention;

[0058] Figure 2 This is a structural diagram of the multi-head transposed attention module provided in an embodiment of the present invention;

[0059] Figure 3 This is a structural diagram of the gated deconvolution module provided in an embodiment of the present invention;

[0060] Figure 4 This is a structural diagram of the diffusion module provided in an embodiment of the present invention. Detailed Implementation

[0061] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. The embodiments are given for illustrative purposes only and should not be construed as limiting the present invention. The accompanying drawings are for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0062] This invention provides a method for restoring multi-category degraded images, such as... Figure 1 As shown, it includes:

[0063] Construct a multi-category degraded image restoration network;

[0064] Training a multi-class degraded image restoration network;

[0065] Multi-class degraded images are input into a multi-class degraded image restoration network that has been trained to restore the images, resulting in restored images.

[0066] like Figure 1 As shown, the constructed multi-class degraded image restoration network includes an image decomposition module, a hierarchical embedding module, a multi-head transposed attention module (referred to as the CSMTAN network in this example), a gated deconvolution module (referred to as the GCSADN network in this example), a diffusion module (model), and a decoder.

[0067] The image decomposition module is used to decompose the input multi-class degraded image I (Input) to obtain the reflection layer R, the light effect layer G, and the shadow layer L;

[0068] The hierarchical embedding module is used to hierarchically embed the reflection layer R, the light effect layer G, and the shadow layer L to obtain the corresponding reflection layer R1, light effect layer G1, and shadow layer L1.

[0069] The multi-head transposed attention module is used to extract features from the reflection layer R1, the light effect layer G1, and the shadow layer L1 by applying the channel attention mechanism (CA), the spatial attention mechanism (SA), and the multi-head transposed attention mechanism, so as to obtain the corresponding reflection layer R3, the light effect layer G3, and the shadow layer L3.

[0070] The gated deconvolution module is used to perform deconvolution operations on the reflection layer R3, the light effect layer G3 and the shadow layer L3 to obtain the corresponding reflection layer R5, the light effect layer G5 and the shadow layer L5.

[0071] The diffusion module is used to fuse the reflection layer R5, the light effect layer G5, and the shadow layer L5 to obtain a comprehensive feature map. Then, analyze the comprehensive feature map. Generate initial noisy image Then, the initial noisy image Denoising is performed to obtain a low-noise image. ;

[0072] The decoder module is used for low-noise images Decode the image to obtain the recovered image O (Output).

[0073] (1) Image decomposition module

[0074] Specifically, the multi-class degraded image I (Input) is damaged due to insufficient lighting, noise interference, motion blur, or environmental factors such as raindrops. The image decomposition module decomposes the multi-class degraded image I (Input) into a reflection layer R, a light effect layer G, and a shadow layer L. This allows each layer to focus on processing specific features, enabling more precise image enhancement and restoration. Each layer can be optimized using different network structures. The decomposition process of the image decomposition module is expressed by the following formula:

[0075]

[0076] in, This represents the Hadama product.

[0077] The inherent color and texture information of multi-class degraded images I (Input) are extracted through multiple feature extraction modules (each including a convolutional layer and a residual module), preserving the image details and structure to obtain the reflection layer R. The residual module enhances the network's feature extraction capability.

[0078] A convolutional network is used to extract highlight features from multi-class degraded images I (Input), capturing light effects such as light spots and reflections, resulting in a light effect layer (G). The convolutional layer extracts local features through filters. Different network depths extract features at different levels: shallow layers capture basic edges and textures, while deeper layers capture high-level abstract features.

[0079] The shadow and illumination distributions of a multi-class degraded image I (Input) are extracted using a convolutional network with skip connections, reflecting the distribution of light sources and shadows in the image, resulting in a shadow layer (L). Skip connections allow information to be passed directly from shallow to deep layers, helping to recover high-resolution features and details.

[0080] (2) Hierarchical Embedded Module

[0081] The hierarchical embedding module adds hierarchical embedding to the decomposed reflection layer (R), light effect layer (G), and shadow layer (L) to obtain the reflection layer (R1), light effect layer (G1), and shadow layer (L1), enabling the network to perceive the different characteristics of each layer.

[0082] Specifically, the hierarchical embedding module includes a reflection layer embedding unit, a light effect layer embedding unit, and a shadow layer embedding unit. The reflection layer embedding unit extracts the reflection layer information of the image, focusing on the image structure and object boundaries. The light effect layer embedding unit extracts the light effect layer information of the image, mainly focusing on illumination and brightness. The shadow layer embedding unit extracts the shadow layer information of the image, used to analyze dark areas and shadows in the image.

[0083] (3) Multi-head transposed attention module

[0084] Figure 2 The diagram shows the structure of the multi-head transposed attention module, which reflects the characteristics of cross-channel attention (CA), spatial attention (SA), and multi-head transposed attention mechanisms. The multi-head transposed attention mechanism generates a transposed attention map by calculating the dot product between the query, key, and value of the reflection layer R1, the light effect layer G1, and the shadow layer L1, capturing long-range dependencies in the feature map. Through this mechanism, CSMTAN can enhance useful information in the input feature map while suppressing noise and irrelevant information.

[0085] After processing with the multi-head transposed attention mechanism, the light effect layer G1 and the reflection layer R1 are subjected to channel attention (CA) by adding the initial feature map and the enhanced feature map to generate the enhanced light effect layer G2 and the reflection layer R2. The shadow layer L2 is subjected to spatial attention (SA) by subtracting the initial feature map and the enhanced feature map to reduce interference from unnecessary features and improve the accuracy and efficiency of feature extraction. Finally, a skip connection is made with each corresponding original layer in the input CSMTAN network, and then the output layers are light effect layer G3, reflection layer R3, and shadow layer L3.

[0086] More specifically, such as Figure 2 As shown, the operation of the multi-head transposed attention module includes the following steps:

[0087] S31. The reflection layer R1, the light effect layer G1, and the shadow layer L1 are each normalized by a normalization layer (Norm) to improve the stability and convergence speed of model training.

[0088] S32. Generate queries (Q), keys (K), and values ​​(V) for the normalized reflection layer R1, light effect layer G1, and shadow layer L1. The Q and K values ​​are used to generate attention maps, compressing dimensions to reduce computational complexity; the V value is used to generate weighted features, preserving the original feature depth. The method for generating any query (Q), key (K), or value (V) is as follows: the input image is transformed by passing it through a 1×1 convolutional layer (Conv) and two 3×3 convolutional layers (Dconv).

[0089] S33. Multiply the transposes of the Q and K values ​​corresponding to each layer (R1, G1, L1) and process them using the Softmax function to generate the transposed attention map (A) of that layer. This step generates a square matrix to describe the relationship and importance between each channel. Then, multiply the transposed attention map (A) of that layer with the V value of that layer and perform feature weighting. Reweight V according to the attention distribution, then transpose it and perform a 1×1 convolution to generate a weighted feature map.

[0090] S34. The weighted feature maps corresponding to the reflection layer R1, the light effect layer G1, and the shadow layer L1 are enhanced by the first context enhancement module to obtain the corresponding reflection layer R2, the light effect layer G2, and the shadow layer L2.

[0091] S35. Make a skip connection between the reflection layer R2, the light effect layer G2, the shadow layer L2 and their corresponding reflection layer R1, the light effect layer G1, and the shadow layer L1 to obtain the reflection layer R3, the light effect layer G3, and the shadow layer L3.

[0092] In step S34, the operations performed by the first context enhancement module include the following steps:

[0093] 341. Use the channel attention mechanism (CA) on the weighted feature map of the reflection layer R1 to generate feature map R1. CA Channel attention (CA) is used to generate feature map G1L1 from the weighted feature maps of the light effect layer G1 and the shadow layer L1. CA Spatial attention (SA) is used to generate feature map L1 by applying spatial attention (SA) to the weighted feature map of the shadow layer L1. SA ;

[0094] S342, Transfer feature map R1 CA and feature map G1L1 CA Perform addition to obtain the reflective layer R2; then perform feature map G1L1. CA and feature map L1 SA Perform addition to obtain the optical effect layer G2; then perform feature map L1. SA The weighted feature map of the shadow layer L1 is subtracted to obtain the shadow layer L2, which reduces the interference of unnecessary features and improves the accuracy and efficiency of feature extraction.

[0095] (4) Gated deconvolution module

[0096] Figure 3The diagram shows the structure of the gated deconvolution module. The light effect layer G3, reflection layer R3, and shadow layer L3, processed by the CSMTAN network, are input into the GCSADN network to obtain the corresponding light effect layer G5, reflection layer R5, and shadow layer L5. The GCSADN network reflects the characteristics of cross-channel attention (CA), spatial attention (SA), gating mechanisms, and deconvolution. The GCSADN network performs feature map transformation and enhancement through a series of convolution operations and gating mechanisms. The gating mechanism controls the flow of information through element-wise multiplication, enabling the network to better learn the relationships between features. The channel attention mechanism (CA) is applied to the gated and deconvolution-processed light effect layer G3 and reflection layer R3, combining the initial and enhanced feature maps through addition to generate the enhanced light effect layer G4 and reflection layer R4. The spatial attention mechanism (SA) is applied to the shadow layer L3, combining the initial and enhanced feature maps through subtraction to reduce interference from unnecessary features and improve the accuracy and efficiency of feature extraction. Finally, a skip connection is made with each corresponding original layer of the input GCSADN network, and then the output layers are light effect layer G5, reflection layer R5, and shadow layer L5.

[0097] refer to Figure 3 The operations performed by the gated deconvolution module include the following steps:

[0098] S41. The reflection layer R3, the light effect layer G3, and the shadow layer L3 are each normalized by a normalization layer (Norm) to improve the stability and convergence speed of model training.

[0099] S42. Each of the normalized reflection layer R3, light effect layer G3, and shadow layer L3 is processed sequentially with a 1×1 convolutional layer (conv) and a 3×3 deconvolutional layer (Dconv) to obtain intermediate feature maps. Each of the normalized reflection layer R3, light effect layer G3, and shadow layer L3 is processed sequentially with a 1×1 convolutional layer (conv), a 3×3 deconvolutional layer (Dconv), a GELU activation function, and layer normalization to obtain gating values. The intermediate feature maps and gating values ​​of each of the normalized reflection layer R3, light effect layer G3, and shadow layer L3 are weighted and superimposed, and then subjected to 1×1 convolution to obtain the enhanced feature map of each layer.

[0100] S43. The enhanced feature maps corresponding to the reflection layer R3, light effect layer G3, and shadow layer L3 are further refined through the second context enhancement module to obtain the corresponding reflection layer R4, light effect layer G4, and shadow layer L4.

[0101] S44. Perform a skip connection between the enhanced feature maps corresponding to the reflection layer R3, light effect layer G3, and shadow layer L3 and their corresponding reflection layer R4, light effect layer G4, and shadow layer L4 to obtain the reflection layer R5, light effect layer G5, and shadow layer L5.

[0102] like Figure 3 As shown, the operations performed by the second context enhancement module are the same as those of the first context enhancement module, specifically including the following steps:

[0103] S341. Apply channel attention (CA) to the enhanced feature maps corresponding to the reflection layer R3 and the optical effect layer G3 to generate feature map R3G3. CA Channel attention (CA) is used to enhance the feature maps of the light effect layer G3 and the shadow layer L3 to generate feature map G3L3. CA Spatial attention (SA) is used to enhance the feature map of the shadow layer L3 to generate feature map L3. SA ;

[0104] S342, Transfer feature map R3G3 CA and feature map G3L3 CA Perform addition to obtain the reflective layer R4; then perform feature map G3L3. CA and feature map L3 SA Perform addition to obtain the optical effect layer G4; then perform feature map L1. SA Subtracting the enhanced feature map of shadow layer L3 from the shadow layer L4 yields shadow layer L4, reducing interference from unnecessary features and improving the accuracy and efficiency of feature extraction.

[0105] (5) Diffusion module / model

[0106] Figure 4 This is a structural diagram of the diffusion module. (Example) Figure 4 As shown, the three layers—the light effect layer G5, the reflection layer R5, and the shadow layer L5—processed by the GCSADN network are added to the diffusion model. First, the feature maps of the three layers are fused into a comprehensive feature map. ,use For comprehensive feature maps Generate initial noisy image Then, forward diffusion is performed, noise is added for gradient descent optimization, and then backward diffusion is performed, starting with Gaussian noise, and gradually recovering the original low-light feature representation through multiple sampling steps. During training, the model parameters are further optimized using the mean squared error loss function.

[0107] More specifically, the operations performed by the diffusion module include the following steps:

[0108] S51. Merge the light effect layer G5, the reflection layer R5, and the shadow layer L5 into a comprehensive feature map. ;

[0109] S52, Comprehensive Feature Map Generate initial noisy image ;

[0110] S53. The initial noisy image is processed through noise diffusion and parameter renormalization. Add noise to generate a noisy image. ;

[0111] S54, Using a noise estimation network For noisy images Perform reverse denoising to generate a low-noise image. .

[0112] In step S51, the feature map fusion is expressed by the formula:

[0113] ,

[0114] concat means concatenating along the channel dimension, that is, stitching the feature maps of the three layers together to form a composite feature map. , integrate feature map The input is fed into a noise generation network to generate an initial noisy image. .

[0115] In step S53, the diffusion formula and parameter renormalization step are used to generate the noise image for each time step. Among them, time step Time step (T represents the total number of diffusion steps), used to represent each step in the diffusion process. Time step Used to control the increase of image noise, wherein the noise is distributed according to a predetermined variance scheme. Gradually increase. Time step Input is obtained through an iterative loop, starting from 0... End. In the diffusion phase, a new noise of a specific level is added to the image obtained from the previous diffusion step at each step.

[0116] The diffusion formula is expressed as:

[0117] ,

[0118] in, Indicates at time step Noisy images, Indicates at time step Noisy images, It is a predefined variance sequence that controls the noise level. Indicates a normal distribution. yes The scaling result is that this scaling keeps the variance of the data within control, preventing the data from becoming too blurry during continuous diffusion. Indicates at time step The variance added to the noise helps simulate real-world uncertainty and image loss. This is achieved by using images from the previous time step. Scale and add to it Gaussian noise with variance is used to generate the current time step. Image .

[0119] Parameter renormalization can be expressed by the formula:

[0120] ,

[0121] in, It is a coefficient that decreases over time, controlling the original data. exist The weights in the equation. It is noise sampled from a standard normal distribution. Indicates the original image The degree of preservation, This represents the added noise, the amount of which increases with time to simulate the gradual degradation of the data. This formula is used to generate the simulation data for each time step. By combining the raw data A scaled version and a suitable amount of noise are used to simulate the gradual degradation process of the data.

[0122] Then, the image obtained by parameter renormalization and diffusion formula is... By merging elements with equal weights, the final image is obtained after T time steps. .

[0123] In the reverse denoising step S54, the sampling step size needs to be determined. Sampling step size This refers to the number of time steps skipped in each iteration of the reverse diffusion (reverse denoising) process. It is used to determine the amount of noise to be reduced in each step of the reverse diffusion process. Sampling step size. Through It was calculated, among which The formula is:

[0124] ,

[0125] in It is the current iteration index. This indicates Control each iteration The update magnitude is the noise reduction step size each time.

[0126] Inverse denoising can be expressed by the following formula:

[0127]

[0128] in, and These are the mean and variance, defined as follows:

[0129] ,

[0130] ,

[0131] and They represent time. and Low-noise images, This represents the context and additional information (determined by the diffusion model itself). Noise estimation network at time t For the current low-noise image Estimated noise. The inverse denoising process is used to recover cleaner data from a noisy image by modifying the current low-noise image. Adjustments and corrections were made based on the noise estimation network. Estimated noise to predict earlier steps in low-noise images .

[0132] In this embodiment, the noise estimation network A convolutional neural network is used. During the training phase: the parameters of the noise estimation network are optimized. To minimize the noise estimation network Training loss:

[0133]

[0134] in, Represents a noise estimation network For the current noisy image Estimated noise, Represents the norm.

[0135] In addition to the above methods, the present invention also provides a system for restoring multi-category degraded images, which includes an intelligent agent for implementing the above-mentioned method for restoring multi-category degraded images.

[0136] In summary, the multi-class degraded image restoration method and system provided in this invention constructs and trains a multi-class degraded image restoration network. This network first decomposes the damaged image into a reflection layer R, a light effect layer G, and a shadow layer L, enabling the network to perform specific processing for different physical properties of the image. Then, hierarchical embedding is added to the features of each layer, allowing the network to perceive the different characteristics of each layer. Next, a cross-channel and spatial multi-head inverse attention network (CSMTAN) is used to extract key information from different image layers. CSMTAN can retain necessary information while removing redundant or interfering information, thereby optimizing the efficiency and effectiveness of subsequent processing steps. Then, a gated deconvolutional network (GCSADN) is used to further refine these features. The gating mechanism controls the flow of information through element-wise multiplication, enabling the network to better learn the relationships between features, enhance image details, and reduce noise. Subsequently, a diffusion model is used to simulate the image degradation and restoration process, effectively restoring the original texture and details of the image. Finally, the image is decoded to obtain the output image.

[0137] This invention combines the architectures of CSMTAN, GCSADN, and a diffusion model, bringing significant improvements to the field of image restoration. This integrated framework effectively processes and integrates information from different image layers through multi-head attention and gating mechanisms, providing comprehensive restoration strategies. Furthermore, through a multi-task learning strategy, the model can simultaneously handle various image degradation problems, greatly improving practicality and efficiency. The dynamic diffusion model provides a natural and effective way to gradually improve image quality by simulating the image degradation and restoration process. This network can simultaneously handle multiple image restoration tasks such as low-light enhancement, denoising, deraining, dehazing, and deblurring, thereby improving image quality.

[0138] Low-light enhancement: Enhances image brightness and contrast using information from the light effect layer G, with a particular focus on detail recovery in low-light environments;

[0139] Image deraining: Utilizing information from the reflective layer R to identify and remove the effects of raindrops in the image, as the reflective layer R helps the model better understand object surfaces and boundaries;

[0140] Image deblurring: Combining information from the reflection layer R and the light effect layer G to process blur caused by perturbation or focus issues, restoring image sharpness;

[0141] Image denoising: mainly relies on information from the reflection layer R and the shadow layer L to reduce noise in the image, especially noise that is common in dark areas and shadows.

[0142] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for restoring multi-class degraded images, characterized in that, include: Construct a multi-category degraded image restoration network; The multi-category degraded image restoration network is trained; Multi-class degraded images are input into a multi-class degraded image restoration network that has been trained to restore the images, resulting in restored images. The constructed multi-class degraded image restoration network includes an image decomposition module, a hierarchical embedding module, a multi-head transpose attention module, a gated deconvolution module, a diffusion module, and a decoder; The image decomposition module is used to decompose the input multi-category degraded image to obtain a reflection layer R, a light effect layer G, and a shadow layer L; The hierarchical embedding module is used to hierarchically embed the reflective layer R, the light effect layer G, and the shadow layer L respectively to obtain the corresponding reflective layer R1, light effect layer G1, and shadow layer L1; The multi-head transposed attention module is used to extract features from the reflection layer R1, the light effect layer G1 and the shadow layer L1 by applying the channel attention mechanism, the spatial attention mechanism and the multi-head transposed attention mechanism, so as to obtain the corresponding reflection layer R3, light effect layer G3 and shadow layer L3. The gated deconvolution module is used to perform deconvolution operations on the reflection layer R3, the light effect layer G3 and the shadow layer L3 to obtain the corresponding reflection layer R5, light effect layer G5 and shadow layer L5; The diffusion module is used to fuse the reflective layer R5, the light effect layer G5, and the shadow layer L5 to obtain a comprehensive feature map. Then, the comprehensive feature map Generate initial noisy image Then, the initial noise image Denoising is performed to obtain a low-noise image. ; The decoder module is used to process the low-noise image. Decode the image to obtain the recovered image; The operation of the multi-head transposed attention module includes the following steps: S31. The reflective layer R1, the light effect layer G1, and the shadow layer L1 are each normalized by a normalization layer. S32. Generate the Q, K, and V values ​​for the normalized reflection layer R1, light effect layer G1, and shadow layer L1 respectively; S33. Multiply the transposes of the Q and K values ​​of each layer (R1, G1, L1) by the softmax function to generate the transposed attention map of that layer. Then multiply the transposed attention map of that layer by the V value of that layer, perform feature weighting, transpose it, and then perform a 1×1 convolution to generate a weighted feature map. S34. The weighted feature maps corresponding to the reflection layer R1, the light effect layer G1, and the shadow layer L1 are enhanced by the first context enhancement module to obtain the corresponding reflection layer R2, the light effect layer G2, and the shadow layer L2. S35. Make a skip connection between the reflection layer R2, the light effect layer G2, the shadow layer L2 and their corresponding reflection layer R1, the light effect layer G1, and the shadow layer L1 to obtain the reflection layer R3, the light effect layer G3, and the shadow layer L3.

2. The method for restoring multi-category degraded images according to claim 1, characterized in that, The operations performed by the first context enhancement module include the following steps: S341. Use a channel attention mechanism to generate feature map R1 from the weighted feature map of the reflection layer R1. CA The weighted feature maps G1 and L1 of the light effect layer G1 and the shadow layer L1 are used to generate feature map G1L1. CA Spatial attention mechanism is used to generate feature map L1 from the weighted feature map of the shadow layer L1. SA ; S342, Transfer feature map R1 CA and feature map G1L1 CA Perform addition to obtain the reflective layer R2; then perform feature map G1L1. CA and feature map L1 SA Perform addition to obtain the optical effect layer G2; then perform feature map L1. SA Subtract the weighted feature map of the shadow layer L1 to obtain the shadow layer L2.

3. The method for restoring multi-category degraded images according to claim 2, characterized in that, The operations performed by the gated deconvolution module include the following steps: S41. The reflective layer R3, the light effect layer G3, and the shadow layer L3 are each normalized by a normalization layer. S42. Each of the normalized reflection layer R3, light effect layer G3, and shadow layer L3 is processed sequentially with a 1×1 convolutional layer and a 3×3 deconvolutional layer to obtain intermediate feature maps. Each of the normalized reflection layer R3, light effect layer G3, and shadow layer L3 is processed sequentially with a 1×1 convolutional layer, a 3×3 deconvolutional layer, a GELU activation function, and layer normalization to obtain a gating value. The intermediate feature maps and gating values ​​of each of the normalized reflection layer R3, light effect layer G3, and shadow layer L3 are weighted and superimposed, and then subjected to a 1×1 convolution to obtain the enhanced feature map of each layer. S43. The enhanced feature maps corresponding to the reflection layer R3, light effect layer G3, and shadow layer L3 are enhanced by the second context enhancement module to obtain the corresponding reflection layer R4, light effect layer G4, and shadow layer L4. S44. Perform a skip connection between the enhanced feature maps corresponding to the reflection layer R3, light effect layer G3, and shadow layer L3 and their corresponding reflection layer R4, light effect layer G4, and shadow layer L4 to obtain the reflection layer R5, light effect layer G5, and shadow layer L5.

4. The method for restoring multi-category degraded images according to claim 3, characterized in that, The operations performed by the second context enhancement module specifically include the following steps: S341. Use the channel attention mechanism to generate feature map R3G3 from the enhanced feature maps corresponding to the reflection layer R3 and the optical effect layer G3. CA The enhanced feature maps G3L3 of the light effect layer G3 and the shadow layer L3 are generated using a channel attention mechanism. CA Spatial attention mechanism is used to generate feature map L3 for the enhanced feature map of the shadow layer L3. SA ; S342, Transfer feature map R3G3 CA and feature map G3L3 CA Perform addition to obtain the reflective layer R4; then perform feature map G3L3. CA and feature map L3 SA Perform addition to obtain the optical effect layer G4; then perform feature map L1. SA Subtract the enhanced feature map of shadow layer L3 to obtain shadow layer L4.

5. The method for restoring multi-category degraded images according to claim 4, characterized in that, The operations performed by the diffusion module include the following steps: S51. Merge the light effect layer G5, the reflection layer R5, and the shadow layer L5 into a comprehensive feature map. ; S52, Comprehensive Feature Map Generate initial noisy image ; S53. The initial noisy image is processed through noise diffusion and parameter renormalization. Add noise to generate a noisy image. ; S54, Using a noise estimation network For noisy images Perform reverse denoising to generate a low-noise image. .

6. The method for restoring multi-category degraded images according to claim 5, characterized in that, In step S53, noise diffusion and parameter renormalization are used to generate a noise image at time step t. Among them, time step T represents the total number of diffusion steps; time step Input is obtained through an iterative loop, starting from 0... Finally, in the diffusion phase, new noise is added to the image obtained from the previous diffusion step at each step, and then the parameters are renormalized and the image obtained from the diffusion formula is... By performing weighted merging, the final noisy image is obtained after T time steps. .

7. The method for restoring multi-category degraded images according to claim 6, characterized in that, Noise diffusion can be expressed by the formula: ， in, Indicates at time step Noisy images, Indicates at time step Noisy images, This represents the variance added to the noise at time step t. Indicates a normal distribution. yes The result of scaling; Use parameter renormalization to generate noisy images at each time step. Expressed as a formula: ， in, It is a coefficient that decreases over time, controlling the initial noisy image. exist The weights in It is noise sampled from a standard normal distribution.

8. The method for restoring multi-category degraded images according to claim 7, characterized in that, During training, the network is estimated by minimizing noise. The training loss is used to optimize the parameters of the noise estimation network. ; Noise estimation network The training loss is expressed as: ， in, This indicates that context and additional information are being considered. Below, noise estimation network Noisy image at time step t Estimated noise, Represents the norm.

9. A system for restoring multi-class degraded images, characterized in that: An intelligent agent is provided for implementing the method for restoring multi-category degraded images as described in any one of claims 1 to 8.

Images