Integrated Image Restoration Method and System

Abstract

This invention provides an integrated image restoration method and system, belonging to the field of image processing. The method includes constructing a text-guided integrated image restoration model (TGMIR), which maps text prompts to a semantic space consistent with image features and achieves hierarchical semantic control at three levels: channel attention, spatial attention, and cross-modal collaborative attention. The image is then restored using the TGMIR. This invention aims to overcome the core limitations of existing integrated image restoration models under multi-degradation conditions, including insufficient degradation semantics, significant cross-degradation interference, weak modal collaboration capabilities, and insufficient feature fusion.

Description

Technical Field

[0001] This invention belongs to the field of image processing, and in particular relates to an integrated image restoration method and system. Background Technology

[0002] Image inpainting is a key technology in computer vision, aiming to restore high-quality, clear images from images with quality degradation. Common types of degradation include noise, rain streaks, haze, motion blur, and low light. Current technological approaches mainly focus on two directions: single degradation restoration and integrated restoration.

[0003] Single-type degradation image inpainting methods train specialized models for specific degradation types. For example, Uformer or Restormer models based on the Transformer architecture perform well in independent tasks such as denoising and deraining. However, these methods are limited by their lack of generality. A single model cannot handle a wide variety of degradation scenarios. In practical deployment, it is necessary to pre-determine the degradation type and select the appropriate model, which is a cumbersome and impractical process.

[0004] Integrated image inpainting methods aim to handle multiple types of degradation with a single model, but this faces a core challenge: the features of different degradation types differ significantly in distribution, and their properties may conflict. For example, forceful denoising may erase key high-frequency edge information used for deblurring, while aggressive deblurring may amplify residual noise. This mutual interference between features is called "degradation interference," which severely limits the performance of integrated models. Existing techniques attempt to alleviate this problem, such as by introducing contrastive learning to obtain more discriminative degradation representations; and by using learnable visual cues to modulate network features to adapt them to a shared parameter space. Recent frameworks such as PromptIR have begun to explore cue learning mechanisms, but their cues are mostly implicit or simple category labels, failing to fully utilize rich semantic information.

[0005] Overall, existing integrated restoration techniques still primarily rely on networks to implicitly infer degradation information from image pixels. This increases the learning difficulty of the model and limits its generalization ability and restoration accuracy in complex, mixed degradation scenarios. The model lacks a clear and in-depth semantic understanding of degradation types, resulting in an inaccurate and unadaptive restoration process.

[0006] Current integrated image restoration methods still face key bottlenecks when dealing with multiple types of degradation. Different degradations differ significantly in distribution and structure, and shared models struggle to effectively distinguish them, often leading to feature conflicts, artifact persistence, and texture loss. Furthermore, existing methods lack stable semantic constraint mechanisms, relying heavily on implicit inferences of degradation types by the model, making them susceptible to task differences and noise, and failing to achieve reliable degradation identification. While existing cue vectors can supplement semantic information, their expressive power is limited and cannot support precise control in complex degradation scenarios. Meanwhile, the current interaction between textual cues and visual features remains superficial, lacking deep fusion at the channel, spatial, and cross-modal levels, making it difficult to form truly semantically driven solutions. Summary of the Invention

[0007] To address the aforementioned shortcomings in existing technologies, this invention provides an integrated image restoration method and system, aiming to overcome the core limitations of existing integrated image restoration models under multi-degradation conditions, including insufficient degradation semantics, significant cross-degradation interference, weak modal collaboration capabilities, and insufficient feature fusion.

[0008] To achieve the above objectives, the technical solution adopted by this invention is: an integrated image restoration method, comprising the following steps:

[0009] S1. Construct a text-guided integrated image restoration model TGMIR. The integrated image restoration model TGMIR is used to map text prompts to a semantic space with consistent image features. It utilizes the triple interactive attention enhancement module TAEB to achieve hierarchical semantic control at three levels: channel attention, spatial attention, and cross-modal collaborative attention. It also utilizes the text-guided two-stage attention fusion mechanism TG-DSAF to fuse the original features and degraded image features processed by the three attention mechanisms.

[0010] S2. Use the integrated image restoration model TGMIR to restore the image.

[0011] Furthermore, the integrated image restoration model TGMIR includes:

[0012] Input layer, used to obtain degraded images ;

[0013] Convolutional layers for degraded image-based processing By capturing low-level texture and edge information, primary features are extracted. ;

[0014] Encoder-decoder framework for use based on primary features High-level semantic feature representations are generated through layer-by-layer transformation. Multiple Transformer blocks are used at each stage of the codec, with the number of Transformer blocks gradually increasing from the top to the bottom. A Text-Guided Feature Enhancement Fusion (TG-FEF) module connects the encoder and decoder at each stage; among them,

[0015] During the encoding phase, based on primary features By downsampling the data while continuously expanding the channel capacity, a latent representation can be obtained. The text guides the visual features; in the decoding stage, the latent features are upsampled and concatenated with the degraded enhanced features processed by the text-guided feature enhancement fusion module TG-FEF in the encoding stage to generate a high-level semantic feature representation. ;

[0016] The output layer is used to represent high-level semantic features. After optimization, a residual image is generated through a second convolutional layer and compared with the degraded image. The image reconstructed from the residuals obtained by addition This completes the integrated image restoration.

[0017] Furthermore, the text-guided feature enhancement fusion module TG-FEF includes a triple interactive attention enhancement module TAEB and a two-stage attention fusion mechanism for text guidance TG-DSAF;

[0018] The Triple Interactive Attention Enhancement Module (TAEB) is used to obtain degenerate enhancement features containing degenerate information through a channel-space-cross-modal triple attention mechanism with semantic similarity feedback.

[0019] The text-guided two-stage attention fusion module TG-DSAF consists of a local-global dual attention module LGDA. The local-global dual attention module LGDA obtains feature weights by calculating global attention and local attention separately. weight The text-guided two-stage attention fusion module TG-DSAF uses two local-global dual attention modules LGDA to process the original features. and enhanced features A two-stage fusion process is performed to ultimately obtain the fusion characteristics. ,in, Indicates original features, This indicates that the degradation enhancement features containing degradation information are obtained through a channel-space-cross-modal triple attention mechanism.

[0020] Furthermore, obtaining the degradation enhancement feature containing degradation information includes the following steps:

[0021] To obtain semantic information, at the channel level, the text encoding t is projected onto the image channel space. Generate channel-wise semantic weights, where the text encoding t is a description of the text's degradation type;

[0022] semantic weights are combined with input image features Multiply each channel sequentially to obtain the channel characteristics. ;

[0023] Channel characteristics Dimensionality reduction is performed, and spatial attention-enhanced features are generated using lightweight deep convolutional methods. ;

[0024] Based on spatial attention enhancement features Generate a Value V and a Key K, based on text encoding. To generate the query value Q;

[0025] Multiply the key value K and the query value Q to obtain the attention weight A. Then, weight the attention weight A onto the value V and restore it to the reduced channel dimension. and the original spatial dimension;

[0026] Based on the recovery results, through Convolution recovers the channel dimension C, yielding cross-modal guided attention features. ;

[0027] Fusion channel features Spatial attention enhancement features and cross-modal guided attention features This yields degraded enhancement features that are text-fused and contain degradation information. .

[0028] Furthermore, the degradation enhancement feature The expression is as follows:

[0029] ;

[0030] in, This means restoring the dimension-reduced difference terms to dimension C and then adding them together. This represents degenerate enhancement features that have undergone text fusion and contain degenerate information. All of these represent learnable coefficients.

[0031] Furthermore, the weights of the degraded enhancement features are obtained. weight This includes the following steps:

[0032] Based on input features X, semantic information Perform dimensional projection to obtain the local injection vectors respectively. and global injection vector ;

[0033] In the local attention branch, pointwise convolution is chosen as the context aggregator;

[0034] Aggregate contexts using a context aggregator;

[0035] Based on the aggregation results, local features are calculated through the bottleneck structure. :

[0036] ;

[0037] in, Indicates original features, and Both represent pointwise convolution. This represents the GELU activation function. Represents the local injection vector;

[0038] In the global attention branch, global average pooling is applied to obtain a global vector, which is then processed by one-dimensional convolution and the sigmoid activation function is applied to calculate the query and key values.

[0039] A global attention graph is generated by calculating the outer product of the query value and the key value. :

[0040] ;

[0041] in, and Let K and Q represent the dimensionally adjusted forms of K and Q, used for outer product calculation, respectively, and T denotes transpose. Indicates the transpose of K2;

[0042] global attention map The process is applied to the original feature channel dimension and global injected text is added to obtain global features. :

[0043] ;

[0044] Here, reshape indicates rearranging the dimensions of the input features to form the Value feature. This represents the attention weighting used to calculate the value, where A represents the attention weight. Represents the global injection vector. It represents the Hadamardi (or Hadama) stack;

[0045] The output is achieved by concatenating global features along the channel dimension. and local features The fusion weights are obtained after passing through the Sigmoid activation function. :

[0046] .

[0047] Furthermore, fusion features The expression is as follows:

[0048] ;

[0049] ;

[0050] Where M represents the Local-Global Dual Attention Module (LGDA), Indicates original features, Z represents the degenerate enhancement feature, and Z represents the initial integrated feature. It represents the Hadamardi (or Hadama) stack.

[0051] The present invention also provides an integrated image restoration system, comprising:

[0052] The first processing module is used to construct a text-guided integrated image restoration model, TGMIR. The integrated image restoration model TGMIR is used to map text prompts to a semantic space with consistent image features. It uses the triple interactive attention enhancement module TAEB to achieve hierarchical semantic control at three levels: channel attention, spatial attention, and cross-modal collaborative attention. It also uses the text-guided two-stage attention fusion mechanism TG-DSAF to fuse the original features and degraded image features processed by the three attention processes.

[0053] The second processing module is used to restore the image using the integrated image restoration model TGMIR.

[0054] The beneficial effects of this invention are:

[0055] This invention proposes a text-guided integrated image restoration framework. By explicitly introducing degradation semantic cues, it effectively reduces cross-degradation interference and enhances the model's ability to identify degradation types. This invention innovatively constructs a three-tiered interactive attention enhancement mechanism for text guidance, enabling fine-tuning of text cues across channel, spatial, and cross-modal dimensions, thereby forming a hierarchical, semantically consistent degradation guidance structure. Furthermore, this invention proposes a two-stage attention fusion mechanism for text guidance, achieving adaptive selection and progressive fusion of the original structure and degradation enhancement features to obtain clearer, more consistent, and more robust restoration results, overcoming the performance bottlenecks of existing integrated methods in semantic understanding and feature fusion. Attached Figure Description

[0056] Figure 1 This is a flowchart of the method of the present invention.

[0057] Figure 2 This is a schematic diagram of the framework of the integrated image restoration model TGMIR.

[0058] Figure 3 This is a schematic diagram of the TAEB (Triple Interaction Attention Enhancement Module) process.

[0059] Figure 4 A schematic diagram of the TG-DSAF (Text-Guided Two-Stage Attention Fusion) mechanism.

[0060] Figure 5 This is a schematic diagram of the local-global dual attention process.

[0061] Figure 6 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0062] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0063] Example 1

[0064] This invention aims to overcome the core limitations of existing integrated image restoration models under multiple degradation conditions, including insufficient degradation semantics, significant cross-degradation interference, weak modal collaboration capabilities, and insufficient feature fusion. Because degradations such as noise, blur, and rain / fog differ greatly in distribution and structural characteristics, and unified models need to share parameters, conflicts easily arise in feature representation, making it difficult for the network to accurately distinguish the dominant degradation type. Existing methods mostly rely on implicit inference and lack explicit semantic constraints, often resulting in misidentification and unstable restoration under complex or mixed degradation conditions. Existing textual prompts also have a relatively shallow application, failing to build deep collaboration at the channel, spatial, and cross-modal levels.

[0065] The basic idea of ​​this invention is to introduce text descriptions as explicit degraded semantic cues, and guide the model to perform consistent, controllable, and semantically interpretable feature modulation under multiple degradation conditions through a cross-modal alignment mechanism. To this end, this invention proposes a text-guided integrated image restoration model, TGMIR, which maps text cues to a semantic space consistent with image features, and achieves hierarchical semantic control at three levels: channel attention, spatial attention, and cross-modal collaborative attention. Based on this, this invention further constructs a text-guided two-stage attention fusion mechanism to achieve dynamic bidirectional selection between the inherent structure of the original image and degradation enhancement features, enabling the restoration process to maintain structural consistency while compensating for degraded regions.

[0066] like Figure 1 As shown, the present invention provides an integrated image restoration method, the implementation method of which is as follows:

[0067] S1. Construct a text-guided integrated image restoration model TGMIR. The integrated image restoration model TGMIR is used to map text prompts to a semantic space with consistent image features. It utilizes the triple interactive attention enhancement module TAEB to achieve hierarchical semantic control at three levels: channel attention, spatial attention, and cross-modal collaborative attention. It also utilizes the text-guided two-stage attention fusion mechanism TG-DSAF to fuse the original features and degraded image features processed by the three attention mechanisms.

[0068] S2. Use the integrated image restoration model TGMIR to restore the image.

[0069] In this embodiment, the integrated image restoration model TGMIR includes:

[0070] Input layer, used to obtain degraded images ;

[0071] Convolutional layers for degraded image-based processing By capturing low-level texture and edge information, primary features are extracted. ;

[0072] Encoder-decoder framework for use based on primary features High-level semantic feature representations are generated through layer-by-layer transformation. Multiple Transformer blocks are used at each stage of the codec, with the number of Transformer blocks gradually increasing from the top to the bottom. A Text-Guided Feature Enhancement Fusion (TG-FEF) module connects the encoder and decoder at each stage; among them,

[0073] During the encoding phase, based on primary features By downsampling the data while continuously expanding the channel capacity, a latent representation can be obtained. The text guides the visual features; in the decoding stage, the latent features are upsampled and concatenated with the degraded enhanced features processed by the text-guided feature enhancement fusion module TG-FEF in the encoding stage to generate a high-level semantic feature representation. ;

[0074] The output layer is used to represent high-level semantic features. After optimization, a residual image is generated through a second convolutional layer and compared with the degraded image. The image reconstructed from the residuals obtained by addition This completes the integrated image restoration.

[0075] In this embodiment, as Figure 2 As shown, the core forward propagation process of a four-level encoder-decoder network is as follows:

[0076] Encoding stage: The input image generates L1 features through the image patch embedding module OverlapPatchEmbed, and then generates L2 / L3 / L4 features through 3 downsampling + Transformer blocks. The four layers of features are fused with the text encoding through the text-guided feature enhancement fusion module TG-FEF to complete the text guidance of the core visual features.

[0077] Decoding stage (cross-modal fusion + feature concatenation): Latent features are upsampled to L3 size and concatenated with L3 features from the encoding stage (after cross-modal collaborative attention processing). After channel compression, the concatenation is sent to the L3 decoder. The above logic is repeated: L3 decoded features are upsampled to L2 decoded and concatenated with L2 encoded features (after text fusion) → L2 decoded; L2 decoded features are upsampled to L1 decoded and concatenated with L1 encoded features (after text fusion) → L1 decoded.

[0078] Output stage: After the L1 decoded features are optimized by the refinement layer, they are output... The convolutional layer generates a residual image, which is then added to the original input image to obtain the final output. (Residual Reconstruction) Figure 2 In the diagram, L1, L2, L3, and L4 represent the levels of the codec. The four encoder levels on the left progressively downsample, with skip connections between each level of the codec. This corresponds to each text-guided feature enhancement fusion module (TG-FEF) in the diagram. Figure 2 middle, Represents the output characteristics of the encoder , and These represent the height and width of the image, respectively. This represents the channel dimension of the feature map obtained after encoder processing. It represents the set of real numbers.

[0079] In this embodiment, the text-guided feature enhancement fusion module TG-FEF includes the triple interactive attention enhancement module TAEB and the text-guided two-stage attention fusion mechanism TG-DSAF;

[0080] The Triple Interactive Attention Enhancement Module (TAEB) is used to obtain degenerate enhancement features containing degenerate information through a channel-space-cross-modal triple attention mechanism with semantic similarity feedback.

[0081] The text-guided two-stage attention fusion module TG-DSAF consists of a local-global dual attention module LGDA. The local-global dual attention module LGDA obtains feature weights by calculating global attention and local attention separately. weight The text-guided two-stage attention fusion module TG-DSAF uses two local-global dual attention modules LGDA to process the original features. and enhanced features A two-stage fusion process is performed to ultimately obtain the fusion characteristics. ,in, Indicates original features, This indicates that the degradation enhancement features containing degradation information are obtained through a channel-space-cross-modal triple attention mechanism.

[0082] In this embodiment, obtaining the degradation enhancement feature containing degradation information includes the following steps:

[0083] To obtain semantic information, at the channel level, the text encoding t is projected onto the image channel space. Generate channel-wise semantic weights, where the text encoding t is a description of the text's degradation type;

[0084] semantic weights are combined with input image features Multiply each channel sequentially to obtain the channel characteristics. ;

[0085] Channel characteristics Dimensionality reduction is performed, and spatial attention-enhanced features are generated using lightweight deep convolutional methods. ;

[0086] Based on spatial attention enhancement features Generate a Value V and a Key K, based on text encoding. To generate the query value Q;

[0087] Multiply the key value K and the query value Q to obtain the attention weight A. Then, weight the attention weight A onto the value V and restore it to the reduced channel dimension. and the original spatial dimension;

[0088] Based on the recovery results, through Convolution recovers the channel dimension C, yielding cross-modal guided attention features. In this embodiment, cross-modal guided attention features The expression is as follows:

[0089] ;

[0090] in, This represents a 1×1 convolution, which restores the reduced-dimensional features to their original channel dimension C. A multiplied by V is a weighted fusion, resulting in cross-modal guided attention features. 。

[0091] Fusion channel features Spatial attention enhancement features and cross-modal guided attention features This yields degraded enhancement features that are text-fused and contain degradation information. :

[0092] ;

[0093] in, This means restoring the dimension-reduced difference terms to dimension C and then re-phaseing them. This represents degenerate enhancement features that have undergone text fusion and contain degenerate information. All of these represent learnable coefficients.

[0094] In this embodiment, the weights of the degradation enhancement features are obtained. weight This includes the following steps:

[0095] Based on the input feature X, the semantic information t is dimensionally projected to obtain the local injection vectors. and global injection vector ;

[0096] In the local attention branch, pointwise convolution is chosen as the context aggregator;

[0097] Aggregate contexts using a context aggregator;

[0098] Based on the aggregation results, local features are calculated through the bottleneck structure. :

[0099] ;

[0100] in, Indicates original features, and Both represent pointwise convolution. This represents the GELU activation function. Represents a local injection vector; Bottleneck structure: Point-wise Conv+Relu+Point-wise Conv: Point-wise Conv: Pointwise convolution (i.e., a representation of 1×1 convolution), used for cross-channel feature fusion and dimension adjustment; Relu: Activation function; Conv 1×1: 1×1 convolution, used for feature dimensionality reduction / upgrading or channel mapping;

[0101] In the global attention branch, global average pooling is applied to obtain a global vector, which is then processed by one-dimensional convolution and the sigmoid activation function is applied to calculate the query and key values.

[0102] A global attention graph is generated by calculating the outer product of the query value and the key value. :

[0103] ;

[0104] in, and Let K and Q represent the dimensionally adjusted forms of K and Q, used for outer product calculation, respectively, and T denotes transpose. Indicates the transpose of K2;

[0105] global attention map The process is applied to the original feature channel dimension and global injected text is added to obtain global features. :

[0106] ;

[0107] Here, reshape indicates rearranging the dimensions of the input features to form the Value feature. This represents the attention weighting used to calculate the value, where A represents the attention weight. Represents the global injection vector. It represents the Hadamardi (or Hadama) stack;

[0108] The output is achieved by concatenating global features along the channel dimension. and local features The fusion weights are obtained after passing through the Sigmoid activation function. :

[0109] .

[0110] In this embodiment, as Figure 2 As shown, input degraded image (H×W represents the spatial dimensions, C represents the channel dimension, H represents the height of the input image, and W represents the width of the input image.) First, it passes through a... Convolutional layers extract primary features This is used to capture low-level texture and edge information. Subsequently, these primary features... It is fed into an encoder-decoder framework consisting of four layers, and high-level semantic feature representations are generated through layer-by-layer transformation. The encoder continuously expands the channel capacity while downsampling to obtain a more compact latent representation. The decoder, on the other hand, is responsible for progressively restoring a clear, high-resolution output. Each stage of the codec employs multiple Transformer blocks, with the number of Transformer blocks gradually increasing from the top to the bottom to achieve a computationally efficient design. Figure 2 In this framework, Restormer is used as the backbone network, which is an encoding / decoding network that includes Transformer blocks in both the encoding and decoding stages. The main component of this framework is the text-guided feature enhancement fusion module TG-FEF, which consists of two sub-modules: the text-guided triple interactive attention enhancement module TAEB and the text-guided two-stage attention fusion mechanism TG-DSAF.

[0111] Features at different levels interact through skip connections to mitigate the loss of detail. Unlike traditional direct feature concatenation, this invention embeds a text-guided feature enhancement fusion module (TG-FEF) at the skip connection stage. This module introduces degenerate text prompts to achieve cross-modal reconstruction and adaptive fusion of features. Specifically, the TG-FEF module uses a pre-trained CLIP model (contrastive language-image pre-trained model) to encode degeneracy type descriptions in text form. This guides the feature enhancement process, enabling image features at the encoding end to carry more explicit degeneracy semantics when transmitted to the decoding end. Through this semantically guided feature interaction mechanism, the network exhibits stronger generalization and robustness in multi-degeneracy scenarios.

[0112] In this embodiment, shared parameters are difficult to effectively handle integrated image restoration in an integrated setup. This invention specifically designs a text-guided feature enhancement fusion module (TG-FEF) to apply perturbations to the encoded features, such as... Figure 2 As shown. Guided by degradation type hints, they can be better aligned with the shared decoder in multi-tasking scenarios. Given image features Degradation type hints As input, the entire process of the integrated image restoration model TGMIR is defined as follows:

[0113] ;

[0114] in, Indicates the encoder input, Represents the image features output by the encoder. , , These represent the height and width of the image obtained after encoder processing, respectively. This represents the channel dimension of the feature map obtained after encoder processing, and t represents the text degradation type hint. This indicates a text-guided two-stage attention fusion operation. This indicates a triple attention enhancement module.

[0115] For example, if the feature length of the text encoder output by the CLIP model (contrastive language-image pre-trained model) is 512, then L=512, and the dimension of the degradation type cue t is 512×1 (i.e., ).

[0116] The Text-Guided Feature Enhancement Fusion Module (TG-FEF) consists of two core components: the Triple Interactive Attention Enhancement Module (TAEB) and the Text-Guided Two-Stage Attention Fusion Mechanism (TG-DSAF).

[0117] The Triple Interactive Attention Enhancement Module (TAEB) mitigates various degradation interferences in integrated image restoration by implementing cross-modal guidance, such as... Figure 3 As shown. Because text data has strong semantic and logical attributes, it is difficult to match and align with image data in the feature space. This invention proposes a channel-space-cross-modal triple attention mechanism for feature enhancement. Specifically, the module takes the semantic information extracted by CLIP as input. First, at the channel level, the text is encoded... (The text in the image is "Text") Projected onto the image channel space , This indicates a text projection operation. Represents the parameters of the Linear layer. Let B represent the activation function GELU, B represent the batch size, and C represent the channel dimension.

[0118] Guided by textual priors, channel-level adaptive reweighting is applied to image features to obtain channel-enhanced features for degradation perception: ,in, This represents the parameters of the two fully connected layers in the channel attention mechanism. Expand the channel weights to broadcast to spatial dimensions. Used to perform dimensional transformation on the vector after text projection. Used to further adjust the feature dimensions to match the channel dimensions of image features (e.g.) Figure 3 Linear layer Represents the GELU activation function (e.g.) Figure 3 In the diagram, GELU stands for Gaussian error linear unit. In the figure, B,num_head represents the number of attention heads, and head_dim represents the dimension per head. It represents the Hadamardi (or Hadama) stack.

[0119] Subsequently, the Triple Interactive Attention Enhancement Module (TAEB) reduces the dimensionality of the image features to... Furthermore, a spatial attention map is generated using lightweight depthwise separable convolution (i.e., after passing through a linear layer, the GELU activation function, and another linear layer), thus obtaining spatial attention enhancement features. This allows for precise emphasis on the spatial location of "degraded areas," among which... Indicates the channel dimension after dimensionality reduction:

[0120] ;

[0121] in, Represents the channel features after dimensionality reduction , This represents depthwise separable convolution. This indicates GroupNorm normalization. It represents the Hadamardi (or Hadama) stack.

[0122] Following the first two levels of enhancement, the Triple Interactive Attention Enhancement (TAEB) module performs cross-modal cross-attention between text and image in the dimensionality-reduced space. This better addresses the challenge of feature dimensionality mismatch between semantic and image information, enabling the retrieval of the region in image features that best matches the current degraded description. In the dimensionality-reduced representation, text vectors are mapped to queries, through... Convolution converts image features Map to Key / Value and rearrange into a multi-head format:

[0123]

[0124] in, This represents the feature mapping function first used in the 1×1 convolution. Then, a cross-attention approach of "single query (text) to global space tokens" is employed to calculate weights for each head: ,in, Used to scale the dot product result.

[0125] The obtained cross-modal features are then processed... Weighted And restore to The dimension is restored to the original channel dimension using a 1×1 convolution.C The cross-enhancement features are obtained: .

[0126] The final three-channel fusion output yields a text-fused degraded enhancement feature containing degradation information. :

[0127] ;

[0128] in, This means restoring the dimension-reduced difference terms to dimension C and then adding them together. This represents enhanced features that have undergone text fusion and contain degraded information. All represent learnable coefficients used to adaptively adjust the triple attention contribution.

[0129] In this embodiment, the features enhanced by the Triple Interactive Attention Enhancement Module (TAEB) can adapt well to specific degradations, but lack the use of inherent feature information. To address this issue, this invention designs a text-guided two-stage attention fusion mechanism, TG-DSAF. It consists of a local-global dual attention module (LGDA) to achieve multi-scale two-stage fusion of original features and degraded image features. The text-guided two-stage attention fusion mechanism TG-DSAF has two main functions: facilitating information interaction between enhanced features and original features, and displaying the perceived degradation type through text guidance for accurate restoration.

[0130] Text-guided two-stage attention fusion process, as follows Figure 4 and Figure 5 As shown, a local-global dual attention module (LGDA) is designed, denoted as M, to obtain the initial integrated features. :

[0131] ;

[0132] in, Indicates original features, Indicates degradation enhancement features, This represents the Hadamard product. Its structure is as follows: Figure 4 As shown, the solid line represents The dashed line represents They stated that the fusion weights consist of real numbers between 0 and 1, which enables the network to... and Soft selection or weighted averaging is performed between them. The initial integration is then passed again through the local-global dual attention module LGDA, and an additive is added before the final output. Convolution enhances channel representation capabilities to obtain fused features. :

[0133] ;

[0134] Local-Global Dual Attention Module (LGDA) Figure 4 and Figure 5 As shown. First, a simple dimensionality projection is needed to obtain vectors for local and global injection. The local attention branch selects point-wise convolution as the context aggregator, which only utilizes point-wise channel interactions at each spatial location. To save parameters, the local channel context is calculated through a bottleneck structure as follows:

[0135] ;

[0136] in, Indicates original features, Features indicating local attention and Both represent pointwise convolution. This indicates that the degenerate text has been reduced in dimensionality to obtain channel-dimensional text prompts.

[0137] The global attention branch first applies global average pooling (GAP) to obtain the global vector. Then, through a one-dimensional convolution with a kernel size of k=3... After processing, the sigmoid activation function is applied to compute the Query and Key. To model global interactions, the outer product of Q and K is calculated to generate a global attention map F.

[0138] ;

[0139] ;

[0140] in, Represents one-dimensional convolution. and Let K and Q represent the dimensionally adjusted forms of K and Q, used for outer product calculation, respectively, and T denotes transpose. express The transpose of is used to ensure that the outer product dimensions match, where K represents the Key value and Q represents the Query value.

[0141] Then apply it to the original feature channel dimension and add the globally injected text to obtain the global features. :

[0142] ;

[0143] in, This represents the Hadamard product; reshape indicates rearranging the dimensions of the input features as the Value feature; and value×A represents calculating the attention weights of the Value. A represents the global dimension of the degraded text, and A represents the attention weight.

[0144] The final output of the Local-Global Dual Attention Module (LGDA) is achieved by concatenating local and global attention features along the channel dimension. and The fused features are obtained after passing through the Sigmoid activation function. .

[0145] In this embodiment, as Figure 5 As shown, the Local-Global Dual Attention Module (LGDA) simultaneously captures local feature details and global feature dependencies, generating weights through a dual-branch attention mechanism for feature modulation (highlighting important information); C×W×H: feature dimensionality information, where C represents the channel dimension, W represents the image width, and H represents the image height. Point-wise Conv: pointwise convolution (another way of describing 1×1 convolution), used for cross-channel feature fusion and dimensionality adjustment; GAP (Global Average Pooling): a pooling operation commonly used in convolutional neural networks (CNNs), averaging the spatial dimension (H×W) of the input features to output global features of dimension [B,C] (capturing global dependencies); ReLU: activation function; Conv 1×1: 1×1 convolution, used for feature dimensionality reduction / upgrading or channel mapping; Transpose: transpose (usually referring to matrix transpose), used to adjust feature dimensions to match the requirements of matrix multiplication, where B represents the batch size and r represents the dimensionality reduction coefficient. This represents the Sigmoid activation function.

[0146] Compared with existing integrated image restoration methods, the integrated image restoration model TGMIR proposed in this invention utilizes text prompts to achieve explicit degradation semantic modeling, and introduces a text-guided triple interactive attention enhancement module TAEB and a text-guided two-stage attention fusion mechanism TG-DSAF into the feature modulation and fusion process. Based on the above key technologies, this invention has the following significant advantages:

[0147] Explicit semantic alignment significantly improves the accuracy and stability of image restoration: Existing integrated image restoration methods rely on the network to infer the degradation type itself, which is easily affected by the differences in the distribution of multiple degradations. This invention provides explicit semantic cues through text descriptions and uses cross-modal attention to align the text embedding with image features, enabling the network to accurately distinguish different degradation components such as noise, blur, rain, and fog even in a shared parameter space. By introducing semantic conditions into the channel-space-cross-modal three-layer attention, this invention effectively suppresses the conflict interference between multiple degradations, achieves more stable feature modulation, and thus obtains more accurate degradation recognition and restoration results.

[0148] The dynamic two-stage fusion mechanism reduces overfitting and retains more original structural information: The text-guided two-stage attention fusion mechanism TG-DSAF proposed in this invention dynamically adjusts the weights of the two types of features in the "structure preservation stage" and the "degradation compensation stage," ensuring that enhanced features only take effect in necessary areas and avoiding excessive restoration of details. This mechanism improves the restoration quality of complex regions while maintaining structural consistency, thereby enhancing the overall visual effect.

[0149] Significantly improved performance across multiple degradations in a unified parameter space, reducing the number of models and network resource consumption: Traditional methods require designing multiple models or a large number of independent sub-networks for different tasks. This invention utilizes textual semantics as a unified control signal, enabling the network to adaptively handle multiple degradations in a shared parameter space without the need for additional independent model training. Therefore, in practical deployments, this invention can significantly reduce model storage size and GPU memory usage, achieving network resource savings and improved inference efficiency.

[0150] Degradation identification without external labels improves data utilization and security: This invention uses text prompts instead of degradation labels, avoiding tedious manual annotation and eliminating the need to upload images to external servers for degradation analysis. This design enables local inference, helping to reduce the potential risk of data leakage during training and application, and significantly improving data security.

[0151] It has strong generalization ability across degradation scenarios and can adapt to complex and mixed degradation scenarios: Through text-guided semantic enhancement, this invention can not only handle single degradation, but also accurately separate noise, blur, rain patterns and other components in mixed degradation scenarios, and avoid mutual interference through a two-stage fusion mechanism.

[0152] Experiments show that the present invention maintains stable performance under multiple datasets and multi-task conditions, demonstrating strong generalization ability. Extensive experiments on multiple benchmark datasets show that the integrated image restoration model TGMIR achieves state-of-the-art performance in several integrated image restoration tasks, including image denoising, image dehazing, and image deraining. Table 1 shows a comparison with state-of-the-art techniques in these three tasks.

[0153] Table 1

[0154]

[0155] PSNR (decibels, higher is better) and SSIM (higher is better) metrics were reported on the full RGB image.

[0156] Example 2

[0157] like Figure 6 As shown, the present invention provides an integrated image restoration system for performing the integrated image restoration method described in Embodiment 1, comprising:

[0158] The first processing module is used to construct a text-guided integrated image restoration model, TGMIR. The integrated image restoration model TGMIR is used to map text prompts to a semantic space with consistent image features. It uses the triple interactive attention enhancement module TAEB to achieve hierarchical semantic control at three levels: channel attention, spatial attention, and cross-modal collaborative attention. It also uses the text-guided two-stage attention fusion mechanism TG-DSAF to fuse the original features and degraded image features processed by the three attention processes.

[0159] The second processing module is used to restore the image using the integrated image restoration model TGMIR.

[0160] like Figure 6 The integrated image restoration system provided in the illustrated embodiment can execute the technical solution shown in the integrated image restoration method of the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be repeated here.

[0161] In this embodiment, the functional units can be divided according to the integrated image restoration method. For example, each function can be divided into its own functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this invention is illustrative and represents only a logical division; in actual implementation, other division methods may be used.

[0162] In this embodiment, the integrated image restoration system, in order to realize the principle and beneficial effects of the integrated image restoration method, includes hardware structures and / or software modules corresponding to the execution of various functions. Those skilled in the art should readily recognize that, based on the illustrative units and algorithm steps described in conjunction with the embodiments disclosed in this invention, the present invention can be implemented in hardware and / or a combination of hardware and computer software. Whether a function is executed by hardware or computer software depends on the specific application and design constraints of the technical solution. Different methods can be used to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

Claims

1. An integrated image restoration method, characterized in that, Includes the following steps: S1. Construct a text-guided integrated image restoration model TGMIR. The integrated image restoration model TGMIR is used to map text prompts to a semantic space with consistent image features. It utilizes the triple interactive attention enhancement module TAEB to achieve hierarchical semantic control at three levels: channel attention, spatial attention, and cross-modal collaborative attention. It also utilizes the text-guided two-stage attention fusion mechanism TG-DSAF to fuse the original features and degraded image features processed by the three attention mechanisms. S2. Restore the image using the integrated image restoration model TGMIR; The integrated image restoration model TGMIR includes: Input layer, used to obtain degraded images ; Convolutional layers for degraded image-based processing By capturing low-level texture and edge information, primary features are extracted. ; Encoder-decoder framework for use based on primary features High-level semantic feature representations are generated through layer-by-layer transformation. Multiple Transformer blocks are used at each stage of the codec, with the number of Transformer blocks gradually increasing from the top to the bottom. A Text-Guided Feature Enhancement Fusion (TG-FEF) module connects the encoder and decoder at each stage; among them, During the encoding phase, based on primary features By downsampling the data while continuously expanding the channel capacity, a latent representation can be obtained. The text guides the visual features; in the decoding stage, the latent features are upsampled and concatenated with the degraded enhanced features processed by the text-guided feature enhancement fusion module TG-FEF in the encoding stage to generate a high-level semantic feature representation. ; The output layer is used to represent high-level semantic features. After optimization, a residual image is generated through a second convolutional layer and compared with the degraded image. The image reconstructed from the residuals obtained by addition Complete integrated image restoration; The text-guided feature enhancement fusion module TG-FEF includes the triple interactive attention enhancement module TAEB and the text-guided two-stage attention fusion mechanism TG-DSAF; The Triple Interactive Attention Enhancement Module (TAEB) is used to obtain degenerate enhancement features containing degenerate information through a channel-space-cross-modal triple attention mechanism with semantic similarity feedback. The text-guided two-stage attention fusion module TG-DSAF consists of a local-global dual attention module LGDA. The local-global dual attention module LGDA obtains feature weights by calculating global attention and local attention separately. weight The text-guided two-stage attention fusion module TG-DSAF uses two local-global dual attention modules LGDA to process the original features. and enhanced features A two-stage fusion process is performed to ultimately obtain the fusion characteristics. ,in, Indicates original features, This indicates that the degradation enhancement features containing degradation information are obtained through a channel-space-cross-modal triple attention mechanism.

2. The integrated image restoration method according to claim 1, characterized in that, Obtaining the degradation enhancement feature containing degradation information includes the following steps: To obtain semantic information, at the channel level, the text encoding t is projected onto the image channel space. Generate channel-wise semantic weights, where the text encoding t is a description of the text's degradation type; semantic weights are combined with input image features Multiply each channel sequentially to obtain the channel characteristics. ; Channel characteristics Dimensionality reduction is performed, and spatial attention-enhanced features are generated using lightweight deep convolutional methods. ; Based on spatial attention enhancement features Generate a Value V and a Key K, based on text encoding. To generate the query value Q; Multiply the key value K and the query value Q to obtain the attention weight A. Then, weight the attention weight A onto the value V and restore it to the reduced channel dimension. and the original spatial dimension; Based on the recovery results, through Convolution recovers the channel dimension C, yielding cross-modal guided attention features. ; Fusion channel features Spatial attention enhancement features and cross-modal guided attention features This yields degraded enhancement features that are text-fused and contain degradation information. .

3. The integrated image restoration method according to claim 2, characterized in that, The degradation enhancement features The expression is as follows: ； in, This means restoring the dimension-reduced difference terms to dimension C and then adding them together. This represents degenerate enhancement features that have undergone text fusion and contain degenerate information. All of these represent learnable coefficients.

4. The integrated image restoration method according to claim 1, characterized in that, The obtained feature weights weight This includes the following steps: Based on the input feature X, the semantic information is dimensionally projected to obtain the local injection vectors. and global injection vector ; In the local attention branch, pointwise convolution is chosen as the context aggregator; Aggregate contexts using a context aggregator; Based on the aggregation results, local features are calculated through the bottleneck structure. : ； in, Indicates original features, and Both represent pointwise convolution. This represents the GELU activation function. Represents the local injection vector; In the global attention branch, global average pooling is applied to obtain a global vector, which is then processed by one-dimensional convolution and the sigmoid activation function is applied to calculate the query and key values. A global attention graph is generated by calculating the outer product of the query value and the key value. : ； in, and Let K and Q represent the dimensionally adjusted forms of K and Q, used for outer product calculations, respectively. T represents the transpose. Indicates the transpose of K2; global attention map The process is applied to the original feature channel dimension and global injected text is added to obtain global features. : ； Here, reshape indicates rearranging the dimensions of the input features to form the Value feature. This represents the attention weighting used to calculate the value, where A represents the attention weight. Represents the global injection vector. It represents the Hadamardi (or Hadama) stack; The output is achieved by concatenating global features along the channel dimension. and local features The fusion weights are obtained after passing through the Sigmoid activation function. : 。 5. The integrated image restoration method according to claim 4, characterized in that, The fusion feature The expression is as follows: ； ； Where M represents the Local-Global Dual Attention Module (LGDA), Indicates original features, Z represents the degenerate enhancement feature, and Z represents the initial integrated feature. It represents the Hadamardi (or Hadama) stack.

6. An integrated image restoration system for performing the integrated image restoration method according to any one of claims 1-5, characterized in that, include: The first processing module is used to construct a text-guided integrated image restoration model, TGMIR. The integrated image restoration model TGMIR is used to map text prompts to a semantic space with consistent image features. It uses the triple interactive attention enhancement module TAEB to achieve hierarchical semantic control at three levels: channel attention, spatial attention, and cross-modal collaborative attention. It also uses the text-guided two-stage attention fusion mechanism TG-DSAF to fuse the original features and degraded image features processed by the three attention processes. The second processing module is used to restore the image using the integrated image restoration model TGMIR.

Images