Multi-scene image degradation integrated recovery method based on visual language model

Abstract

The invention provides a multi-scene image degradation integrated recovery method based on a visual language model, which realizes high-quality image reconstruction under various weather degradation conditions, and comprises the following steps: inputting a given degraded image into an encoder for encoding to obtain encoding features; querying the pre-trained visual language model through a cross-modal prompt generator to generate a multi-scale degradation perception cross-modal prompt; the pre-trained visual language model takes a problem and a degraded image as input; the degeneration perception cross-modal prompt is input into the restoration trunk through a guide attention alignment module to be aligned and fused with the coding features, and final output of the guide attention alignment module is obtained; refining and fusing the degraded image and the final output of the attention alignment guiding module through a double feature compensation module to obtain the final total output; and carrying out decoding processing and image reconstruction on the final total output through a decoder to obtain a high-quality output image.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision, and relates to image restoration and multi-modal semantic understanding technology, in particular to a multi-scene image degradation integrated recovery method based on a visual language model. BACKGROUND

[0002] Adverse weather removal is a low-level vision task aiming to restore high-quality images degraded by environmental conditions such as rain, fog and snow. In early work, specific task networks are mainly customized for a single degradation type, and need to be retrained or modified in structure for different scenes, which limits the application of the model in the real world.

[0003] To improve the generalization ability of the model, the latest research turns to use an integrated model to handle different weather degradation. Most of these methods use an encoder-decoder structure, but these frameworks usually rely on implicit feature learning without explicit modeling of the underlying degradation condition mechanism, resulting in limited adaptability under different weather conditions. Secondly, the introduction of pre-trained visual language models shows advantages in using semantic knowledge to guide recovery. These methods reveal the potential of cross-modal prompt guidance to eliminate adverse weather by using semantic prompts from visual language models, but they often treat text information as a global background, thus lacking explicit spatial alignment between prompts and degradation areas, limiting their ability to capture complex and diverse degradation patterns. SUMMARY

[0004] To solve the problems in the prior art, the application provides a multi-scene image degradation integrated recovery method based on a visual language model, which generates multi-scale degradation perception cross-modal prompts through a cross-modal prompt generator, and realizes adaptive explicit spatial alignment of semantic prompts and degradation areas with the help of a guided attention alignment module, and enhances detail and structure recovery in combination with a dual feature compensation module, thereby realizing high-quality image reconstruction under various weather degradation.

[0005] The technical scheme adopted by the application to solve the technical problems is as follows:

[0006] A multi-scene image degradation integrated recovery method based on a visual language model, comprising the following steps:

[0007] Step 1: input a given degraded image into an encoder to obtain encoded features;

[0008] Step 2: generate multi-scale degradation perception cross-modal prompts through a cross-modal prompt generator to query a pre-trained visual language model; the pre-trained visual language model takes questions and degraded images as input;

[0009] Step 3: Align and fuse the degradation perception cross-modal cues into the restoration backbone with the encoded features through the guided attention alignment module to obtain the final output of the guided attention alignment module;

[0010] Step 4: Refine and fuse the degraded image and the final output of the guided attention alignment module through the dual feature compensation module to obtain the final total output;

[0011] Step 5: Decode and reconstruct the final total output through the decoder to obtain a high-quality output image.

[0012] Based on the above scheme, the method designs a cross-modal cue generator, a guided attention alignment module, and a dual feature compensation module. The pre-trained visual language model is used to generate the semantic description of the question and the multi-scale degradation perception cross-modal cues of the input degraded image, which promotes the context perception recovery and improves the structure fidelity. The guided attention alignment module inputs these cues into the backbone network, and adaptively aligns the semantic cues with the degraded areas. The dual feature compensation module enhances the structure fidelity by separating the degradation artifacts from the scene content, effectively preserving fine textures and structural details. This method can be applied to multiple degraded image restoration, providing an efficient and accurate method for image restoration that can be applied in various fields such as autonomous driving, medical image analysis, satellite remote sensing image processing, etc.

[0013] Further, the step 1 is specifically:

[0014] Given a degraded image I LQ , the initial features X t are first extracted by the convolutional layer, and then X t is processed by the four-level encoder to obtain the encoded features.

[0015] Further, step 2 specifically includes:

[0016] Step 2.1: Input the question Q t and the degraded image I LQ into the pre-trained visual language model to obtain the question answer T text , the specific formula is as follows:

[0017] T text = PVL(Q t , I LQ ), T text ∈R L×N (1)

[0018] Where: PVL(·) represents the pre-trained visual language model; L represents the description length; N represents the channel dimension;

[0019] Step 2.2: Freeze the text encoder to extract T text Encoding as embedded feature T emb The specific formula is as follows:

[0020] T emb =Enc text (T text ), T emb ∈R L×C (2)

[0021] Among them: Enc text This indicates that the text encoder is frozen;

[0022] To ensure compatibility with the restoration model, T emb Projected onto the embedding space with channel dimension C;

[0023] Step 2.3: Using a baseline architecture image encoder to extract data from degraded image I LQ Extracting visual features I emb The specific formula is as follows:

[0024] I emb =Enc img (I LQ ),I emb ∈R H×W×C (3)

[0025] Among them: Enc img The image encoder represents the baseline architecture; H, W, and C represent the image height, width, and channel dimension, respectively.

[0026] Step 2.4: Use spatial cross-attention mechanism to integrate T emb and I emb This facilitates the modeling of relationships between different modalities. The query Q is obtained by projecting text features, while the key K and value V are obtained by projecting image features. The specific formulas are as follows:

[0027]

[0028] in: These are the corresponding projection matrices;

[0029] Step 2.5: Finally, the degenerate perception cross-modal cue P is calculated using the spatial cross-attention mechanism. da The specific formula is as follows:

[0030]

[0031] Where: β represents the scaling factor.

[0032] Furthermore, step 3 specifically includes:

[0033] Step 3.1: Convert the encoded features F obtained in Step 1 into... ori The Transformer module is used to extract the degradation features F. de The specific formula is as follows:

[0034] F de =TFM(F ori (6)

[0035] Where: TFM(·) represents the Transformer operation;

[0036] Step 3.2: The cue alignment module in the guided attention alignment module will degenerate the perception of cross-modal cues P da and degradation characteristics F de Alignment is performed to obtain the output F t1 The specific formula is as follows:

[0037] Q d =D3C1(F de ),K d =D3C1(F de ),V d =D3C1(F de (7)

[0038] Q p =D3C1(P da ),K p =D3C1(P da ),V p =D3C1(P da (8)

[0039] F m =CA(C1(w[Q d Q p ]),K d V d (9)

[0040] F n =CA(Q) d ,C1(w[K d ,K p ]),C1(w[V d V p ])) (10)

[0041] F i =C1(w[F m ,F n ])+F de (11)

[0042] F t1=C1D3φC1D3(LN(F i ))+F i (12)

[0043] Among them: Q d ,K d V d These represent the query feature, key feature, and value feature in the downgrade features, respectively; Q p ,K p V p represents the query feature, key feature, and value feature in the perception prompt, respectively; D3(·) and C1(·) represent a 3×3 deep convolutional network and a 1×1 convolutional network, respectively; CA(·) represents the cross attention operation; C1D3φC1D3(·) represents the lightweight convolutional module operation; LN represents layer normalization; w represents the concatenation operation of two features in the channel dimension;

[0044] Step 3.3: The cue enhancement module in the guided attention alignment module injects spatial information cues using cue features, and refines the features of the degradation region through a series of residual modules. Specifically: given a degradation-perceived cross-modal cue P da and degradation characteristics F de The cue enhancement module in the guided attention alignment module first connects their projected representations along the channel dimension. Then, multiple residual modules process the fused features to selectively recover fine structure and details. The specific formula is as follows:

[0045] F u =Res(w[Q d Q p ])⊙K d (13)

[0046] F v =Res(w[K d ,K p ])⊙Q d (14)

[0047] F t2 =Res(C1(w[F u ,F v ])+y(F de (15)

[0048] Where: ⊙ represents element-wise product; y(·)=LN(Res(·)); Res represents the residual module;

[0049] Step 3.4: Obtain the final output y of the guided attention alignment module map The specific formula is as follows:

[0050]

[0051] Where R(·) represents the ReLU activation function.

[0052] Furthermore, step 4 specifically includes:

[0053] Step 4.1: The feature compensation branch of the dual feature compensation module calculates the degraded image I. LQ The variance between the maximum and minimum channel components is used to effectively refine the degraded image content, and finally, local details are refined through convolution operations. The specific formula is as follows:

[0054]

[0055] C out =C1(C5(C3C1(C map (18)

[0056] Where: r, g, b represent the image's color channels red, green, and blue; C i i∈1,3,5 represents a convolution operation;

[0057] Step 4.2: The degradation recovery branch of the dual feature compensation module introduces linear transformation and ReLU activation function to further refine the final output of the guided attention alignment module. The specific formula is as follows:

[0058] y out =sigmoid(LN(R(LN(y)) map (19)

[0059] Where: LN represents linear transformation; sigmoid represents sigmoid activation function; R(·) represents ReLU activation function;

[0060] Step 4.3: Integrate the two branches to obtain the final output, as shown in the following formula:

[0061] y t+1 =y out ⊙C out (20)

[0062] Where ⊙ represents element-wise product.

[0063] The beneficial effects of this invention include:

[0064] A cross-modal cue generator is introduced, which generates text embeddings using a pre-trained visual language model and fuses them with visual features to generate degradation-aware cross-modal cuees. A guided attention alignment module is designed to adaptively adjust feature responses and highlight feature-related regions, improving the semantic guidance of the model and establishing a spatial correspondence between degradation-aware cross-modal cuees and degradation regions. In addition, a dual feature compensation module is used to separate degradation artifacts from scene content and enhance key visual details to improve structural fidelity, resulting in high-quality restored images. Attached Figure Description

[0065] Figure 1 This is a model framework diagram for a multi-scene image degradation integrated restoration method based on a visual language model. Detailed Implementation

[0066] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0068] Example 1: This example uses Windows system as the development environment, PyCharm as the development platform, and Python as the development language. It adopts the multi-scene image degradation integrated restoration method based on visual language model of the present invention to complete the restoration work for images with different severe weather conditions.

[0069] In this embodiment, the multi-scene image degradation integrated restoration method based on a visual language model includes the following steps:

[0070] Step 1: First, input the given degraded image into the encoder for encoding to obtain the encoded features;

[0071] Step 2: Query the pre-trained visual language model using a cross-modal cue generator to generate multi-scale degradation-aware cross-modal cues;

[0072] Step 3: Align and fuse the degenerate perception cross-modal cue input into the restored backbone with the encoded features through the guided attention alignment module to obtain the final output of the guided attention alignment module;

[0073] Step 4: The degraded image and the final output of the guided attention alignment module are refined and fused through the dual feature compensation module to obtain the final total output;

[0074] Step 5: Decode the final output and reconstruct the image using a decoder to obtain a high-quality output image.

[0075] This implementation uses peak signal-to-noise ratio and structural similarity index measurements.

[0076] Based on the above steps, this invention was compared with models such as SPANet, JSTASR, RESCAN, DesnowNet, DDMSNet, NAFNet, Restormer, and All-in-One on different datasets. As can be seen from Tables 1, 2, and 3, the method proposed in this invention is generally superior to other methods in terms of various indicators on these two common test sets.

[0077] Table 1 Comparison of different models on the Snow100K-S and Snow100K-L datasets

[0078]

[0079]

[0080] Note: The best results are highlighted.

[0081] Table 2 Comparison of different models on the Outdoor-Rain dataset

[0082]

[0083]

[0084] Note: The best results are highlighted.

[0085] Table 3 Comparison of different models on the RainDrop dataset

[0086]

[0087] Note: The best results are highlighted.

[0088] In summary, this invention provides a multi-scene image degradation integrated restoration method based on a visual language model. It extracts degradation-perceived cross-modal cues by integrating a pre-trained visual language model and visual features; it designs an adaptive attention alignment module that adjusts feature responses and highlights feature-related regions to improve the semantic guidance of the model and establishes a spatial correspondence between degradation-perceived cross-modal cues and degradation regions; furthermore, it enhances structural fidelity by separating degradation artifacts from scene content and enhancing key visual details through a dual feature compensation module, resulting in a high-quality restored image.

[0089] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A multi-scene image degradation integrated restoration method based on a visual language model, characterized in that, The method includes the following steps: Step 1: Input the given degraded image into the encoder for encoding to obtain encoded features; Step 2: Query the pre-trained visual language model using a cross-modal cue generator to generate multi-scale degradation-aware cross-modal cue; the pre-trained visual language model takes the question and the degradation image as input; Step 3: Align and fuse the degenerate perception cross-modal cue input into the restored backbone with the encoded features through the guided attention alignment module to obtain the final output of the guided attention alignment module; Step 4: The degraded image and the final output of the guided attention alignment module are refined and fused through the dual feature compensation module to obtain the final total output; Step 5: Decode the final output and reconstruct the image using a decoder to obtain a high-quality output image.

2. The multi-scene image degradation integrated restoration method based on a visual language model according to claim 1, characterized in that, Step 1 specifically involves: Given a degraded image I LQ First, the initial feature X is extracted by the convolutional layer. t After X t The encoded features are obtained by processing with a four-level encoder.

3. The multi-scene image degradation integrated restoration method based on a visual language model according to claim 1 or 2, characterized in that, Step 2 specifically includes: Step 2.1: Question Q t and degraded image I LQ As input to a pre-trained visual language model, the answer T is obtained. text The specific formula is as follows: T text =PVL(Q t ,I LQ ),T text ∈R L×N (1) Where: PVL(·) represents a pre-trained visual language model; L represents the description length; N represents the channel dimension; Step 2.2: Freeze the text encoder to extract T text Encoding as embedded feature T emb The specific formula is as follows: T emb =Enc text (T text ),T emb ∈R L×C (2) Among them: Enc text This indicates that the text encoder is frozen; Then T emb Projected onto the embedding space with channel dimension C; Step 2.3: Using a baseline architecture image encoder to extract data from degraded image I LQ Extracting visual features I emb The specific formula is as follows: I emb =Enc img (I LQ ),I emb ∈R H×W×C (3) Among them: Enc img The image encoder represents the baseline architecture; H, W, and C represent the image height, width, and channel dimension, respectively. Step 2.4: Use spatial cross-attention mechanism to integrate T emb and I emb The query Q is obtained from the projection of text features, while the key K and value V are obtained from the projection of image features. The specific formulas are as follows: in: These are the corresponding projection matrices; Step 2.5: Finally, the degenerate perception cross-modal cue p is calculated using the spatial cross-attention mechanism. da The specific formula is as follows: Where: β represents the scaling factor.

4. The multi-scene image degradation integrated restoration method based on a visual language model according to claim 3, characterized in that, Step 3 specifically includes: Step 3.1: Convert the encoded features F obtained in Step 1 into... ori The Transformer module is used to extract the degradation features F. de The specific formula is as follows: F de =TFM(F ori ) (6) Where: TFM(·) represents the Transformer operation; Step 3.2: The cue alignment module in the guided attention alignment module will degenerate the perception of cross-modal cues P da and degradation characteristics F de Alignment to obtain the output The specific formula is as follows: Q d =D3C1(F de ),K d =D3C1(F de ),V d =D3C1(F de ) (7) Q p =D3C1(P da ),K p =D3C1(P da ),V p =D3C1(P da ) (8) F m =CA(C1(w[Q d ,Q p ]),K d ,V d ) (9) F n =CA(Q d ,C1(w[K d ,K p ]),C1(w[V d ,V p ])) (10) F i =C1(w[F m ,F n ])+F de (11) Among them: Q d ,K d V d These represent the query feature, key feature, and value feature in the downgrade features, respectively; Q p ,K p V p represents the query feature, key feature, and value feature in the perception prompt, respectively; D3(·) and C1(·) represent a 3×3 deep convolutional network and a 1×1 convolutional network, respectively; CA(·) represents the cross attention operation; C1D3φC1D3(·) represents the lightweight convolutional module operation; LN represents layer normalization; w represents the concatenation operation of two features in the channel dimension; Step 3.3: Given a degradation-aware cross-modal cue P da and degradation characteristics F de The cue enhancement module in the guided attention alignment module first connects their projected representations along the channel dimension. Then, multiple residual modules process the fused features to selectively recover fine structure and details. The specific formula is as follows: F u =Res(w[Q d ,Q p ])⊙K d (13) F v =Res(w[K d ,K p ])⊙Q d (14) Where: ⊙ represents element-wise product; y(·)=LN(Res(·)); Res represents the residual module; Step 3.4: Obtain the final output y of the guided attention alignment module map The specific formula is as follows: Where R(·) represents the ReLU activation function.

5. The multi-scene image degradation integrated restoration method based on a visual language model according to claim 4, characterized in that, Step 4 specifically includes: Step 4.1: The feature compensation branch of the dual feature compensation module calculates the degraded image I. LQ The variance between the maximum and minimum channel components is used to refine the degraded image content, and finally, local details are refined through convolution operations. The specific formula is as follows: C out =C1(C5(C3C1(C map ))) (18) Where: r, g, b represent the image's color channels red, green, and blue; C i i∈1,3,5 represents a convolution operation; Step 4.2: The degradation recovery branch of the dual feature compensation module introduces linear transformation and ReLU activation function to further refine the final output of the guided attention alignment module. The specific formula is as follows: y out =sigmoid(LN(R(LN(y map )))) (19) Where: LN represents linear transformation; sigmoid represents sigmoid activation function; R(·) represents ReLU activation function; Step 4.3: Integrate the two branches to obtain the final output, as shown in the following formula: and t+1 / and out ⊙C out (20) Where ⊙ represents element-wise product.

Images