A text-guided zero-shot image restoration and enhancement method and system

Abstract

This invention discloses a text-guided zero-shot image restoration and enhancement method and system. The method includes: acquiring an original degraded image and constructing positive and negative text pools; generating a restored image and a degraded image through a restoration network and a degraded network, respectively; constructing a prior loss; obtaining reconstructed restored and degraded images through a restoration model and a degraded model, and calculating a reconstruction loss; converting the original degraded image, the restored image, and the reconstructed restored image into image embedding vectors, and converting the text in the positive and negative text pools into text embedding vectors, and selecting positive and negative text embedding vectors; mapping the image embedding vectors of the restored image and the selected positive and negative text embedding vectors to a multimodal embedding space and calculating a contrastive learning loss; constructing a total loss function based on the prior loss, reconstruction loss, and contrastive learning loss for iterative training, and outputting the restored image from the restoration model as the final result. This invention improves the fusion capability of multimodal information and the quality of image restoration and enhancement.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a text-guided zero-sample image restoration and enhancement method and system. Background Technology

[0002] Image restoration and enhancement aims to recover high-quality, clear images from damaged observations. Early image restoration methods mainly constructed prior knowledge by establishing mathematical and physical assumptions and observing the statistical features of a large number of natural images. Typical examples include dark channel priors, total variational models, and local smoothing constraints. With the development of deep learning, researchers have used convolutional neural networks or Transformers to handle the current mainstream image restoration and enhancement tasks, such as image dehazing, low-light enhancement, and underwater image enhancement. However, most deep learning-based methods are highly dependent on large-scale image datasets for training. In the absence of sufficient data or in zero-sample scenarios, the generalization ability of such methods is often severely limited, making it difficult to achieve ideal restoration results.

[0003] Existing zero-shot image restoration methods have the following drawbacks:

[0004] (1) Most of them are designed for a single degradation task, making it difficult to achieve image restoration for multiple tasks within a unified framework;

[0005] (2) The auxiliary information such as text modality and CLIP multi-scale features has not yet been integrated to improve the feature representation capability of the network;

[0006] (3) It lacks a dynamic optimization mechanism and cannot dynamically select the optimal text semantics based on the specific degradation degree of the current image. Summary of the Invention

[0007] To overcome the defects and shortcomings of existing technologies, this invention provides a text-guided zero-sample image restoration and enhancement method and system. This invention can effectively and dynamically utilize text information to guide image restoration and enhancement, significantly improving the ability to fuse multimodal information and the quality of image restoration and enhancement in various complex degradation scenarios.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] This invention provides a text-guided zero-shot image restoration and enhancement method, comprising the following steps:

[0010] Obtain the original degraded image and construct a positive text pool and a negative text pool to describe the image quality;

[0011] The original degraded image is processed by a restoration network and a degradation network to generate a restored image and a degraded image, respectively;

[0012] Construct a priori guidance module to build corresponding prior losses based on the restored and degenerate maps;

[0013] The original degraded image, the output of the prior guidance module, and the degraded image are used by the restoration model to obtain a reconstructed image. The reconstructed image is obtained by using the restoration image, the degraded image, and the output of the prior guidance module. The reconstruction loss is calculated based on the reconstructed degraded image and the original degraded image.

[0014] The original degraded image, the restored image, and the reconstructed restored image are converted into image embedding vectors. The text in the positive text pool and the negative text pool are converted into text embedding vectors. The positive text embedding vector and the negative text embedding vector are then selected.

[0015] The image embedding vector of the restored image is mapped to the selected positive text embedding vector and negative text embedding vector into the multimodal embedding space to calculate the comparative learning loss;

[0016] The total loss function is constructed based on prior loss, reconstruction loss, and contrastive learning loss, and iterative training is performed. The restored image output by the restoration model is then used as the final output.

[0017] As a preferred technical solution, the restoration network includes a CLIP image encoder, a multimodal multiscale decoder, a restoration block, and a final processing block;

[0018] CLIP image encoder extracts multi-scale features from the original degraded image;

[0019] The restoration block extracts features from the original degraded image and outputs image features through multi-layer convolution, batch normalization, activation function and channel attention operations;

[0020] The multimodal multiscale decoder is equipped with multiple text-guided edge attention blocks. Based on the multiscale features extracted by the CLIP image encoder, the positive text embedding vector, and the image features output by the restoration block, it performs feature enhancement and multimodal information fusion.

[0021] The final processing block has multiple convolutional layers, and the output features of the multimodal multiscale decoder are processed by the final processing block to obtain the restored image.

[0022] As a preferred technical solution, the multimodal multiscale decoder has four text-guided edge attention blocks. The first edge attention block takes the image features output by the restoration block, the fourth scale feature extracted by the CLIP image encoder, and the positive text embedding vector as input and outputs the first image feature.

[0023] The second edge attention block takes the first image features, the third scale features extracted by the CLIP image encoder, and the text embedding vector as input, and outputs the second image features.

[0024] The third edge attention block takes the second image features, the second-scale features extracted by the CLIP image encoder, and the text embedding vector as input, and outputs the third image features.

[0025] The fourth edge attention block takes the third image feature, the first-scale feature extracted by the CLIP image encoder, and the text embedding vector as input, and outputs the fourth image feature.

[0026] As a preferred technical solution, the text-guided edge attention block enhances the features of the input image and fuses multimodal information to obtain multimodal features;

[0027] The multi-scale features extracted by the CLIP image encoder and the text embedding vector are fused to obtain the fused features;

[0028] Edge information in the horizontal and vertical directions of the image is extracted based on convolution operation, and the extracted edge information is added together to obtain the edge fusion feature;

[0029] After performing element-wise multiplication on the multimodal features and edge fusion features, the output features are combined with the input image features to obtain the output features.

[0030] As a preferred technical solution, the degenerate network has multiple consecutive degenerate blocks and convolutional blocks, and each degenerate block includes multiple convolutions, batch normalization, and ReLU activation functions.

[0031] As a preferred technical solution, the original degraded image, the output of the prior guidance module, and the reconstructed image obtained by the restoration model from the degraded image are represented as follows:

[0032] ;

[0033] in, Represents the reconstructed restored image. Represents the original degraded image. This represents the output of the prior guidance module. This represents the degradation graph output by the degradation network. This indicates an element-wise dot division operation.

[0034] As a preferred technical solution, the restored image, the degraded image, and the output of the prior guidance module are processed by the degradation model to obtain a reconstructed degraded image, which is represented as follows:

[0035] ;

[0036] in, Represents a reconstructed, degraded image. This represents the degradation graph output by the degradation network. This represents the restored image output by the restoration network. This indicates the output of the prior guidance module.

[0037] As a preferred technical solution, the reconstruction loss is calculated based on the reconstructed degraded image and the original degraded image, specifically including:

[0038] The reconstruction loss is calculated by constraining the reconstructed degraded image and the original degraded image based on the L2 norm.

[0039] As a preferred technical solution, the text in the positive text pool and the negative text pool are converted into text embedding vectors, and the positive text embedding vectors and negative text embedding vectors are obtained by filtering, specifically including:

[0040] Based on the CLIP text encoder, the positive text in the positive text pool and the negative text in the negative text pool are transformed into the same multimodal embedding space to obtain the positive text embedding vector and the negative text embedding vector before filtering.

[0041] Calculate the cosine similarity between the text embedding vector before filtering, the image embedding vector of the restored image, and the image embedding vector of the reconstructed image, and calculate the optimal score corresponding to the text embedding vector.

[0042] Calculate the cosine similarity between the negative text embedding vector before filtering and the image embedding vector of the original degraded image, the image embedding vector of the restored image, and the image embedding vector of the reconstructed restored image, and calculate the optimal score corresponding to the negative text embedding vector;

[0043] Positive and negative text embedding vectors are obtained based on the optimal score selection.

[0044] This invention also provides a text-guided zero-shot image restoration and enhancement system for implementing the above-mentioned text-guided zero-shot image restoration and enhancement method, comprising: an original degraded image acquisition module, a text pool construction module, a restoration network, a degraded network, a prior guidance module, a prior loss construction module, a restoration model construction module, a degraded model construction module, a reconstruction loss calculation module, an embedding vector conversion module, a text embedding vector filtering module, a contrastive learning loss calculation module, an iterative training module, and an output module;

[0045] The original degraded image acquisition module is used to acquire the original degraded image;

[0046] The text pool construction module is used to construct positive and negative text pools that describe image quality;

[0047] The restoration network is used to generate a restored image from the original degraded image;

[0048] The degradation network is used to generate a degradation map from the original degradation image;

[0049] The prior guidance module is used to construct the prior guidance module;

[0050] The prior loss construction module is used to construct the corresponding prior loss based on the restored map and the degenerate map;

[0051] The restoration model construction module is used to construct the restoration model. The original degraded image, the output of the prior guidance module, and the degraded image are used to obtain the reconstructed restored image through the restoration model.

[0052] The degradation model construction module is used to construct a degradation model. The restored image, degradation image, and the output of the prior guidance module are used to obtain a reconstructed degradation image through the degradation model.

[0053] The reconstruction loss calculation module is used to calculate the reconstruction loss based on the reconstructed degraded image and the original degraded image;

[0054] The embedding vector conversion module is used to convert the original degraded image, the restored image, and the reconstructed restored image into image embedding vectors, and to convert the text in the positive text pool and the negative text pool into text embedding vectors.

[0055] The text embedding vector filtering module is used to filter and obtain positive text embedding vectors and negative text embedding vectors;

[0056] The contrastive learning loss calculation module is used to map the image embedding vector of the restored image and the selected positive text embedding vector and negative text embedding vector to the multimodal embedding space to calculate the contrastive learning loss.

[0057] The iterative training module is used to construct a total loss function based on prior loss, reconstruction loss, and contrastive learning loss for iterative training.

[0058] The output module is used to output the restored image output by the restoration model as the final result.

[0059] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0060] (1) This invention uses multimodal fusion to introduce text information and image features, increases the information available in zero-sample image restoration and enhancement tasks, accelerates model convergence speed, achieves the technical effect of obtaining high-quality output without training set, and significantly improves image processing performance.

[0061] (2) This invention constructs a restoration and enhancement framework that can handle various types of image degradation. Based on the restoration and enhancement results, text embedding vectors are dynamically selected. The selected text embedding vectors are used as multimodal information and image features for fusion. At the same time, the image embedding vectors of the restored image are mapped to the multimodal embedding space along with the selected positive and negative text embedding vectors to calculate the contrastive learning loss. The contrastive learning constrains the output of the restoration network and the text that describes the clear image. Then, based on the contrastive learning loss, reconstruction loss and the prior loss of the corresponding task, a total loss function is constructed for iterative training. The restored image output by the restoration model is used as the final high-quality clear image. It can effectively and dynamically utilize text information to guide the restoration and enhancement of images, significantly improving the model's ability to fuse multimodal information and the quality of image restoration and enhancement in various complex degradation scenarios. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating the text-guided zero-shot image restoration and enhancement method of the present invention.

[0063] Figure 2 This paper presents a schematic diagram of the overall implementation architecture of a text-guided zero-sample image restoration and enhancement method based on the present invention.

[0064] Figure 3 This is a schematic diagram of the network architecture of the restored network of the present invention;

[0065] Figure 4 This is a schematic diagram of the cosine similarity between a degraded image and its corresponding sharp image, obtained by the CLIP image encoder.

[0066] Figure 5 This is a schematic diagram of the network architecture of the text-guided edge attention block of the present invention;

[0067] Figure 6 This is a schematic diagram of the network architecture of the degradation network of the present invention;

[0068] Figure 7 This is a schematic diagram of the network architecture of the visual language contrastive learning module of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0070] Example 1

[0071] like Figure 1 , Figure 2As shown, this embodiment provides a text-guided zero-shot image restoration and enhancement method, including the following steps:

[0072] S1: Obtain the degraded image as the input image;

[0073] S2: Construct cue words, construct a positive text pool describing clear images and a negative text pool describing degraded images;

[0074] In this embodiment, prompt words such as "Please generate 100 sentences describing clear images" and "Please generate 100 sentences describing {degradation} images" are input into the large language model to generate corresponding candidate text sets. Here, {degradation} is a variable dynamically configured based on the actual degradation type of the input image. For example, when processing a foggy image, this variable is configured as foggy.

[0075] Positive and negative text pools were selected from the candidate text set. The selection criteria were: the text content only describes the image quality and does not contain any semantic information about the image content. The text in the positive text pool is used to represent clear images, such as "Light and shadows play naturally, enhancing the scene without any blur". The text in the negative text pool is used to represent degraded images, such as "The photo has a low resolution, with visible pixelation and blurring".

[0076] S3: Construct restoration and degradation networks. The restoration network fuses multimodal features, multi-scale features, and textual features to output a restored image. The degradation network adaptively outputs a degradation map based on the input image. ;

[0077] like Figure 3 As shown, the restoration network includes a CLIP image encoder, a multimodal multiscale decoder, a restoration block, and a final processing block;

[0078] Multi-scale features extracted based on the CLIP image encoder are represented as follows:

[0079] ;

[0080] in, This represents a pre-trained CLIP image encoder. This indicates the first image extracted by the CLIP image encoder. Features at each scale Represents the original degraded image, such as Figure 4As shown, because the features extracted by the CLIP image encoder in this embodiment exhibit strong robustness against image degradation—specifically, the features of the degraded image and the corresponding sharp image at these scales have extremely high cosine similarity—this embodiment preferably selects the features from the first four scales, i.e. ;

[0081] The multimodal multiscale decoder is equipped with multiple text-guided edge attention blocks. Based on the multiscale features extracted by the CLIP image encoder, the positive text embedding vector, and the image features output by the restoration block, the image features and text features are fused, while enhancing edge details. Preferably, four text-guided edge attention blocks are set.

[0082] Specifically, the first edge attention block uses image features output from the restoration block. Features extracted at the fourth scale by the CLIP image encoder and text embedding vector As input, the first image feature is output. ;

[0083] The second edge attention block uses the first image features output by the first edge attention block. Features extracted at the third scale by the CLIP image encoder and text embedding vector As input, the output is the second image feature. ;

[0084] The third edge attention block uses the second image features from the output of the second edge attention block. Features extracted at the second scale by the CLIP image encoder and text embedding vector As input, the output is the third image feature. ;

[0085] The fourth edge attention block uses the third image features from the output of the third edge attention block. Features extracted at the first scale by the CLIP image encoder and text embedding vector As input, the output is the fourth image feature. ;

[0086] In this embodiment, the text embedding vector The initial value is randomly generated, and subsequent values ​​are generated by the visual language contrast learning module.

[0087] like Figure 5As shown, the text-guided edge attention block first undergoes feature enhancement and multimodal information fusion to obtain multimodal features, represented as follows:

[0088] ;

[0089] in, Represents multimodal features, This represents pixel attention operations. This indicates the instance normalization operation. express convolution, express Activation function Representing image features, ;

[0090] By fusing the multi-scale features extracted by the CLIP image encoder and the text embedding vector, the fused features are obtained, represented as:

[0091] ;

[0092] in, Indicates fusion features, This indicates the nearest neighbor upsampling operation. Indicates a fully connected layer. Represents the embedding vector of the main text. express convolution, This represents the element-wise dot product operation.

[0093] Edge information in the x and y directions is extracted from the fused features using convolution operations. The extracted edge information is then summed to obtain the edge fusion features. , is represented as:

[0094] ;

[0095] Multimodal features, edge blending features, and image features Combining, represented as:

[0096] ;

[0097] in, This represents the output features of the edge attention block.

[0098] The restoration block extracts features from the input degraded image, specifically as follows:

[0099] ;

[0100] in, express convolution, Indicates batch normalization. express Activation function This indicates a channel attention operation. This represents the image features extracted from the restored block.

[0101] The final processing block consists of two 3×3 convolutional layers and a sigmoid function, representing the output features of the multimodal, multi-scale decoder. The final processed block yields the restored image, specifically represented as follows:

[0102] ;

[0103] in, Represents the restored image. express Activation function.

[0104] like Figure 6 As shown, the degenerate network has multiple consecutive degenerate blocks and convolutional blocks. In this embodiment, four blocks are preferred, with each degenerate block including two convolutional blocks. Convolution, a batch normalization, and a ReLU activation function are used in the degradation network to estimate the degradation map, which characterizes the degradation information of an image under light scattering. Specifically, it is represented as follows:

[0105] ;

[0106] in, Represents a degenerate graph. This represents four consecutive degenerate blocks. express convolution.

[0107] S4: Input the restored image output by the restoration network and the degraded image output by the degradation network into the adaptive prior guidance module. The prior guidance module integrates prior knowledge of various images and can use different priors according to the degradation type of the input image to construct the prior loss of the corresponding task.

[0108] In this embodiment, the Dark Channel Prior (DCP) designed for fog images can output atmospheric light values, and the Color Attenuation Prior (CAP) can be used to constrain the restored image of the restoration network to have low brightness and high saturation, satisfying the characteristics of a real fog-free image. The Background Light Estimation Prior (BLE) designed for underwater turbid images can obtain the global background light from the input image. The Gamma Correction designed for low-light images can adjust the output of the degradation network. For low-light enhancement tasks, the color constancy loss constraint is used to make the average color of each sensor channel in the entire image range tend to gray. For underwater enhancement tasks, no task-specific prior loss is used. For other restoration and enhancement tasks, some prior knowledge can be selectively used to constrain the network to achieve better results. Finally, the output of the prior guidance module is obtained. ;

[0109] Specifically, in the defogging task, the output The atmospheric light values ​​estimated by the prior DCP for the dark passage are output in the underwater enhancement mission. The global background light is estimated from the prior BLE output for background light estimation. In the low-light enhancement task, it is set to 0. The above design is to be consistent with the actual imaging mechanism under various degradation scenarios, so as to better conform to the physical imaging model corresponding to different degradation scenarios.

[0110] S5: Construct restoration and degradation models, and reconstruct the restored images respectively. and degraded images ;

[0111] The restored image is obtained by reconstructing the model. Specifically, it is expressed as:

[0112] ;

[0113] in, This represents the original degraded image. This represents the output of the prior guidance module. This represents the degradation graph output by the degradation network. This indicates an element-wise division operation. The restoration model explicitly models the degradation bias term and degradation intensity. By removing the degradation components, performing normalization correction, and finally reconstructing the imaging result, it achieves effective restoration of the degraded image.

[0114] Degraded image is obtained by reconstructing the degradation model. Specifically, it is expressed as:

[0115] ;

[0116] in, The degraded model uses the degraded graph to perform complementary weighting on the scene's effective information and the degradation priors, thereby achieving explicit decoupling and reorganization of the degradation components and content components, and thus obtaining a degradation reconstruction result that is more consistent with the real imaging process.

[0117] Constructing a reconstruction loss based on the L2 norm for degraded images And the original degraded image To impose constraints, specifically:

[0118] ;

[0119] in, This indicates the losses incurred during reconstruction.

[0120] S6: Construct a visual language contrastive learning module, and constrain the output of the restoration network based on the visual language contrastive learning module;

[0121] like Figure 7 As shown, the visual language contrastive learning module includes a positive text selection unit, a negative text selection unit, a CLIP image encoder, and a CLIP text encoder;

[0122] The CLIP image encoder in the visual language contrastive learning module will convert the original degraded image... The restored image of the restored network output. The restored image obtained by reconstructing the restoration model Transforming into a multimodal embedding space yields the corresponding image embedding vector, specifically represented as:

[0123] ;

[0124] ;

[0125] ;

[0126] in, , , These represent degraded images. Restoration diagram Image restoration Embedded vector, Indicates CLIP image encoder;

[0127] Preferably, in order to reduce the overall resource consumption of the model, the CLIP image encoder of the visual language contrastive learning module can reuse the CLIP image encoder in the restoration network, and the CLIP image encoder of the visual language contrastive learning module freezes the network parameters.

[0128] CLIP text encoder will extract the text from the text pool. Negative text in the negative text pool Transforming into the same multimodal embedding space is specifically represented as follows:

[0129] ;

[0130] ;

[0131] in, This represents the embedding vector of the main text before filtering. This represents the negative text embedding vector before filtering. This indicates the CLIP text encoder, which transforms the input text into a text embedding vector with the same dimensions as the image embedding vector output by the CLIP image encoder.

[0132] Text embedding vector Input text selection units, and obtain the filtered text embedding vector based on the random optimal score selection algorithm. ;

[0133] Negative text embedding vector Input negative text selection unit, and obtain the filtered negative text embedding vector based on the random optimal score selection algorithm. ;

[0134] In this embodiment, the random optimal score selection algorithm ensures that the text used in each iteration is closely related to the restoration result obtained in the current iteration. The specific implementation process is as follows:

[0135] A certain number of positive and negative texts are randomly selected from the positive text pool and the negative text pool. The positive texts used should be close to the restored image and far away from the degraded image, while the negative texts are exactly the opposite. The positive and negative texts are converted into corresponding text embedding vectors by the CLIP text encoder.

[0136] Calculate the score corresponding to each text embedding vector:

[0137] ;

[0138] ;

[0139] in, Represents cosine similarity. Indicates the weight value;

[0140] Using a random optimal score selection algorithm, in each iteration, the score corresponding to each text embedding vector is calculated until the highest value remains unchanged within a set number of iterations. Then, a set number of text embedding vectors with the highest scores are selected. and negative text embedding vectors Preferably, the text embedding vector with the highest score and the top 10 text embedding vectors can be selected as the filtered main text embedding vectors. and negative text embedding vectors Embed the filtered text into a vector. This is passed as a cue to the multimodal, multiscale decoder;

[0141] In this embodiment, the weight value Adaptive weights can be used, specifically as follows:

[0142] ;

[0143] in, Indicates the first iteration Indicated by integer division, as the quality of the restored image continuously improves, the similarity between the negative text and the two restored images gradually decreases, and its constraint effect weakens accordingly. Therefore, a linearly increasing weight value is adopted. The contribution of the recovered image can be maintained in subsequent iterations, thereby further improving the optimization effect of self-supervised learning.

[0144] Embed the filtered text into a vector Filtered negative text embedding vectors Restoration diagram Embedded vector Mapping to the shared multimodal embedding space, the contrastive learning loss is calculated and expressed as:

[0145] ;

[0146] This embodiment calculates the embedding vector of the restored image in a multimodal embedding space. The contrastive learning loss between the selected text embedding vectors constrains the alignment of the output of the restoration network with the text describing the clear image, that is, it constrains the alignment of the embedding vector of the restoration network output with the embedding vector of the main text.

[0147] S7: Construct a total loss function based on contrastive learning loss, reconstruction loss, and prior loss corresponding to the task, and input the degraded image. Perform self-supervised iterative training until the network converges, and then output the restored image from the restoration model. As the final high-quality, clear image.

[0148] The step numbers in the above embodiments are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0149] The performance of the text-guided zero-shot image restoration and enhancement method of this invention was comprehensively evaluated. The evaluation metrics used included the full-reference metrics PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) to measure image fidelity. For the output results without reference images, NIQE (Natural Image Quality Evaluator), NIMA (Neural Image Assessment), FADE (Fog Aware Density Evaluator), and UCIQE (Underwater Color Image Quality Evaluation) were used for comprehensive evaluation. The SOTS-outdoor, HSTS, Fattal's, and RTTS datasets were used in the dehazing scenario. The experimental results are shown in Table 1 below. The results compare the proposed method with the PSD (Principled Synthetic-to-real Dehazing), IDB (Isomorphic Dual-Branch), USID (Unsupervised Single Image Dehazing), ICDehazing (Illumination Controllable Dehazing), DGD (Depth-Guidance-Dehazing), ZIR (Zero-shot single Image Restoration), and RSF-Dehaze (Region Similarity Filling Dehaze) algorithms.

[0150] Table 1. Comparison of metrics with existing methods on synthetic (SOTS, HSTS) and real-world (Fattal's, RTTS) fog datasets.

[0151]

[0152] In Table 1, bold and underlined text represent the best and second-best results, respectively. "↑" ("↓") indicates that the larger (smaller) the value, the better. As shown in Table 1, the method proposed in this invention achieves the best performance in both fidelity and perceived quality indicators, while requiring fewer model parameters.

[0153] The underwater enhancement task was tested on the UIEB, U45, and EUVP datasets. The experimental results are shown in Table 2 below. The results demonstrate the effectiveness of this invention compared to algorithms such as MMLE (Minimal Color Loss and Locally Adaptive Contrast Enhancement), ICSP (Illumination Channel Sparsity Prior), C3HLM, and UIEC. 2 Comparison of metrics for the following algorithms: -Net (Underwater Image Enhancement Convolution Neural Network using 2Color Space), NU2Net (Normalization-based U-shape UIE Network), UIEDP (Underwater Image Enhancement with Diffusion Prior), UDNet (UncertaintyDistribution Network), and UUIR (Unsupervised and Untrained underwater Image Restoration).

[0154] Table 2. Comparison of metrics with existing methods on synthetic (UIEB, EUVP) and real-world (U45) underwater degradation datasets.

[0155]

[0156] In Table 2, bold and underlined text represent the best and second-best results, respectively. "↑" ("↓") indicates that the larger (smaller) the value, the better. As can be seen from Table 2, compared with existing underwater image enhancement methods, the method proposed in this invention has achieved the best results in PSNR, SSIM and UCIQE evaluation indicators, verifying its superiority and effectiveness.

[0157] The LOLv1 and LOLv2-Real datasets were used to evaluate the performance of low-light enhancement. The experimental results are shown in Table 3. The results show the performance comparison between the present invention and the following algorithms: RUAS (Retinex-inspired Unrolling with Architecture Search), SCI (Self-Calibrated Illumination), COLIE (Context-Based Low-Light Image Enhancement), Zero-DCE (Zero-Reference Deep Curve Estimation), Zero-DCE++ (an extended version of Zero-DCE), DUNP (Discrepant Untrained Network Priors), GDP (Generative Diffusion Prior), FourierDiff (Fourier Priors-Guided Diffusion), SeedOptimize (Seed Optimization), NeurBR, and IniRetinex (Initialization Retinex).

[0158] Table 3. Comparison of metrics with existing methods on real-world low-light datasets LOLv1 and LOLv2-Real.

[0159]

[0160] In Table 3, bold and underlined text represent the best and second-best results, respectively. "↑" ("↓") indicates that the larger (smaller) the value, the better. As shown in Table 3, the method proposed in this invention achieves the highest PSNR and SSIM compared to existing low-light enhancement algorithms.

[0161] This invention can effectively and dynamically utilize text information to guide image restoration and enhancement, significantly improving the model's ability to fuse multimodal information and the quality of image restoration and enhancement in various complex degradation scenarios.

[0162] Example 2

[0163] This embodiment provides a text-guided zero-shot image restoration and enhancement system to implement the text-guided zero-shot image restoration and enhancement method of Embodiment 1 above. It includes: an original degraded image acquisition module, a text pool construction module, a restoration network, a degraded network, a prior guidance module, a prior loss construction module, a restoration model construction module, a degraded model construction module, a reconstruction loss calculation module, an embedding vector transformation module, a text embedding vector selection module, a contrastive learning loss calculation module, an iterative training module, and an output module.

[0164] In this embodiment, the original degraded image acquisition module is used to acquire the original degraded image;

[0165] In this embodiment, the text pool construction module is used to construct a positive text pool and a negative text pool describing image quality;

[0166] In this embodiment, the restoration network is used to generate a restored image from the original degraded image;

[0167] In this embodiment, the degradation network is used to generate a degradation map from the original degradation image;

[0168] In this embodiment, the prior guidance module is used to construct the prior guidance module;

[0169] In this embodiment, the prior loss construction module is used to construct the corresponding prior loss based on the restored map and the degraded map;

[0170] In this embodiment, the restoration model construction module is used to construct a restoration model. The original degraded image, the output of the prior guidance module, and the degraded image are processed by the restoration model to obtain a reconstructed restored image.

[0171] In this embodiment, the degradation model construction module is used to construct a degradation model, and the restored image, degradation image, and the output of the prior guidance module are used to obtain a reconstructed degradation image through the degradation model;

[0172] In this embodiment, the reconstruction loss calculation module is used to calculate the reconstruction loss based on the reconstructed degraded image and the original degraded image;

[0173] In this embodiment, the embedding vector conversion module is used to convert the original degraded image, the restored image, and the reconstructed restored image into image embedding vectors, and to convert the text in the positive text pool and the negative text pool into text embedding vectors.

[0174] In this embodiment, the text embedding vector filtering module is used to filter and obtain positive text embedding vectors and negative text embedding vectors;

[0175] In this embodiment, the contrastive learning loss calculation module is used to map the image embedding vector of the restored image and the selected positive text embedding vector and negative text embedding vector to the multimodal embedding space to calculate the contrastive learning loss.

[0176] In this embodiment, the iterative training module is used to construct a total loss function based on prior loss, reconstruction loss, and contrastive learning loss for iterative training;

[0177] In this embodiment, the output module is used to output the restored image output by the restoration model as the final result.

[0178] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A text-guided zero-shot image restoration and enhancement method, characterized in that, Includes the following steps: Obtain the original degraded image and construct a positive text pool and a negative text pool to describe the image quality; The original degraded image is processed by a restoration network and a degradation network to generate a restored image and a degraded image, respectively; Construct a priori guidance module to build corresponding prior losses based on the restored and degenerate maps; The original degraded image, the output of the prior guidance module, and the degraded image are used by the restoration model to obtain a reconstructed image. The reconstructed image is obtained by using the restoration image, the degraded image, and the output of the prior guidance module. The reconstruction loss is calculated based on the reconstructed degraded image and the original degraded image. The original degraded image, the restored image, and the reconstructed restored image are converted into image embedding vectors. The text in the positive text pool and the negative text pool are converted into text embedding vectors. The positive text embedding vector and the negative text embedding vector are then selected. The image embedding vector of the restored image is mapped to the selected positive text embedding vector and negative text embedding vector into the multimodal embedding space to calculate the comparative learning loss; The total loss function is constructed based on prior loss, reconstruction loss, and contrastive learning loss, and iterative training is performed. The restored image output by the restoration model is then used as the final output.

2. The text-guided zero-shot image restoration and enhancement method according to claim 1, characterized in that, The restoration network includes a CLIP image encoder, a multimodal multiscale decoder, a restoration block, and a final processing block; CLIP image encoder extracts multi-scale features from the original degraded image; The restoration block extracts features from the original degraded image and outputs image features through multi-layer convolution, batch normalization, activation function and channel attention operations; The multimodal multiscale decoder is equipped with multiple text-guided edge attention blocks. Based on the multiscale features extracted by the CLIP image encoder, the positive text embedding vector, and the image features output by the restoration block, it performs feature enhancement and multimodal information fusion. The final processing block has multiple convolutional layers, and the output features of the multimodal multiscale decoder are processed by the final processing block to obtain the restored image.

3. The text-guided zero-shot image restoration and enhancement method according to claim 2, characterized in that, The multimodal multiscale decoder has four text-guided edge attention blocks. The first edge attention block takes the image features output by the restoration block, the fourth scale feature extracted by the CLIP image encoder, and the positive text embedding vector as input, and outputs the first image feature. The second edge attention block takes the first image features, the third scale features extracted by the CLIP image encoder, and the text embedding vector as input, and outputs the second image features. The third edge attention block takes the second image features, the second-scale features extracted by the CLIP image encoder, and the text embedding vector as input, and outputs the third image features. The fourth edge attention block takes the third image feature, the first-scale feature extracted by the CLIP image encoder, and the text embedding vector as input, and outputs the fourth image feature.

4. The text-guided zero-shot image restoration and enhancement method according to claim 2, characterized in that, The text-guided edge attention block enhances the features of the input image and fuses multimodal information to obtain multimodal features; The multi-scale features extracted by the CLIP image encoder and the text embedding vector are fused to obtain the fused features; Edge information in the horizontal and vertical directions of the image is extracted based on convolution operation, and the extracted edge information is added together to obtain the edge fusion feature; After performing element-wise multiplication on the multimodal features and edge fusion features, the output features are combined with the input image features to obtain the output features.

5. The text-guided zero-shot image restoration and enhancement method according to claim 1, characterized in that, The degenerate network has multiple consecutive degenerate blocks and convolutional blocks. Each degenerate block includes multiple convolutions, batch normalization, and ReLU activation functions.

6. The text-guided zero-shot image restoration and enhancement method according to claim 1, characterized in that, The original degraded image, the output of the prior guidance module, and the reconstructed image obtained from the degraded image through the restoration model are represented as follows: ； in, Represents the reconstructed restored image. Represents the original degraded image. This represents the output of the prior guidance module. This represents the degradation graph output by the degradation network. This indicates an element-wise dot division operation.

7. The text-guided zero-shot image restoration and enhancement method according to claim 1, characterized in that, The restored image, the degraded image, and the output of the prior guidance module are processed by the degradation model to obtain the reconstructed degraded image, which is represented as follows: ； in, Represents the reconstructed degraded image. This represents the degradation graph output by the degradation network. This represents the restored image output by the restoration network. This indicates the output of the prior guidance module.

8. The text-guided zero-shot image restoration and enhancement method according to claim 1, characterized in that, The reconstruction loss is calculated based on the reconstructed degraded image and the original degraded image, specifically including: The reconstruction loss is calculated by constraining the reconstructed degraded image and the original degraded image based on the L2 norm.

9. The text-guided zero-shot image restoration and enhancement method according to claim 1, characterized in that, The text from the positive and negative text pools is converted into text embedding vectors, and then filtered to obtain positive and negative text embedding vectors, specifically including: Based on the CLIP text encoder, the positive text in the positive text pool and the negative text in the negative text pool are transformed into the same multimodal embedding space to obtain the positive text embedding vector and the negative text embedding vector before filtering. Calculate the cosine similarity between the text embedding vector before filtering, the image embedding vector of the restored image, and the image embedding vector of the reconstructed image, and calculate the optimal score corresponding to the text embedding vector. Calculate the cosine similarity between the negative text embedding vector before filtering and the image embedding vector of the original degraded image, the image embedding vector of the restored image, and the image embedding vector of the reconstructed restored image, and calculate the optimal score corresponding to the negative text embedding vector; Positive and negative text embedding vectors are obtained based on the optimal score selection.

10. A text-guided zero-shot image restoration and enhancement system, characterized in that, The method for implementing the text-guided zero-shot image restoration and enhancement method according to any one of claims 1-9 includes: an original degraded image acquisition module, a text pool construction module, a restoration network, a degraded network, a prior guidance module, a prior loss construction module, a restoration model construction module, a degraded model construction module, a reconstruction loss calculation module, an embedding vector conversion module, a text embedding vector filtering module, a contrastive learning loss calculation module, an iterative training module, and an output module. The original degraded image acquisition module is used to acquire the original degraded image; The text pool construction module is used to construct positive and negative text pools that describe image quality; The restoration network is used to generate a restored image from the original degraded image; The degradation network is used to generate a degradation map from the original degradation image; The prior guidance module is used to construct the prior guidance module; The prior loss construction module is used to construct the corresponding prior loss based on the restored map and the degenerate map; The restoration model construction module is used to construct the restoration model. The original degraded image, the output of the prior guidance module, and the degraded image are used to obtain the reconstructed restored image through the restoration model. The degradation model construction module is used to construct a degradation model. The restored image, degradation image, and the output of the prior guidance module are used to obtain a reconstructed degradation image through the degradation model. The reconstruction loss calculation module is used to calculate the reconstruction loss based on the reconstructed degraded image and the original degraded image; The embedding vector conversion module is used to convert the original degraded image, the restored image, and the reconstructed restored image into image embedding vectors, and to convert the text in the positive text pool and the negative text pool into text embedding vectors. The text embedding vector filtering module is used to filter and obtain positive text embedding vectors and negative text embedding vectors; The contrastive learning loss calculation module is used to map the image embedding vector of the restored image and the selected positive text embedding vector and negative text embedding vector to the multimodal embedding space to calculate the contrastive learning loss. The iterative training module is used to construct a total loss function based on prior loss, reconstruction loss, and contrastive learning loss for iterative training. The output module is used to output the restored image output by the restoration model as the final result.

Images