Image restoration method and system based on single-step diffusion model and multi-modal CFG

Abstract

The invention provides an image restoration method and system based on a single-step diffusion model and multi-modal CFG. The method comprises the following steps: constructing a multi-mode prompt generation network based on a graph vector generation network; based on the multi-modal prompt generation network, respectively extracting different modal prompt information; fusing the prompt information of the different modes to generate positive and negative prompt embedding of a unified dimension; encoding a to-be-processed image to a submerged space vector, and performing CFG double forward reasoning in cooperation with the positive and negative prompt embedding to obtain a final noise prediction result; and on the basis of a single-step diffusion model, the final noise prediction result is utilized to calculate and obtain a final required image output. According to the method, a multi-mode CFG guiding strategy is provided for a single-step diffusion model, three-way complementary prompting is adopted, text modes and picture modes are combined for prompting, and better comprehensive performance can be obtained in various blind restoration / super-division scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and in particular to an image restoration method and system based on a single-step diffusion model and a multimodal CFG, as well as a corresponding computer terminal and a computer-readable storage medium. Background Art

[0002] The one-step diffusion (OSD) model has become an important direction in the fields of blind image restoration and super-resolution due to its fast inference speed and ease of implementation. However, most methods using classifier-free guidance (CFG) rely on a fixed negative prompt (see the technical paper "Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Bo Dai, Fanghua Yu, Wanli Ouyang, Yu Qiao, and Chao Dong, "DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior," in CVPR, 2023."), lacking the ability to adaptively characterize degraded input features.

[0003] There are two special methods for CFG designed for single-step diffusion models: 1. Using SDXL to batch synthesize 100,000 low-quality images based on a negative quality vocabulary (such as "oil painting, cartoon, blur, dirty, messy, low quality, deformation, low resolution, oversmooth"), and form positive / negative paired data with the high-quality original images. During training, both positive and negative cues are fed into the network to guide the network to learn the distribution of content that should be generated and content that must be avoided, respectively, thereby enabling the model to truly understand the meaning of negative cues (see the technical paper "Fanghua Yu, Jinjin Gu, Zheyuan Li et al., "Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration in the Wild," CVPR 2024."). During training, the target high-quality image is replaced with a synthesized low-quality image with a 5% probability, using the same negative quality vocabulary, allowing the model to simultaneously learn the concepts of "high quality" and "low quality" (see the technical paper "Aiping Zhang, Zongsheng Yue, Renjing Pei et al., "Degradation-Guided One-Step Image Super-Resolution with Diffusion Priors," arXiv 2024.").

[0004] However, these two methods often produce more distortion because they do not use the true degradation of the original image as a prompt during training, but only use a unified prompt word, resulting in:

[0005] 1. It is difficult to suppress multiple heterogeneous degradations (such as noise, compression artifacts, and human motion blur) simultaneously;

[0006] 2. The coupling degree between negative prompt information and positive prompt information is low, and the guidance effect is limited;

[0007] 3. The fidelity to the details in the image is insufficient, and the model randomly generates new details. Summary of the Invention

[0008] In response to the above-mentioned deficiencies in the prior art, the present invention provides an image restoration method and system based on a single-step diffusion model and multimodal CFG, and also provides a corresponding computer terminal and computer-readable storage medium.

[0009] According to one aspect of the present invention, there is provided an image restoration method based on a single-step diffusion model and a multimodal CFG, comprising:

[0010] Construct a multimodal prompt generation network based on graph vector generation network;

[0011] Extracting different modal prompt information based on the multimodal prompt generation network;

[0012] Fusing the different modal prompt information to generate positive and negative prompt embeddings of unified dimensions;

[0013] Encode the image to be processed into a latent space vector, and perform CFG double forward reasoning with the positive and negative cue embedding to obtain the final noise prediction result;

[0014] Based on the single-step diffusion model, the final restored image output is calculated using the final noise prediction result.

[0015] According to another aspect of the present invention, an image restoration system based on a single-step diffusion model and a multimodal CFG is provided, comprising:

[0016] A multimodal prompt information generation module is used to build a multimodal prompt generation network based on a graph vector generation network; based on the multimodal prompt generation network, extract different modal prompt information respectively; and fuse the different modal prompt information to generate a positive and negative prompt embedding of a unified dimension;

[0017] A noise prediction module is used to encode the image to be processed into a latent space vector and perform CFG double forward reasoning in conjunction with the positive and negative cue embedding to obtain the final noise prediction result;

[0018] The image restoration module is based on a single-step diffusion model and uses the final noise prediction result to calculate the final restored image output.

[0019] According to a third aspect of the present invention, a computer terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the terminal can be used to execute the method described above in the present invention, or to execute the system described above in the present invention.

[0020] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it can be used to execute the method described above in the present invention, or to run the system described above in the present invention.

[0021] Due to the adoption of the above technical solution, the present invention breaks through the limitations of the traditional single-modal fixed negative prompt and has at least one of the following beneficial effects compared with the existing technology:

[0022] The image restoration method and system based on the single-step diffusion model and multimodal CFG provided by the present invention adopt adaptive negative prompt generation technology, which can construct negative prompt information in real time according to the actual degradation of the input image, fully suppress various heterogeneous degradations such as noise, compression artifacts and local motion blur, and solve the problem in the existing technology that fixed prompt words are difficult to adapt to complex degradation scenarios.

[0023] The image restoration method and system based on the single-step diffusion model and multimodal CFG provided by the present invention adopt multimodal collaborative prompt fusion technology, combine image features, residual noise vectors and semantic / segmentation information to form complementary guidance signals, and through the first proposed "three-way complementary prompt" system, fuse the image residual noise branch, local degradation mask branch, high-quality image branch and text understanding prompts, and form positive and negative prompt embedding through Transformer encoding fusion, which improves the model's recognition and recovery capabilities of complex degraded features and the guidance effect, and solves the problems of low coupling of existing prompt information and limited guidance effect.

[0024] The image restoration method and system based on the single-step diffusion model and multimodal CFG provided by the present invention realize a fine-grained high-fidelity reconstruction mechanism through the process of noise prediction. It can fully restore the overall lighting, color and geometric structure of the input image, while accurately reconstructing the texture and edges masked by degradation in a fine-grained manner, reducing random artifacts and texture drift phenomena, and significantly improving subjective visual consistency and objective perception indicators such as LPIPS and DISTS, ensuring that subjective visual consistency and objective perception similarity indicators (such as LPIPS and DISTS) are improved simultaneously, solving the problem of unstable effect of existing models in detail restoration.

[0025] The image restoration method and system based on the single-step diffusion model and multimodal CFG provided by the present invention, combined with an improved single-step diffusion CFG guidance mechanism, collaboratively calculates the noise prediction results through dual forward noise prediction and multimodal prompting, taking into account both high restoration quality and high inference speed, and improving the practical application value of the single-step diffusion model in blind restoration and super-resolution scenarios.

[0026] The image restoration method and system based on the single-step diffusion model and multimodal CFG provided by the present invention improve the overall performance of the single-step diffusion model in image restoration tasks through the combination of adaptive negative prompts, multimodal collaborative prompts, high-fidelity restoration and an improved CFG guidance mechanism, and overcome the shortcomings of the existing technology in degradation adaptability, detail fidelity and reasoning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0028] Figure 1 This is a workflow diagram of an image restoration method based on a single-step diffusion model and multimodal CFG in a preferred embodiment of the present invention.

[0029] Figure 2 Schematic diagram of the component modules of an image restoration system based on a single-step diffusion model and multimodal CFG in a preferred embodiment of the present invention.

[0030] Figure 3 A network diagram is generated by graph vectors in a preferred embodiment of the present invention.

[0031] Figure 4 A network diagram is generated for multimodal prompting in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process. It should be noted that those skilled in the art may make various modifications and improvements without departing from the scope of the present invention, and these modifications and improvements fall within the scope of protection of the present invention.

[0033] Existing CFG methods for single-step diffusion models often suffer from the following technical issues: difficulty in simultaneously suppressing multiple heterogeneous degradations; low coupling between negative and positive prompts, limiting guidance effectiveness; and insufficient fidelity to image details, with the model randomly generating new details. To address these issues, one embodiment of the present invention provides an image restoration method based on a single-step diffusion model and multimodal CFG. This method proposes a multimodal CFG guidance strategy for the single-step diffusion model. Unlike the single-path prompting in existing technologies, this method proposes a "three-way complementary prompting" system for the first time, combining text and image modalities for prompting. This system can achieve comprehensive performance superior to existing methods in various blind restoration / super-resolution scenarios.

[0034] Specifically, if Figure 1 As shown, the image restoration method based on the single-step diffusion model and multimodal CFG provided in this embodiment may include:

[0035] S1, building a multimodal prompt generation network based on graph vector generation network;

[0036] S2, based on the multimodal prompt generation network, extracts prompt information of different modalities respectively;

[0037] S3, fuses the different modal prompt information to generate positive and negative prompt embeddings of unified dimensions;

[0038] S4, encode the image to be processed (low-resolution image) into a latent space vector, and perform CFG double forward reasoning with positive and negative cue embedding to obtain the final noise prediction result;

[0039] S5, based on the single-step diffusion model, using the final noise prediction result, calculates the final restored image output (high-resolution image).

[0040] In some preferred embodiments, the above S1, constructing a multimodal prompt generation network based on a graph vector generation network, may further include:

[0041] A graph vector generation network is provided, comprising a backbone network and three branch networks. The backbone network and the three branch networks each employ a multi-layer stacked residual Swin Transformer Block module as their main structure. The branch networks are trained as follows to construct a multimodal cue generation network:

[0042] Using high-quality images as training targets, we obtain a positive-prompt branch network.

[0043] The difference between low-quality and high-quality images is used as the training target to obtain the residual noise branch network;

[0044] The locally degraded segmentation mask generated by random noise is used as the training target to obtain a locally degraded branch network.

[0045] In some preferred embodiments, the above S2, based on the multimodal prompt generation network, extracting prompt information of different modalities, may further include:

[0046] S21, based on the branch network of the multimodal prompt generation network, extracts the following prompt information respectively:

[0047] Extract high-quality prompt vectors through the positive prompt branch network of the multimodal prompt generation network to obtain positive prompt vectors;

[0048] The noise cue vector is extracted through the residual noise branch network of the multimodal cue generation network to obtain the noise vector of the negative cue;

[0049] The local degradation cue vector is extracted through the local degradation branch network of the multimodal cue generation network to obtain the local degradation vector of the negative cue;

[0050] S22, introduces the image quality assessment model to generate image semantic text prompt information.

[0051] In some preferred embodiments, the above S3, fusing different modal prompt information to generate positive and negative prompt embeddings of unified dimensions, may further include:

[0052] Splice {P T , P H} and {P R , P M} and form a dimensionally consistent positive prompt embedding P through Transformer Encoder pos and negative hint embedding P neg Among them, P T For text prompts, P H For positive prompts, P R For noise prompt, P M This indicates local degradation.

[0053] In some preferred embodiments, the above S4, encoding the image to be processed into a latent space vector, and performing CFG double forward reasoning with positive and negative cue embedding to obtain the final noise prediction result, may further include:

[0054] S41, encode the low-resolution image to be processed into a latent space vector Z t ; The low-resolution image may be a picture downsampled to 128×128 or a picture with noise;

[0055] S42, the latent space vector Z t Embed P with positive prompt pos Perform a forward diffusion prediction to obtain the forward noise prediction ε pos ; The latent space vector Z t Embed P with negative prompt neg Perform a forward diffusion prediction to obtain the negative noise prediction ε neg ;

[0056] S43, the positive noise prediction and the negative noise prediction are integrated to obtain the final noise prediction result ε:

[0057] ε=ε neg +λ cfg (ε pos -ε neg );

[0058] Where λ cfg is a predetermined hyperparameter used to balance the strength of positive noise and negative noise.

[0059] In some preferred embodiments, the above S5, based on the single-step diffusion model, uses the final noise prediction result to calculate the final restored image output, and may further include:

[0060] The DDIM single-step diffusion model is used to output the final latent space vector to obtain the required high-resolution image (i.e., a picture with no noise, clear subject, and resolution greater than 512×512), which is expressed as:

[0061]

[0062] Where, is the latent space vector corresponding to the predicted high-quality image, z LQ is the latent space vector corresponding to the input low-quality image, is a constant related to the number of time steps t taken, t is the time step hyperparameter passed into the DDIM single-step diffusion model (in the DDIM single-step diffusion model, this parameter is generally fixed and then passed into the model), and ε is the final noise prediction result.

[0063] Based on the same inventive concept, an embodiment of the present invention further provides an image restoration system based on a single-step diffusion model and a multimodal CFG.

[0064] Specifically, if Figure 2 As shown, the image restoration system based on the single-step diffusion model and multimodal CFG provided by this embodiment may include:

[0065] The multimodal prompt information generation module is used to build a multimodal prompt generation network based on the graph vector generation network; based on the multimodal prompt generation network, it extracts prompt information of different modalities; and fuses the prompt information of different modalities to generate positive and negative prompt embeddings of unified dimensions;

[0066] The noise prediction module is used to encode the image to be processed into a latent space vector and perform CFG double forward reasoning with positive and negative cue embedding to obtain the final noise prediction result;

[0067] Image restoration module, based on the single-step diffusion model, uses the final noise prediction result to calculate the final restored image output.

[0068] The following further describes in detail the implementation of each functional module constituting the system provided by the above embodiment of the present invention in conjunction with the preferred implementation manner.

[0069] 1. The multimodal prompt information generation module, wherein the implementation method thereof further preferably includes:

[0070] 1. Network structure building unit:

[0071] It consists of a backbone network and three branch networks, each of which includes: downsampling convolution → residual Swin Transformer Block (RSTB) × N → {residual Swin Transformer Block (RSTB) × N → upsampling convolution} × 3.

[0072] When inputting the model, the image vector before upsampling convolution is taken and then processed as the prompt vector.

[0073] 2. Positive prompt branch network construction unit:

[0074] Using high-quality images (I HQ ) as the training target of the first branch to generate high-quality prompts P H .

[0075] 3. Residual noise branch network construction unit: (negative prompt - noise vector)

[0076] Using the difference between low-quality and high-quality images (I LQ -I HQ ) is used as the training target of the second branch to generate noise prompt P R .

[0077] 4. Local Degradation Attention Branch Network Construction Unit: (Negative Prompt - Local Degradation Mask)

[0078] The local degradation mask generated in random noise is used as the training target of the third branch to generate the local degradation prompt P M .

[0079] 5. Text prompt generation unit:

[0080] Using the multimodal image quality assessment model (Image Quality Assessment, IQA), the image content is summarized into a string of text and provided to the model as a text prompt P T .

[0081] 6. Hint fusion and negative hint generation unit:

[0082] Splice {P T , P H} and {P R , P M} and form positive / negative hint embedding p through Transformer Encoder pos and P neg The positive and negative prompt dimensions remain consistent, making it easier for downstream UNet to share the conditional projection layer.

[0083] The functions implemented by this module are different from the single-path prompts of existing technologies. It adopts a "three-way complementary prompt" system, combining text modality and image modality to generate corresponding positive and negative prompts.

[0084] 2. The noise prediction module implements a multimodal CFG reasoning mechanism, and its implementation method further preferably includes:

[0085] 1. Dual forward noise prediction unit

[0086] The low-quality latent space vector Z t With P pos Forward once to get ε pos ;

[0087] Use P neg Then forward once more to get ε neg .

[0088] 2. CFG noise calculation unit

[0089] Calculate the noise prediction ε=ε using the definition of CFG neg +λ cfg (ε pos -ε neg ); Further preferably, λ cfg =3.5.

[0090] The image restoration module is updated in a single step, and its implementation method further preferably includes:

[0091] The DDIM single-step formula is used to output the final latent space vector to obtain the final restored image output.

[0092] The system provided by the above embodiment of the present invention is configured according to the above steps and parameter settings (such as λ cfg By simply setting the hyperparameters of t and t, we can complete the process of obtaining multimodal positive and negative cue embeddings and using these embeddings to obtain the final high-quality image, implement the multimodal CFG guidance strategy of the present invention, and achieve comprehensive performance that is superior to existing methods in various blind restoration / super-resolution scenarios.

[0093] It should be noted that the steps in the method provided by the present invention can be implemented by using the corresponding components in the system. Those skilled in the art can refer to the technical solution of the system to implement the step flow of the method, and can also refer to the technical solution of the method to implement the composition of the system. That is, the embodiments in the system and the embodiments in the method can be understood as preferred examples of each other, and will not be elaborated here.

[0094] An embodiment of the present invention further provides a computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor can be used to execute any one of the methods described in the foregoing embodiments of the present invention, or to execute any one of the systems described in the foregoing embodiments of the present invention.

[0095] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above-mentioned methods), computer instructions, etc., and the above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. In addition, the above-mentioned computer programs, computer instructions, data, etc. can be called by the processor.

[0096] The processor is configured to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0097] The processor and memory can be independent structures or integrated structures. When the processor and memory are independent structures, the memory and processor can be coupled via a bus.

[0098] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, it can be used to execute any method of the above embodiments of the present invention, or to run any system of the above embodiments of the present invention.

[0099] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one location to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC. Alternatively, the ASIC can be located in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0100] The technical solution provided by the above embodiment of the present invention is further described in detail below with reference to a specific application example.

[0101] In this specific application example, the method and system provided by the above embodiment of the present invention are applied to image restoration tasks such as blind image restoration and super-resolution reconstruction. Through the multimodal CFG guidance strategy, the restoration effect of the model under various complex degradation conditions is effectively improved. The specific implementation process includes the following steps:

[0102] Step 1: Training and building a multimodal prompt generation network

[0103] In this specific application example, we first construct a multimodal cue generation network consisting of a backbone network and three branch networks. The backbone network uses a multi-layer residual Swin Transformer Block (RSTB) module stack to form an encoder and decoder structure, and each branch network extracts different modal cue information:

[0104] The positive hint branch uses high-quality images as training targets to extract high-quality hint vectors;

[0105] The residual noise branch uses the residual of high-quality and low-quality images as training targets to extract noise cue vectors;

[0106] The local degradation branch uses the degradation mask as the training target to extract the local degradation hint vector;

[0107] At the same time, a multimodal image quality assessment model is introduced to generate image semantic text prompts.

[0108] During the training phase, the four cues are fused through Transformer encoding to generate positive and negative cue embeddings of uniform dimensions (P pos With P neg ) for downstream diffusion models to call.

[0109] Figure 3 Generate a network graph for a graph vector. Figure 3 In, H Dis the convolution downsampling module, H E It is a feature extraction module composed of two residual SwinTransformer Block modules, H R1 、H R2 、H R3 It is the feature reconstruction module in branches 1, 2, and 3, and is also composed of two residual Swin Transformer Block modules.

[0110] Figure 4 Generate a network diagram for multimodal prompts. Figure 4 As shown in the figure, the high-quality hint vector obtained from the first positive hint branch and the picture description vector obtained from the introduced multimodal picture quality assessment model are combined through a multi-layer perceptron to obtain the positive hint embedding P pos , which can be input into the diffusion model.

[0111] Step 2: Single-step diffusion reasoning guided by multimodal CFG

[0112] After the prompt generation network training is completed, in the blind restoration task, the low-quality image to be processed is first encoded into the latent space vector Z t , and perform CFG double forward reasoning with hint embedding:

[0113] Use P pos Perform a forward diffusion prediction to obtain the forward noise prediction ε pos ;

[0114] Use P neg Perform a forward diffusion prediction to obtain the negative noise prediction ε neg ;

[0115] Calculation formula through CFG: ε=ε neg +λ cfg (ε pos -ε neg )(where λ cfg Generally, 3.5 is selected to fuse the positive and negative prompt information to obtain the final noise prediction result;

[0116] A single-step DDIM sampling formula is used to calculate the final restored high-quality image output.

[0117] Through the implementation of the above-mentioned specific application examples, tests and verifications were carried out in a variety of blind restoration data sets (such as blurred portrait restoration, multi-source degraded image restoration, etc.), and the experimental results of the method and system provided by the above-mentioned embodiment of the present invention in the above-mentioned practical applications showed that: compared with the existing diffusion model method based on fixed negative prompts or single-modal prompts, the present invention has achieved significant improvements in multiple objective evaluation indicators (such as LPIPS, DISTS, NIQE, etc.) and subjective visual effects, especially in high-complexity degradation, texture detail restoration and motion blur suppression. At the same time, the present invention still maintains the efficient and fast reasoning advantage of the single-step diffusion model, and has good industrial application and landing value. Table 1 shows the comparison results of the present invention and other current optimal methods on the public data set PERSONA-Val. As can be seen from the table, the present invention has great advantages over other methods in numerical indicators.

[0118] Table 1

[0119]

[0120] The image restoration method and system based on the single-step diffusion model and multimodal CFG provided by the above embodiments of the present invention utilize the multimodal CFG guidance strategy of the single-step diffusion model to break through the limitations of traditional single-modal fixed negative prompts and achieve: adaptive negative prompt generation: constructing negative prompts in real time based on the actual degradation of the input image, fully suppressing noise, artifacts and local motion blur; multimodal collaboration: combining image features, residual noise vectors and semantic / segmentation information to form complementary guidance signals, thereby improving the model's ability to identify and recover complex degradation; high fidelity: while fully restoring the overall illumination, color and geometric structure of the input image, reconstructing the texture and edges masked by the degradation in a fine-grained manner, avoiding excessive fabrication or texture drift, and ensuring that subjective visual consistency and objective perceptual similarity indicators (such as LPIPS and DISTS) are simultaneously improved.

[0121] Matters not mentioned in the above embodiments of the present invention are well known in the art.

[0122] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various variations or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. An image restoration method based on a single-step diffusion model and multimodal CFG, characterized in that: include: Construct a multimodal prompt generation network based on graph vector generation network; Extracting different modal prompt information based on the multimodal prompt generation network; Fusing the different modal prompt information to generate positive and negative prompt embeddings of unified dimensions; Encode the image to be processed into a latent space vector, and perform CFG double forward reasoning with the positive and negative cue embedding to obtain the final noise prediction result; Based on the single-step diffusion model, the final restored image output is calculated using the final noise prediction result.

2. The image restoration method based on the single-step diffusion model and multimodal CFG according to claim 1, characterized in that: Construct a multimodal prompt generation network based on a graph vector generation network, including: A graph vector generation network is provided, comprising a backbone network and three branch networks. The backbone network and the three branch networks each employ a multi-layer stacked residual Swin Transformer Block module as their main structure. The branch networks are trained as follows to construct a multimodal cue generation network: Using high-quality images as training targets, we obtain a positive-prompt branch network. The difference between low-quality and high-quality images is used as the training target to obtain the residual noise branch network; The locally degraded segmentation mask generated by random noise is used as the training target to obtain a locally degraded branch network.

3. The image restoration method based on the single-step diffusion model and multimodal CFG according to claim 1, characterized in that: Based on the multimodal prompt generation network, different modal prompt information is extracted respectively, including: Based on the branch network of the multimodal prompt generation network, the following prompt information is extracted respectively: Extract high-quality prompt vectors through the positive prompt branch network of the multimodal prompt generation network to obtain positive prompt vectors; The noise cue vector is extracted through the residual noise branch network of the multimodal cue generation network to obtain the noise vector of the negative cue; The local degradation cue vector is extracted through the local degradation branch network of the multimodal cue generation network to obtain the local degradation vector of the negative cue; Introduce the image quality assessment model to generate image semantic text prompt information.

4. The image restoration method based on the single-step diffusion model and multimodal CFG according to claim 1, characterized in that: The different modal prompt information is fused to generate a positive and negative prompt embedding of a unified dimension, including: Splice {P T , P H } and {P R , P M } and form a dimensionally consistent positive prompt embedding P through Transformer Encoder pos and negative hint embedding P neg Among them, P T For text prompts, P H For positive prompts, P R For noise prompt, P M This indicates local degradation.

5. The image restoration method based on the single-step diffusion model and multimodal CFG according to claim 1, characterized in that: The image to be processed is encoded into a latent space vector, and CFG double forward reasoning is performed in conjunction with the positive and negative hint embedding to obtain the final noise prediction result, including: Encode the low-resolution image to be processed into the latent space vector Z t ; The latent space vector Z t Embed P with positive prompt pos Perform a forward diffusion prediction to obtain the forward noise prediction ε pos ; The latent space vector Z t Embed P with negative prompt neg Perform a forward diffusion prediction to obtain the negative noise prediction ε neg ; By fusing the positive noise prediction and the negative noise prediction, the final noise prediction result ε is: e=e neg +λ cfg (e pos -e neg ); Where λ cfg is a predetermined hyperparameter used to balance the strength of positive noise and negative noise.

6. The image restoration method based on the single-step diffusion model and multimodal CFG according to claim 1, characterized in that: Based on the single-step diffusion model, the final restored image output is calculated using the final noise prediction result, including: The DDIM single-step diffusion model is used to output the final latent space vector to obtain the required high-resolution image, which is expressed as: Where, is the latent space vector corresponding to the predicted high-quality image, z LQ is the latent space vector corresponding to the input low-quality image, is a constant related to the number of time steps t taken, t is the time step hyperparameter passed into the DDIM single-step diffusion model, and ε is the final noise prediction result.

7. An image restoration system based on a single-step diffusion model and multimodal CFG, characterized in that: include: A multimodal prompt information generation module, which is used to build a multimodal prompt generation network based on a graph vector generation network; Based on the multimodal prompt generation network, different modal prompt information is extracted respectively; the different modal prompt information is fused to generate positive and negative prompt embeddings of unified dimensions; A noise prediction module is used to encode the image to be processed into a latent space vector and perform CFG double forward reasoning in conjunction with the positive and negative cue embedding to obtain the final noise prediction result; The image restoration module is based on a single-step diffusion model and uses the final noise prediction result to calculate the final restored image output.

8. A computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When executing the computer program, the processor can be used to perform the method according to any one of claims 1 to 6, or run the system according to claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can be used to perform the method according to any one of claims 1 to 6, or to run the system according to claim 7.

Images