Image restoration methods, systems, and devices based on multimodal large model-driven approaches.

Abstract

This invention belongs to the field of digital image processing technology and discloses an image restoration method, system, and device based on a multimodal large model. The invention includes receiving a degraded image to be restored; inputting the degraded image and preset multi-task text prompts into a multimodal large model; analyzing and processing the degraded image to generate operation instructions and visual description prompts; encoding the operation instructions and visual description prompts to obtain corresponding task intent vectors and content guidance vectors; fusing the task intent vectors and content guidance vectors through an image restoration model; reconstructing the degraded image; and outputting the restored image. This invention can automatically and accurately diagnose image problems and provide rich guidance information from both operation and content dimensions, achieving intelligent, automated, and high-fidelity image restoration.

Description

Technical Field

[0001] This invention belongs to the field of digital image processing technology, specifically relating to an image restoration method, system, and device based on multimodal large model-driven processing. Background Technology

[0002] Digital images inevitably suffer quality degradation during acquisition, transmission, and storage due to various factors. Common types of degradation include Gaussian noise caused by sensor limitations, motion blur caused by camera or object movement, optical haze caused by atmospheric particulate scattering, and rain streaks, raindrops, or snowflakes formed in inclement weather. Furthermore, unfavorable lighting conditions can lead to underexposure or limited dynamic range. The goal of image restoration techniques is to reconstruct a potentially high-quality, clear version of these degraded images.

[0003] With the rapid development of deep learning technology, especially convolutional neural networks (CNNs) and the Transformer architecture, significant progress has been made in the field of image restoration. However, current technological approaches mainly face the following key challenges:

[0004] Task-specific models: These models are designed for a single type of degradation (such as denoising or deblurring) and typically achieve high performance. However, their drawback is a lack of generalization ability, making them unable to handle the diverse degradation problems that are prevalent in the real world.

[0005] "Integrated" blind recovery models: To overcome the problem of model specialization, researchers have developed models capable of handling multiple degradations within a single network. However, these models are mostly "blind" recovery models, meaning they autonomously judge and repair without explicit external guidance. This approach leads to an uncontrollable repair process, and when multiple degradations coexist, the model is prone to producing compromise, suboptimal repair results, making it difficult to meet users' personalized needs.

[0006] Preliminary instruction-guided recovery: Recent attempts have explored using natural language instructions to guide the recovery process, allowing users to specify the recovery task via text (such as "deblur"), which improves the model's controllability and flexibility to some extent. However, this paradigm still suffers from two core bottlenecks:

[0007] A) Reliance on user expertise: It requires users to accurately identify and verbally describe the type of image degradation. Non-expert users may be unable to provide precise instructions, thus affecting the restoration results. For batch automated processing scenarios, manually entering instructions is even more impractical.

[0008] B) Limited guidance information: The limited operational instructions address the "how-to-restore" question, but lack guidance on the "what-to-be" question. Especially when image information is severely lost, the model lacks prior knowledge of the ideal scene content, easily producing mediocre results that do not conform to real-world logic or lack detail.

[0009] In conclusion, the current field of image restoration urgently needs a new technical solution that can automatically and accurately diagnose image problems and provide rich guidance information from both operational and content dimensions, thereby achieving truly intelligent, automated, and high-fidelity image restoration. Summary of the Invention

[0010] The purpose of this invention is to address the shortcomings of existing technologies by providing an image restoration method, system, and device based on multimodal large model-driven technology. This invention can solve the core technical problems of existing image restoration technologies, such as strong reliance on user expertise, limited guidance information dimensions, low automation, and poor restoration results under severe degradation.

[0011] Specifically, the present invention is implemented using the following technical solutions.

[0012] On one hand, the present invention provides an image restoration method based on multimodal large model driven by the present invention, comprising:

[0013] A degraded image to be restored is received. The degraded image and a preset multi-task text prompt are input into a multimodal large model. The multimodal large model analyzes and processes the degraded image according to the preset multi-task text prompt, and generates operation instructions and visual description prompts for the degraded image.

[0014] The operation instructions and visual description prompts are encoded to obtain corresponding task intent vectors and content guidance vectors, respectively.

[0015] The degraded image, task intent vector, and content guidance vector are input into the image restoration model. The image restoration model fuses the task intent vector and content guidance vector, reconstructs the degraded image, and outputs the restored image.

[0016] The image restoration model's network backbone includes an encoder, a bottleneck layer, and a decoder. Both the encoder and decoder are composed of stacked Transformer blocks at several levels. The network backbone also integrates an ICB module and a CAFM module. The ICB module is deployed at each level of the encoder and decoder, dynamically modulating the image features of the current level with channel attention based on the task intent vector, and dynamically weighting the image features at each level to achieve task routing. The CAFM module is deployed at the bottleneck layer of the image restoration model's network backbone, using the content guidance vector and a cross-attention mechanism to inject corresponding scene content into the image features to achieve scene content guidance.

[0017] Furthermore, the specific calculation process for dynamically modulating the image features of the current level based on the task intent vector and dynamically weighting the image features of each level is shown in formulas (1) and (2):

[0018] (1),

[0019] (2),

[0020] in, This is the task intent vector. Let σ be a learnable linear layer within the ICB module, where σ is the Sigmoid activation function. For the generated channel attention mask, This represents element-wise multiplication. For image features at each level, This refers to the modulated output characteristics.

[0021] Furthermore, in the CAFM module, the image features of the bottleneck layer serve as the query, while the content-guided vector serves as both the key and the value.

[0022] Furthermore, the overall structure of the preset multi-task text prompt includes a global format contract and two parallel sub-task specifications;

[0023] The global format contract is located at the beginning of the multitasking text prompt, establishes the overall task, and specifies that the final output is a JSON object containing the key names—command and description;

[0024] The two parallel subtask specifications follow the global format contract and define in detail how each key-value pair in the JSON object should generate the corresponding operation instruction or content description. The first subtask specification defines the value corresponding to the key name "command" to guide the multimodal large model to generate the operation instruction, and the second subtask specification defines the value corresponding to the key name "description" to guide the multimodal large model to generate the visual description prompt.

[0025] Furthermore, the encoding of the operation instructions and visual description prompts to obtain corresponding task intent vectors and content guidance vectors includes:

[0026] The operation command is input to the first text encoder, which is a pre-trained language model specializing in sentence-level semantic similarity understanding. The first text encoder compresses the entire operation command text into a fixed-dimensional vector that can represent its core intent. The fixed-dimensional vector is then subjected to an additional linear projection layer for dimension adaptation, and finally a fixed-dimensional task intent vector is generated to represent the function of the recovery task to be performed.

[0027] The visual description prompt is input into a second text encoder, which is a pre-trained language model with visual-language cross-modal alignment capability. The second text encoder transforms the visual description prompt text into a sequence containing lexical information, which is then output as a content guidance vector.

[0028] Further, the process of inputting the degraded image, task intent vector, and content guidance vector into the image restoration model, fusing the task intent vector and content guidance vector, reconstructing the degraded image using the image restoration model, and outputting the restored image includes:

[0029] 3-1) Input the degraded image into the encoder of the image restoration model;

[0030] 3-2) Guided downsampling: During the encoder's step-by-step downsampling and feature extraction process, the ICB module of each layer is activated, and the image features are modulated using the task intent vector to guide the image restoration model to focus on the restoration task specified by the operation instruction.

[0031] 3-3) The image features obtained through the guided downsampling reach the bottleneck layer of the image restoration model;

[0032] 3-4) Content Enhancement: The CAFM module deployed in the bottleneck layer is activated. Using the content-guided vector, scene content is injected into image features through a cross-attention mechanism to obtain content-enhanced features.

[0033] 3-5) Feature upsampling: The content-enhanced features are fed into the decoder of the image restoration model for feature upsampling;

[0034] 3-6) Guided Reconstruction: During the process of feature upsampling and image detail reconstruction performed by the decoder at each level, the ICB module of each layer is activated and continuously guided by the task intent vector to ensure that the reconstruction process strictly follows the operation instructions.

[0035] The decoder outputs the recovered image at the last layer (3-7).

[0036] On the other hand, the present invention also provides a training method for an image restoration model based on a multimodal large model, used to train the image restoration model used in the aforementioned image restoration method based on a multimodal large model, comprising:

[0037] The system receives degraded images to be restored from the training set, and inputs each degraded image and a multi-task text prompt into the multimodal large model. It pre-generates corresponding operation instructions and visual description prompts for the degraded image offline and stores them as annotation information. The training set contains pairs of degraded images and clear images.

[0038] The operation instructions and visual description prompts are encoded to obtain task intent vector and content guidance vector, respectively;

[0039] The degraded image, task intent vector, and content guidance vector are input into the image restoration model. The task intent vector and content guidance vector are fused to reconstruct the degraded image, and the restored image is output. The parameters of the first and second text encoders of the image restoration model are kept frozen, and only the parameters of the image restoration model and the additional linear projection layer are optimized.

[0040] Based on the clear image corresponding to the output restored image and the degraded image, pixel loss, perceptual loss and cross-entropy loss are calculated, and a composite loss function is formed by weighting the pixel loss, perceptual loss and cross-entropy loss.

[0041] Based on the composite loss function, the image restoration model is trained using training samples containing the paired degraded and clear images, thereby optimizing the parameters of the image restoration model and the linear projection layer.

[0042] Furthermore, using The optimizer is trained using a cosine annealing learning rate scheduling strategy with preheating until the image restoration model converges.

[0043] Furthermore, this invention also provides an image restoration system based on a multimodal large model, implementing the aforementioned image restoration method based on a multimodal large model, including a text prompt generation module, a text prompt encoding module, and an image restoration model:

[0044] The text prompt generation module includes preset multi-task text prompt words, a multimodal large model, and a JSON parser. The module receives a degraded image and uses the multimodal large model to generate operation instructions defining the restoration operation and visual description prompts depicting the ideal scene. The multi-task text prompt words establish the overall image restoration task through a global format contract and specify that the output of the multimodal large model is a JSON object containing the key names `command` and `description`. Two sub-task specifications guide the multimodal large model to analyze the input degraded image, outputting a JSON format string containing the analysis results. The JSON parser extracts the values ​​corresponding to the key names `command` and `description` from the JSON format string, defining them as operation instructions and visual description prompts, respectively.

[0045] The text prompt encoding module encodes the operation instructions and visual description prompts to obtain corresponding task intent vectors and content guidance vectors, respectively.

[0046] The image restoration model receives the degraded image, the task intent vector, and the content guidance, and performs pixel-level repair and enhancement operations on the degraded image based on the task intent vector and the content guidance, and finally outputs the restored image.

[0047] In another aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described image restoration method based on multimodal large model driving.

[0048] The beneficial effects of the image restoration method, system, and device based on multimodal large model-driven architecture of the present invention are as follows:

[0049] The present invention relates to an image restoration method, system, and device driven by a multimodal large model, which constructs a fully automated closed loop for instruction generation and execution. First, a powerful multimodal large model is used to perform "expert-level diagnosis" on the input degraded image, automatically generating two types of decoupled text instructions: one is an "operation instruction" defining the restoration operation, and the other is a "visual description prompt" depicting the ideal content. Subsequently, these two instructions are injected into different layers of an image restoration network through different encoding and fusion mechanisms, respectively achieving "task routing" for network behavior and "semantic guidance" for content reconstruction, thereby collaboratively completing high-quality image restoration.

[0050] To address the issue of existing instruction-guided image restoration technologies heavily relying on manual diagnosis and input, resulting in low automation and high barriers to entry, this invention presents an image restoration method, system, and device driven by a multimodal large model. This method employs a fully automated dual-path instruction generation technology based on a multimodal large model. Specifically, by designing an integrated multi-task text prompt, a powerful multimodal large model is driven to autonomously analyze the input image, thereby generating "operation instructions" defining the restoration operation in a single, automated process. This achieves "expert-level" automatic diagnosis and restoration instruction generation for image degradation types, completely eliminating reliance on user expertise and manual operation. This allows even non-professional users to easily utilize the system and provides core technical support for large-scale, automated image processing workflows, significantly improving the system's practicality and application scope.

[0051] To address the problem that existing image restoration technologies only provide single operational guidance on "how to restore," resulting in blurred details and mediocre performance when processing severely degraded images due to a lack of content priors, this invention proposes a novel dual-path semantic guidance framework based on a multimodal large model. This framework automatically generates image restoration operation instructions while simultaneously generating "visual description cues" depicting the ideal scene content. These cues are then injected into the bottleneck layer of the image restoration network through a CAFM (Cross-Attention Fusion Module), successfully supplementing the network with high-level semantic and stylistic guidance on "what should be restored." This enables the image restoration network to generate realistic, textured, and detailed content based on rich textual descriptions when reconstructing areas with severely missing information, thereby significantly improving the fidelity and visual upper limit of image restoration, especially in extreme scenarios such as severe blurring and low light.

[0052] To address the issues of feature interference and lack of fine-grained control in the restoration process arising from multi-task processing in existing "integrated" models, this invention presents an image restoration method, system, and device driven by a multimodal large model, employing a functionally decoupled heterogeneous fusion network architecture. This architecture injects two different types of guidance information (task intent vector and content guidance vector) at different strategic locations within the network through different fusion mechanisms (ICB module and CAFM module), achieving hierarchical and precise control over the "task routing" and "content guidance" of the image restoration process. The ICB module adapts to the current operational task at each layer of the encoder and decoder, while the CAFM module injects content at the bottleneck layer. This decoupled design effectively avoids negative interference between different restoration targets, improves the model's execution efficiency and the accuracy of the restoration results, and enables the system to handle complex hybrid degradation problems in the real world more efficiently and robustly. Attached Figure Description

[0053] Figure 1 This is a system schematic diagram according to an embodiment of the present invention.

[0054] Figure 2 This is a structural diagram of the image restoration model according to an embodiment of the present invention.

[0055] Figure 3 This is a schematic diagram of the method flow according to an embodiment of the present invention. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to the embodiments and the accompanying drawings.

[0057] One embodiment of the present invention is an image restoration system based on multimodal large model driven, such as... Figure 1 As shown, it includes a text prompt generation module, a text prompt encoding module, and an image restoration model.

[0058] The text prompt generation module serves as the intelligent decision-making front-end and task planning center of the image restoration system. It receives a single degraded image and uses a multimodal large model to generate two completely different but complementary text instructions: "operation instructions" defining the restoration operation and "visual description prompts" depicting the ideal scene, providing comprehensive and intelligent guidance for the subsequent image restoration process.

[0059] The text prompt generation module includes preset multi-task text prompts, a multimodal large model, and a JSON parser. The multi-task text prompts establish the overall image restoration task through a global format contract and specify that the output of the multimodal large model is a JSON (JavaScript Object Notation) object containing specific key names—command and description. Two completely separate sub-task specifications guide the multimodal large model to analyze the input degraded image, outputting a JSON string containing the analysis results. The JSON parser extracts the values ​​corresponding to the key names from the JSON string, defining them as operation instructions and visual description prompts, respectively.

[0060] The text prompt encoding module encodes the operation instructions and visual description prompts, and converts them from natural language text format into numerical, high-dimensional vector representations that the image restoration model can understand and calculate, respectively, to obtain the corresponding task intent vector and content guidance vector.

[0061] The image restoration model is the main entity that performs specific image restoration tasks. After the degraded image, task intent vector, and content guidance vector are input into the image restoration model, the model fuses the task intent vector and content guidance vector, performs pixel-level repair and enhancement operations on the degraded image, and finally outputs a high-quality restored image.

[0062] like Figure 2 As shown, the backbone of the image restoration model is a high-performance image restoration network with an encoder-decoder structure, including an encoder, a bottleneck layer, and a decoder. Both the encoder and decoder are composed of several layers of stacked Transformer blocks, used to extract and reconstruct image features step by step. The backbone of the image restoration model also integrates two different fusion modules: the ICB module (Instruction Condition Block) and the CAFM module (Cross-Attention Fusion Module).

[0063] The ICB module is deployed at each level of the encoder and decoder in the backbone of the image restoration model network. Based on the received task intent vector, the ICB module dynamically modulates the image features of the current level using channel attention modulation and dynamically weights the image features of each level to achieve task routing. The specific calculation process is shown in formulas (1) and (2):

[0064] (1),

[0065] (2),

[0066] in, This is the task intent vector. Let σ be a learnable linear layer within the ICB module, where σ is the Sigmoid activation function. For the generated channel attention mask, This represents element-wise multiplication. For image features at each level, The output features are modulated. In this way, the image restoration model can selectively activate or suppress specific feature channels according to the operation instructions to adapt to different image restoration tasks.

[0067] The CAFM module is deployed in the bottleneck layer of the image restoration model's network backbone, where semantic information is richest. The CAFM module utilizes content-guided vectors (CVMs) and a cross-attention mechanism to inject relevant scene content into image features, thus achieving scene content guidance. In the CAFM module, the image features of the bottleneck layer act as the query, while the CVMs serve as both the key and value. This allows image features to proactively query and aggregate relevant content and style information from visual descriptive cues and textual descriptions, thereby guiding the image restoration model to generate more reasonable and richer details when image information is severely lacking.

[0068] The image restoration method based on multimodal large model driven by the present invention, such as Figure 3 As shown, it includes the following steps:

[0069] 1. Receive a degraded image to be restored, input the degraded image and a preset multi-task text prompt into a multimodal large model, the multimodal large model analyzes and processes the degraded image according to the preset multi-task text prompt, and generates operation instructions and visual description prompts for the degraded image.

[0070] The multimodal large model generates structured data containing operational instructions and visual descriptive cues in a single, automated process. The multimodal large model can be an existing multimodal large model with visual understanding and text generation capabilities, such as the GPT-4o model with advanced visual language understanding capabilities.

[0071] The core objective of this step is to leverage the intelligent analysis capabilities of a multimodal large model to comprehensively "diagnose" and "conceptualize" the input degraded image, and automatically generate decoupled text prompts for the two functions. Specifically, this includes:

[0072] 1-1) Input reception

[0073] It receives a degraded image of arbitrary size to be restored as initial input.

[0074] 1-2) Prompts for building and committing

[0075] The image restoration system has pre-set structured, integrated multi-task text prompts. These prompts are designed to guide a pre-trained multimodal large model to simultaneously perform two tasks: image quality diagnosis and ideal scene content description.

[0076] The input degraded image and the preset multi-task text prompt are submitted together to the multimodal large model. The multi-task text prompt used in this invention is structurally designed as a functionally decoupled, structured set of multi-task instructions. Its overall structure includes a global format contract and two parallel sub-task specifications. The global format contract, located at the beginning of the multi-task text prompt, establishes the overall task and mandates that the final output must be a JSON (JavaScript Object Notation) object containing specific key names—command and description. The global format contract is the structural foundation for ensuring machine readability and automated processing. The two parallel sub-task specifications follow the global format contract, defining in detail how each key in the JSON object should generate the corresponding operation instruction or content description. The first sub-task specification defines the value corresponding to the key name "command," guiding the multimodal large model to generate the "operation instruction" (command), while the second sub-task specification defines the value corresponding to the key name "description," guiding the multimodal large model to generate the "visual description prompt" (description). These two sub-task specifications are functionally completely separate, ensuring the purity of both operational instructions and visual descriptive cues—the two types of semantic guidance information. Below is an example of the specific content of a multi-task text prompt:

[0077] Analyze the provided image. Perform two tasks and format the output as a JSON object with two keys: 'command' and 'description'. (This is a global formatting contract.)

[0078] For 'command': A concise, actionable command to eliminate thedegradation and restore the image to its clear state. Use simple verbs (eg,"remove," "eliminate," "clear") and specify the degradation type identified in the image like rain, fog, snow. (This is the first subtask specification)

[0079] For 'description': Ignoring degradation phenomena like rain, fog, and snow, describe what the content should look like under normal conditions with vivid, artistic detail. (This is the second sub-task specification)

[0080] 1-3) Response parsing and instruction dispatch

[0081] The multimodal large model analyzes and processes the input degraded image based on preset multi-task text prompts, and returns a JSON string containing the analysis results. The JSON parser within the text prompt generation module automatically performs the following operations on this string to obtain the "operation instructions" and "visual description prompts":

[0082] The value corresponding to the key name "command" is precisely extracted and defined as an "operation instruction." The operation instruction specifies the recovery task to be performed. For example, the operation instruction could be "Remove the rain streaks."

[0083] The values ​​corresponding to the key name "description" are precisely extracted and defined as "visual description cues." Visual description cues depict the ideal scene content and style without degradation. For example, a visual description cue could be: "A grand, modern white arena stands proudly against a backdrop of blue sky and drifting clouds. Tall palm trees line the street, while cars move steadily along the road in front. The sleek architecture and lively urban setting suggest a place of energy and gathering, where the structure serves as both a landmark and a beacon of activity."

[0084] Second, the operation instructions and visual description prompts are encoded to obtain the corresponding task intent vector and content guidance vector, respectively.

[0085] This step aims to transform the two types of plain text prompts (operation instructions and visual description prompts) generated in Step 1 into numerical tensors containing rich semantic information that can be understood and processed by deep learning networks. This invention employs a heterogeneous encoding scheme, using different encoders for different textual indications. Specifically, it includes:

[0086] Encode the operation instructions:

[0087] The "operation instructions" are input into the first text encoder. The first text encoder is a pre-trained language model specializing in sentence-level semantic similarity understanding. Typical examples of this type of model include DistilBERT-base-uncased, which is based on the BERT architecture and fine-tuned for sentence embedding tasks. Its core technical characteristic is that it is trained to map complete sentences with similar meanings but different expressions to neighboring positions in a high-dimensional vector space. The first text encoder compresses the entire operation instruction text into a fixed-dimensional vector that represents its core intent. This fixed-dimensional vector is then passed through an additional linear projection layer for dimension adaptation, ultimately generating a fixed-dimensional (e.g., 256-dimensional) "task intent vector" to represent the function of the recovery task to be performed.

[0088] Encode visual description cues:

[0089] The visual description cues are input into the second text encoder. The second text encoder is a pre-trained language model with cross-modal visual-language alignment capabilities. A typical example of this type of model is the OpenAI CLIP ViT-L / 14 text encoder. Its core technical feature is that during pre-training, it utilizes massive amounts of image-text data pairs for comparative learning, thereby constructing a shared multimodal embedding space. In this space, the vector representation of a piece of text (e.g., "a beach under the sun") is spatially highly aligned with the image vector representation of the visual content described by that text. The second text encoder transforms the visual description cues into a sequence containing rich token information, outputting a "content-guided vector" representing the scene content, such as a "content-guided vector" of shape [1, 77, 768], which preserves detailed descriptions of scene content, object details, and artistic style.

[0090] III. Image Feature Extraction and Image Restoration

[0091] This step inputs dual semantic guidance information (i.e., task intent vector and content guidance vector) consisting of numerical tensors of the degraded image, operation instructions, and visual description prompts into the image restoration model. The image restoration model then fuses the task intent vector and content guidance vector to reconstruct the degraded image and output a high-quality restored image.

[0092] The CAFM module, located in the bottleneck layer of the image restoration model's backbone, utilizes content-guided vectors and a cross-attention mechanism to inject prior knowledge of scene content (i.e., encoded visual descriptive cues) into image features, thereby achieving scene content guidance. In the CAFM module, the image features in the bottleneck layer act as the query, while the content-guided vectors serve as both the key and value. This allows the image features to proactively query and aggregate relevant content and style information from the visual descriptive cues, thus guiding the image restoration model to generate more reasonable and richer details even when image information is severely lacking.

[0093] The image restoration model is fed with dual semantic guidance information (i.e., task intent vector and content guidance vector) consisting of numerical tensors of the degraded image, operation instructions, and visual description cues. The model then fuses the task intent vector and content guidance vector to reconstruct the degraded image and output a high-quality restored image. The detailed steps include the following:

[0094] 3-1) Input the degraded image into the encoder of the image restoration model.

[0095] 3-2) Guided downsampling: During the encoder's step-by-step downsampling and feature extraction process, the ICB module of each layer is activated. The image features are modulated using the task intent vector obtained in step two, guiding the image restoration model to focus on the restoration task specified by the operation instructions.

[0096] 3-3) The image features obtained through the guided downsampling reach the bottleneck layer of the image restoration model.

[0097] 3-4) Content Enhancement: The CAFM module deployed in the bottleneck layer is activated. Using the content guidance vector obtained in step 2, the scene content is injected into the image features through the cross-attention mechanism to obtain content-enhanced features.

[0098] 3-5) Feature upsampling: The content-enhanced features are fed into the decoder of the image restoration model for feature upsampling.

[0099] 3-6) Guided Reconstruction: During the process of feature upsampling and image detail reconstruction in the decoder, the ICB module of each layer is also activated, continuously using the task intent vector obtained in step 2 to guide the reconstruction process and ensure that the reconstruction process strictly follows the operation instructions.

[0100] 3-7) The final layer output of the decoder restores the image.

[0101] Preferably, in another embodiment, to achieve ideal performance of the image restoration system of the present invention, supervised training of the image restoration model is required. This includes:

[0102] Receive degraded images to be restored from the training set, which contains a large-scale, diverse, paired "degraded image-clear image" set. Input each degraded image and a multi-task text prompt word into the multimodal large model. Pre-generate corresponding operation instructions and visual description prompts for the degraded image offline and store them as annotation information.

[0103] The operation instructions and visual description prompts are encoded to obtain task intent vector and content guidance vector, respectively;

[0104] The degraded image, task intent vector, and content guidance vector are input into the image restoration model. The task intent vector and content guidance vector are fused to reconstruct the degraded image and output the restored image. The parameters of the first and second text encoders of the image restoration model are kept frozen, and only the parameters of the image restoration model and the additional linear projection layer are optimized.

[0105] Based on the clear image corresponding to the output restored image and the degraded image, pixel loss (such as L1 loss), perceptual loss (such as LPIPS) and cross-entropy loss are calculated, and a composite loss function is formed by weighting the pixel loss, perceptual loss and cross-entropy loss.

[0106] Based on the composite loss function, the image restoration model is trained using training samples containing the paired "degraded image - clear image" pairs, thereby optimizing the parameters of the image restoration model and the linear projection layer.

[0107] Preferably, the AdamW optimizer is used, along with a cosine annealing learning rate scheduling strategy with preheating, for training until the image restoration model converges.

[0108] The present invention relates to an image restoration method, system, and device driven by a multimodal large model, which constructs a fully automated closed loop for instruction generation and execution. First, a powerful multimodal large model is used to perform "expert-level diagnosis" on the input degraded image, automatically generating two types of decoupled text instructions: one is an "operation instruction" defining the restoration operation, and the other is a "visual description prompt" depicting the ideal content. Subsequently, these two instructions are injected into different layers of an image restoration network through different encoding and fusion mechanisms, respectively achieving "task routing" for network behavior and "semantic guidance" for content reconstruction, thereby collaboratively completing high-quality image restoration.

[0109] To address the issue of existing instruction-guided image restoration technologies heavily relying on manual diagnosis and input, resulting in low automation and high barriers to entry, this invention presents an image restoration method, system, and device driven by a multimodal large model. This method employs a fully automated dual-path instruction generation technology based on a multimodal large model. Specifically, by designing an integrated multi-task text prompt, a powerful multimodal large model is driven to autonomously analyze the input image, thereby generating "operation instructions" defining the restoration operation in a single, automated process. This achieves "expert-level" automatic diagnosis and restoration instruction generation for image degradation types, completely eliminating reliance on user expertise and manual operation. This allows even non-professional users to easily utilize the system and provides core technical support for large-scale, automated image processing workflows, significantly improving the system's practicality and application scope.

[0110] To address the problem that existing image restoration technologies only provide single operational guidance on "how to restore," resulting in blurred details and mediocre performance when processing severely degraded images due to a lack of content priors, this invention proposes a novel dual-path semantic guidance framework based on a multimodal large model. This framework automatically generates image restoration operation instructions while simultaneously generating "visual description cues" depicting the ideal scene content. These cues are then injected into the bottleneck layer of the image restoration network through a CAFM (Cross-Attention Fusion Module), successfully supplementing the network with high-level semantic and stylistic guidance on "what should be restored." This enables the image restoration network to generate realistic, textured, and detailed content based on rich textual descriptions when reconstructing areas with severely missing information, thereby significantly improving the fidelity and visual upper limit of image restoration, especially in extreme scenarios such as severe blurring and low light.

[0111] To address the issues of feature interference and lack of fine-grained control in the restoration process arising from multi-task processing in existing "integrated" models, this invention presents an image restoration method, system, and device driven by a multimodal large model, employing a functionally decoupled heterogeneous fusion network architecture. This architecture injects two different types of guidance information (task intent vector and content guidance vector) at different strategic locations within the network through different fusion mechanisms (ICB module and CAFM module), achieving hierarchical and precise control over the "task routing" and "content guidance" of the image restoration process. The ICB module adapts to the current operational task at each layer of the encoder and decoder, while the CAFM module injects content at the bottleneck layer. This decoupled design effectively avoids negative interference between different restoration targets, improves the model's execution efficiency and the accuracy of the restoration results, and enables the system to handle complex hybrid degradation problems in the real world more efficiently and robustly.

[0112] In some embodiments, certain aspects of the above-described techniques may be implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly implemented on a non-transitory computer-readable storage medium. The software may include instructions and certain data that, when executed by one or more processors, manipulate one or more processors to perform one or more aspects of the above-described techniques. The non-transitory computer-readable storage medium may include, for example, magnetic or optical disk storage devices, solid-state storage devices such as flash memory, cache, random access memory (RAM), or other non-volatile memory devices. The executable instructions stored on the non-transitory computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats interpreted or otherwise executed by one or more processors.

[0113] Computer-readable storage media can include any storage medium or combination of storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media can include, but are not limited to, optical media (e.g., optical discs (CDs), digital versatile optical discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tapes, or magnetic hard disks), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical systems (MEMS) based storage media. Computer-readable storage media can be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., a magnetic hard disk drive), removably attached to a computing system (e.g., an optical disc or universal serial bus-based (USB) flash memory), or coupled to a computer system via a wired or wireless network (e.g., network-accessible storage (NAS)).

Claims

1. An image restoration method based on multimodal large model-driven approach, characterized in that, include: The system receives a degraded image to be restored, inputs the degraded image and a preset multi-task text prompt into a multimodal large model, and the multimodal large model analyzes and processes the degraded image according to the preset multi-task text prompt to generate operation instructions and visual description prompts for the degraded image. The operation instructions and visual description prompts are encoded to obtain corresponding task intent vectors and content guidance vectors, respectively. The degraded image, task intent vector, and content guidance vector are input into the image restoration model. The image restoration model fuses the task intent vector and content guidance vector, reconstructs the degraded image, and outputs the restored image. The image restoration model's network backbone includes an encoder, a bottleneck layer, and a decoder. Both the encoder and decoder are composed of stacked Transformer blocks at several levels. The network backbone also integrates an ICB module and a CAFM module. The ICB module is deployed at each level of the encoder and decoder, dynamically modulating the image features of the current level with channel attention based on the task intent vector, and dynamically weighting the image features at each level to achieve task routing. The CAFM module is deployed at the bottleneck layer of the image restoration model's network backbone, using the content guidance vector and a cross-attention mechanism to inject corresponding scene content into the image features to achieve scene content guidance. The process of encoding the operation instructions and visual description prompts to obtain corresponding task intent vectors and content guidance vectors includes: The operation command is input to the first text encoder, which is a pre-trained language model specializing in sentence-level semantic similarity understanding. The first text encoder compresses the entire operation command text into a fixed-dimensional vector that can represent its core intent. The fixed-dimensional vector is then subjected to an additional linear projection layer for dimension adaptation, and finally a fixed-dimensional task intent vector is generated to represent the function of the recovery task to be performed. The visual description prompt is input into a second text encoder, which is a pre-trained language model with visual-language cross-modal alignment capability. The second text encoder transforms the visual description prompt text into a sequence containing lexical information, which is then output as a content guidance vector.

2. The image restoration method based on multimodal large model-driven as described in claim 1, characterized in that, The specific calculation process for dynamically modulating the image features of the current level based on the task intent vector and dynamically weighting the image features of each level is detailed in Formulas 1 and 2. （1）， （2）， in, This is the task intent vector. Let σ be a learnable linear layer within the ICB module, where σ is the Sigmoid activation function. For the generated channel attention mask, This represents element-wise multiplication. For image features at each level, This refers to the modulated output characteristics.

3. The image restoration method based on multimodal large model-driven model according to claim 1, characterized in that, In the CAFM module, the image features of the bottleneck layer serve as the query, while the content-guided vector serves as both the key and the value.

4. The image restoration method based on multimodal large model-driven model according to claim 1, characterized in that, The overall structure of the preset multi-task text prompt includes a global format contract and two parallel sub-task specifications. The global format contract is located at the beginning of the multitasking text prompt, establishes the overall task, and specifies that the final output is a JSON object containing the key name command and description; The two parallel subtask specifications follow the global format contract and define in detail how each key-value pair in the JSON object should generate the corresponding operation instruction or content description. The first subtask specification defines the value corresponding to the key name "command" to guide the multimodal large model to generate the operation instruction, and the second subtask specification defines the value corresponding to the key name "description" to guide the multimodal large model to generate the visual description prompt.

5. The image restoration method based on multimodal large model-driven model according to claim 1, characterized in that, The process involves inputting the degraded image, task intent vector, and content guidance vector into an image restoration model. After fusing the task intent vector and content guidance vector, the image restoration model reconstructs the degraded image and outputs the restored image, including: 3-1) Input the degraded image into the encoder of the image restoration model; 3-2) Guided downsampling: During the encoder's step-by-step downsampling and feature extraction process, the ICB module of each layer is activated, and the image features are modulated using the task intent vector to guide the image restoration model to focus on the restoration task specified by the operation instruction. 3-3) The image features obtained through the guided downsampling reach the bottleneck layer of the image restoration model; 3-4) Content Enhancement: The CAFM module deployed in the bottleneck layer is activated. Using the content-guided vector, scene content is injected into image features through a cross-attention mechanism to obtain content-enhanced features. 3-5) Feature upsampling: The content-enhanced features are fed into the decoder of the image restoration model for feature upsampling; 3-6) Guided Reconstruction: During the process of feature upsampling and image detail reconstruction performed by the decoder at each level, the ICB module of each layer is activated and continuously guided by the task intent vector to ensure that the reconstruction process strictly follows the operation instructions. The decoder outputs the recovered image at the last layer (3-7).

6. A training method for an image restoration model based on a multimodal large model, used to train the image restoration model used in the image restoration method based on a multimodal large model as described in claim 1, characterized in that, include: Receive degraded images to be restored from the training set, and input each degraded image and a multi-task text prompt into the multimodal large model. Offline, pre-generate corresponding operation instructions and visual description prompts for the degraded image, and store them as annotation information. The training set contains pairs of degraded images and clear images; The operation instructions and visual description prompts are encoded to obtain task intent vector and content guidance vector, respectively; The degraded image, task intent vector, and content guidance vector are input into the image restoration model. The task intent vector and content guidance vector are fused together to reconstruct the degraded image and output the restored image. The parameters of the first and second text encoders of the image restoration model are kept frozen, and only the parameters of the image restoration model and the additional linear projection layer are optimized. Based on the clear image corresponding to the output restored image and the degraded image, pixel loss, perceptual loss and cross-entropy loss are calculated, and a composite loss function is formed by weighting the pixel loss, perceptual loss and cross-entropy loss. Based on the composite loss function, the image restoration model is trained using training samples containing the paired degraded and clear images, thereby optimizing the parameters of the image restoration model and the linear projection layer.

7. The image restoration model training method based on multimodal large model-driven method according to claim 6, characterized in that, use The optimizer is trained using a cosine annealing learning rate scheduling strategy with preheating until the image restoration model converges.

8. An image restoration system based on multimodal large model-driven imaging, implementing the image restoration method based on multimodal large model-driven imaging as described in any one of claims 1-5, characterized in that, This includes a text prompt generation module, a text prompt encoding module, and an image restoration model: The text prompt generation module includes preset multi-task text prompt words, a multimodal large model, and a JSON parser. The module receives a degraded image and uses the multimodal large model to generate operation instructions defining the restoration operation and visual description prompts depicting the ideal scene. The multi-task text prompt words establish the overall image restoration task through a global format contract and specify that the output of the multimodal large model is a JSON object containing the key names `command` and `description`. Two sub-task specifications guide the multimodal large model to analyze the input degraded image, outputting a JSON format string containing the analysis results. The JSON parser extracts the values ​​corresponding to the key names `command` and `description` from the JSON format string, defining them as operation instructions and visual description prompts, respectively. The text prompt encoding module encodes the operation instructions and visual description prompts to obtain corresponding task intent vectors and content guidance vectors, respectively. The image restoration model receives the degraded image, the task intent vector, and the content guidance, and performs pixel-level repair and enhancement operations on the degraded image based on the task intent vector and the content guidance, and finally outputs the restored image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the image restoration method based on multimodal large model driving as described in any one of claims 1-5.

Images