An interactive integrated image restoration fusion method and device

Abstract

The application discloses an interactive integrated image restoration and fusion method and device, and the method comprises the following steps: constructing an interactive restoration and fusion module based on a multi-modal feature injection module, which is used for injecting image information of an infrared mode into a network with a pre-trained latent diffusion model as a backbone, so as to realize integrated interactive multi-modal image restoration and fusion; and constructing a fusion image structure restoration module, which is used for refining the structure and details of the restoration and fusion result by using original image information in the iterative denoising process of the latent diffusion model. The device comprises a processor and the processor. The application solves the problem of structure and detail loss caused by latent space fusion, so as to generate a fusion image with richer information and higher visual quality.

Description

Technical Field

[0001] This invention relates to the fields of multimodal image fusion and image restoration, and particularly to an interactive integrated image restoration and fusion method and apparatus. Background Technology

[0002] The research focus in the field of image fusion is undergoing a profound paradigm shift, evolving from the early pursuit of effectively integrating multimodal information to a core driving force focused on improving the performance of downstream tasks. Traditional image fusion methods, such as those based on autoencoders, convolutional neural networks, and generative adversarial networks, while achieving significant results in specific scenarios, still fall short in meeting the complex and diverse needs of downstream tasks.

[0003] Against this backdrop, the Text-IF (text-guided degradation perception and interactive image fusion) model pioneered the unified processing of image fusion and image restoration tasks, significantly enhancing the support capability of fusion results for downstream applications. In recent years, diffusion models, with their outstanding performance in visual content generation, have rapidly become a research hotspot in the field of image fusion, powerfully driving a profound shift in the field from functional to task-oriented approaches. For example, the DDFM (denoising diffusion for multimodal image fusion) model was the first to introduce diffusion models into image fusion tasks; the DIFF-IF (multimodal image fusion through diffusion models and fusion knowledge priors) model further improved the model's adaptability and fusion effect through a carefully crafted training strategy; and the Text-Difuse (an interactive multimodal image fusion framework based on text modulation diffusion models) innovatively combines the de-degradation process with the fusion task, thereby strengthening its task-oriented characteristics. This series of advancements has not only significantly broadened the application boundaries of image fusion technology but also provided solid technical support for improving the performance of downstream tasks.

[0004] However, the above methods often rely on carefully designed complex fusion networks and still face challenges in dealing with a variety of complex hybrid degradation problems. Summary of the Invention

[0005] This invention provides an interactive integrated image restoration and fusion method and apparatus. It designs a unified framework for robust restoration and fusion effects through flexible user language interaction, integrating image fusion and image restoration tasks into the latent diffusion process. By performing efficient multimodal interactive fusion in the latent space, it solves the degradation problem in the source image. Furthermore, this invention designs a dedicated structural restoration module for the fusion task to address the loss of structural details caused by latent space fusion, thereby generating a fused image with richer information and higher visual quality. See the description below for details.

[0006] A first aspect: an interactive integrated image restoration and fusion method, the method comprising:

[0007] An interactive restoration and fusion module based on a multimodal feature injection module is constructed to inject infrared modal image information into a network with a pre-trained latent diffusion model as the backbone, so as to achieve integrated interactive multimodal image restoration and fusion.

[0008] A fused image structure restoration module is constructed to refine the structure and details of the restored fusion result using the original image information during the iterative denoising process of the latent diffusion model.

[0009] The interactive recovery fusion module based on the multimodal feature injection module is as follows:

[0010] Using a pre-trained latent diffusion model as the backbone network, the generative prior knowledge learned by the backbone network for inpainting visible light images is fully preserved, as well as the high degree of freedom of text interaction provided by the pre-trained text encoder.

[0011] The multimodal feature injection module takes the sum of infrared and visible light image information as input, extracts the fusion features of the two modalities through deep feature extraction, and injects them into the frozen backbone network to perform image restoration and fusion tasks.

[0012] By introducing prior knowledge of the fusion image, the problem of the diffusion model being unable to be trained due to the lack of real reference images in the fusion task is solved.

[0013] The multimodal feature injection module is as follows:

[0014] Visible light images and infrared images are respectively processed by corresponding image encoders to obtain corresponding latent space representations. The latent space information of the visible light image is concatenated with the pure noise information and then added to the shallow space information of the infrared image to obtain fused information. This information is then input into the multimodal feature injection module.

[0015] The structure of this module is a complete replica of the backbone network U-Net encoder. It maps the input image to an initial feature map through a convolutional layer, and then consists of multiple hierarchical downsampling modules stacked together. Each downsampling module contains several convolutional blocks with residual connections.

[0016] Each convolutional block consists of a convolutional layer, a normalization layer, and an activation function. A convolutional layer or pooling layer with a stride of 2 is used to halve the spatial resolution of the feature map while increasing the channel dimension.

[0017] This structure is repeatedly stacked from top to bottom to form a feature pyramid that can extract multi-scale features. Finally, each multi-scale feature map extracted from each layer in the encoder body is connected to an independent 1x1 zero convolutional layer. The output of the zero convolutional layer constitutes the final output and is used to inject into the corresponding layer of the backbone network U-Net decoder.

[0018] The prior knowledge of the fused image is as follows:

[0019] For each pair of multimodal source images, an advanced fusion algorithm is used to generate multiple candidate fusion results;

[0020] Design a quantitative evaluation mechanism that includes structural similarity, gradient, and visual fidelity to score the multiple candidate fusion results;

[0021] Based on the scoring results, a unique and optimal fusion result is adaptively selected for each pair of source images and used as the pseudo-real value for training the network.

[0022] The fused image structure repair module is a U-Net-based encoder-decoder architecture.

[0023] The system receives the initial fused image, the original visible light image, and the original infrared image output by the backbone network in the current denoising step as inputs. The three inputs are then concatenated along the channel dimension to form a composite input tensor containing the content to be repaired and structural reference information.

[0024] A 1x1 convolutional layer is used to process the composite input tensor to fuse and adjust the channel dimensions, thereby achieving integrated information processing starting from the shallowest network layer.

[0025] The encoder backbone consists of multiple stacked NAFNet modules that do not contain non-linear activation functions, used for hierarchical downsampling and deep feature extraction of the features after front-end fusion;

[0026] The decoder is used to upsample the depth features extracted by the encoder step by step to restore the spatial resolution of the image and output a high-fidelity fused image with structural correction and enhanced detail texture.

[0027] A second aspect is an interactive integrated image restoration and fusion apparatus, the apparatus comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the apparatus to perform the method described in any one of the first aspects.

[0028] Third aspect, a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of the first aspects.

[0029] The beneficial effects of the technical solution provided by this invention are:

[0030] 1. This invention designs a more concise and efficient fusion framework. By combining the image fusion process with the latent space diffusion process containing de-degradation prior knowledge, it achieves highly condensed structure and integrated processing of multimodal information fusion and de-degradation.

[0031] 2. This invention proposes an interactive, integrated image inpainting and fusion framework. This framework, for the first time, unifies image inpainting and image fusion tasks within a latent diffusion process. This method effectively solves the problem of significant quality degradation in existing fusion techniques when faced with source image degradation (e.g., low light, noise). By simultaneously performing inpainting during fusion, it greatly improves image fusion performance in complex degradation scenarios. In various complex test scenarios, the fused images output by this invention exhibit excellent performance. This framework can not only accurately and thoroughly remove various degradation problems from the source image but also achieve high-quality multimodal information fusion. Compared with other existing advanced de-degradation and fusion models, the output results of this invention show significant superiority in both de-degradation effect and fusion quality—two key indicators.

[0032] 3. This invention designs an interactive guidance mechanism based on natural language prompts, allowing users to customize the restoration and fusion process of multimodal images through flexible text commands. This design gives the model unprecedented flexibility and interactivity, avoiding the drawbacks of traditional methods that require manually designing complex networks to cope with different degradation scenarios, and enabling a single framework to robustly handle multiple hybrid degradation problems.

[0033] 4. This invention proposes a Structure Repair Module (SRM) specifically designed for fusion tasks. This module addresses the problem of image structural detail loss caused by potential spatial compression operations. By learning and supplementing effective details from the source image, this module significantly enhances the detail fidelity and visual quality of the final fusion result. Extensive experiments on multiple public datasets such as LLVIP, MSRS, and RoadScene demonstrate that the method proposed in this invention outperforms the current state-of-the-art competing methods in all performance metrics. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the overall network structure;

[0035] Figure 2This is a flowchart of an interactive, integrated image restoration and fusion method. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0037] Diffusion models, with their representative examples DDPM (Denoising Diffusion Probabilistic Model) and DDIM (Denoising Diffusion Latent Model), have demonstrated outstanding visual generation capabilities. These models, through a progressively noisy diffusion process on the data distribution and learning its inverse denoising process, can gradually generate high-resolution images from pure noise. To further enhance the controllability and guidance of the generated results, Latent Diffusion Models (LDMs) have emerged. LDMs place the diffusion process within a compact latent space and interact with the latent representation of the image by introducing conditional information (e.g., text descriptions), achieving effective control over the generated content. This powerful image-text alignment capability and the model's deep understanding of visual semantics have enabled LDMs to achieve significant progress in image restoration. For example, models such as Autodir and Spire fully utilize the potential of latent diffusion models in image restoration and their convenient language interaction features. These research results provide a solid foundation of prior knowledge for de-degradation in current work.

[0038] To address the problems in the background art, namely, the poor performance of existing image fusion methods when processing source images degraded by noise, low light, etc., and the general lack of user interactivity, this invention proposes an interactive, integrated image inpainting and fusion framework. This framework can achieve multimodal information fusion and simultaneous image inpainting within the latent space, thereby improving the quality and usability of the final fused image. The framework mainly includes an interactive latent space inpainting and fusion module, an efficient multimodal feature injection module, and a fused image structure inpainting module. This method avoids the serious negative impact of source image degradation on the fusion result, and by integrating inpainting and fusion, it better utilizes multimodal features to improve the quality and information content of the final image.

[0039] Example 1

[0040] This invention provides an interactive integrated image restoration and fusion method, see [link to relevant documentation]. Figure 1 and Figure 2 The method includes the following steps:

[0041] 101: Constructing an interactive, integrated repair and fusion network, its core architecture employs a pre-trained interactive repair potential diffusion model as the backbone network, such as... Figure 1 The recovery and fusion process is shown in the lower half of the network. To fully preserve the model's powerful generative prior knowledge, the core parameters of this diffusion model are frozen during training. At the input, the network receives images in both visible and infrared modalities. First, Gaussian noise is applied to the visible light image, and then the original image and the noisy image are concatenated along the channel dimension to form the basic input of the backbone network. To achieve multimodal information fusion, the infrared image is fused with this concatenated input and then fused through a dedicated multimodal feature injection module, such as... Figure 1 The recovery and fusion process, as shown in the upper half of the network, efficiently integrates its information into the processing flow of the backbone network.

[0042] Meanwhile, to achieve highly flexible language control, this invention introduces a pre-trained CLIP (Contrastive Language-Image Pre-trained Model) text encoder. A key technical solution is that the CLIP text encoder maintains its parameters frozen throughout training. This design fully inherits its profound semantic understanding capabilities learned from massive amounts of data, allowing the model to directly utilize its rich image-text alignment knowledge without learning from scratch. To fully leverage its interactive potential, this invention also specifically constructs a rich, layered de-degradation descriptive corpus for different degradation scenarios. These carefully customized text instructions are designed to train the model to understand various language styles, from direct commands to abstract descriptions, thereby achieving truly flexible and robust text interaction. For example, for noisy or blurred degradation scenarios, this invention constructs a large number of de-degradation language descriptions such as "remove graininess and noise in the image," "restore the smooth details that the image should have," "make the details of objects in the image clearly visible," and "fix problems caused by defocusing." These carefully customized text instructions, combined with the frozen encoder, endow the model with the ability to understand and execute complex and diverse user commands, thereby achieving truly flexible and robust text interaction. The user's language input is converted into an embedding vector by the frozen CLIP encoder and then fed into the processing stream of the backbone network. The core mechanism of the network lies in a T-step iterative inverse denoising process. In each denoising step, the backbone network not only repairs degraded regions but also performs deep fusion of cross-modal information and cross-attention fusion of image and text information. Crucially, the intermediate results of each denoising step do not directly enter the next iteration but are fed into a structure restoration module along with the original visible light and infrared images. This module uses the original high-fidelity image as a reference to perform detail correction and structure enhancement on the intermediate results. The output is then used as the input for the next round of denoising. This process iterates until the final step, thereby generating a fused image with rich detail and accurate structure.

[0043] 102: Construct a multimodal feature injection module, designed to securely and efficiently integrate infrared modal information into the backbone network. Structurally, this module completely replicates the encoder portion of the backbone network. This homologous architecture with the backbone network encoder ensures that the extracted features are highly compatible with the native features of the backbone network in terms of scale and semantic hierarchy, laying the foundation for subsequent seamless fusion. Functionally, this module receives the fused multimodal image (i.e., the sum of the infrared image and the stitched input) as input, extracts its deep features, and then feeds them layer by layer through a series of zero-convolutional layers into the decoder of the backbone network.

[0044] Multimodal images include: degradation-affected visible light images V d The images, along with infrared image I, are derived from a multi-modal training dataset specifically constructed for multiple degradation scenes in this embodiment of the invention. This embodiment of the invention collects three of the most widely used datasets (LLVIP, RoadScene, and MSRS), and designs five different degradation algorithms to perform degradation operations on the originally clean images in these datasets. The degradation type d includes five degradation scenes: low light, rain, fog, blur, or noise, thereby making the training data of this embodiment more closely resemble real-world imaging conditions. The degraded visible light image V... d Variational autoencoder ε using a pre-trained latent space diffusion model r Encoding into the latent space A feature injection module is inserted into the backbone network to process the infrared image input. This module encodes the infrared image I into the same latent space as the visible light image information through a downsampling layer.

[0045] The module restores the training process of the fusion network as follows:

[0046] Since image fusion tasks lack absolutely true reference images, and the training of latent space diffusion models is completed during the noise addition and denoising process, it is essential to have true reference images for training the latent space diffusion model. Therefore, this invention proposes a pseudo-truth value construction method based on knowledge integration to create a high-quality "image fusion knowledge prior library" as the pseudo-truth value F for training. gSpecifically, this invention selects five industry-recognized advanced fusion algorithms, each with its own advantages in different aspects, to generate five candidate fusion results for each pair of source images V and I. Considering that the performance of each algorithm varies in different scenarios, this invention further designs a sophisticated selection mechanism to ensure the optimality of the pseudo-true value. This mechanism is a computational method optimized specifically for infrared-visible light fusion scenarios after extensive experimental comparisons: by quantifying the candidate results using a structural similarity index, a gradient-based fusion performance metric index, and a visual fidelity index, it adaptively selects a unique, optimal fusion result for each pair of source images as the training pseudo-true value F. g This strategy ensures that the model in this embodiment of the invention can learn a rigorously selected ideal target, thereby significantly improving the final training performance. Then, F... g Through ε r Encoding into latent space representation The user-input text prompts are encoded into text embeddings using a pre-trained text model. txt ,right Gaussian noise is added based on the random sampling time step t to obtain the noisy latent variable Z. t , will Z t and The input to the backbone network yields:

[0047]

[0048] in, For the features output by the backbone network encoder at step t, ε r (·) represents the U-Net encoder of the backbone network, and T represents the total number of denoising steps required. The feature injection module combines the infrared image I and the Z-axis of the backbone network. t and Combining the results, we get:

[0049]

[0050] in, For the features output by the encoder of the feature input module at step t, ε c (·) represents the U-Net encoder of the feature injection module, Z I These are the features of the original infrared image. Combining the two features obtained above and passing them through decoder D yields the latent variable Z. t Estimation of added noise:

[0051]

[0052] By constraining noise loss and loss parameters commonly used in multimodal image fusion, the model achieves a unified approach to image restoration and image fusion tasks. The inference process of this restoration and fusion network module is as follows: using pure Gaussian noise Z... T As an initial latent variable, in the user language prompt e txt Latent space information of visible light images affected by degradation and infrared image latent space information Z I Under the joint conditions, through model RF θ (·) Perform T-step iterative denoising process, the calculation method is as follows:

[0053]

[0054] Where t decreases from T to 0, model RF θ (·)for Figure 1 The recovery and fusion process shown consists of a pre-trained latent space diffusion model and a multimodal feature injection module.

[0055] 103: Construct a fusion image structure restoration module that employs a three-input design to achieve accurate structure restoration. Its overall architecture is based on a U-Net-like encoder-decoder. Specifically, the module receives three inputs: the image output from the latent space diffusion model and the multimodal original image (i.e., the degradation-affected visible light image V). d (And infrared image I). Before entering the encoder, these three images are first stitched together along the channel dimension to form a composite input tensor containing all the information. Subsequently, a 1x1 convolutional layer is used to fuse and adjust the channel dimension of the stitched image, and its output is then fed into the encoder backbone, which is composed of stacked NAFBlock (no activation function module). Through this early fusion strategy, the network can simultaneously perceive the content to be repaired and structural reference information from the shallowest layer, and perform integrated deep processing and fusion of this information in subsequent downsampling and feature extraction processes;

[0056] This module will restore the fusion result F output by the latent space diffusion model. ldm Visible light images affected by degradation V d The infrared image I is used as input, and the final fusion result F is output. end The calculation method is as follows:

[0057] F end =ζ(F ldm V d ,I),

[0058] Wherein, ξ is the structure repair module based on the encoder-decoder architecture of U-Net morphology.

[0059] In the joint training structure recovery module and latent space diffusion model RF θ When (·), freeze the parameters of the pre-trained latent space diffusion model and use the output F of the structure recovery module. end Replace F ldm :

[0060]

[0061] in, The other two represent structural similarity loss, intensity loss, gradient loss, and color loss. Representing F end V and I represent the original visible light image and the original infrared image, respectively; SSIM represents structural similarity calculation; and MAX represents taking the maximum value. F represents gradient evaluation. CbCr This represents color calculation.

[0062] During the inference process, for each denoising step t, the latent representation Z generated by the latent space diffusion model... t Used to estimate the denoising result of the current step. The calculation method is as follows:

[0063]

[0064] in, For time-step dependent scaling factors, ∈ θ (·) represents the noise prediction. The resulting... The image is then remapped back to pixel space by the decoder, and information is enhanced by the image fusion result restoration module before being encoded back into the latent space as input for the next step of denoising.

[0065]

[0066] in, For VAE decoder, This refers to the information in the pixel space that has been enhanced and then fused to restore the image information.

[0067] In summary, this method proposes an advanced multimodal interactive inpainting fusion framework, aiming to fuse multi-source information to achieve high-fidelity image inpainting and generation. The essence of this framework lies in the synergistic effect of three core components: an interactive latent space inpainting and fusion network as the foundation of generation, a multimodal feature injection module as an information bridge, and a fusion image structure inpainting module as quality assurance. The framework uses a pre-trained latent diffusion model with frozen parameters as the interactive latent space inpainting and fusion network to fully preserve its powerful image generation priors. By freezing the CLIP text encoder and utilizing a customized instruction corpus, this embodiment of the invention achieves unprecedented high-degree-of-freedom human-computer interaction. By constructing a dedicated dataset containing complex hybrid degradation and designing a pseudo-truth value generation method based on knowledge integration, this embodiment of the invention cleverly solves the core challenges in model training. A multimodal feature injection module is designed to safely introduce new modal information (such as infrared images) into this pre-trained system. This module replicates the structure of the backbone encoder and injects new features into the backbone decoder through "zero convolutional layers" with weights initialized to zero. This ensures stability during the initial training phase while achieving learnable and adaptive information injection strength. Crucially, the fusion image structure restoration module acts as an iterative refiner throughout the restoration process. In each of the T-step denoising steps, this module compares the intermediate output of the backbone network with the original high-fidelity multimodal image, actively correcting structural biases and enhancing detail textures to ensure the generation process consistently converges towards the high-fidelity goal. The entire network's workflow is a precise iterative loop of "restoration-calibration." In each denoising step, the backbone network, guided by the multimodal features provided by the injection module, generates an intermediate restoration result. This result does not directly proceed to the next step but is first fed into the structure restoration module, where it is refined using the original image as a benchmark. The corrected and optimized result serves as the input for the next round of denoising, and this process is repeated T times. Leveraging the powerful generation capabilities of the backbone network, the flexible information supplementation of the injection module, and the continuous fidelity constraints of the repair module, this method ultimately outputs an image that integrates the advantages of multimodal processing and boasts highly accurate details and structure. This method effectively addresses the industry pain point of severe source image degradation significantly impacting fusion quality, while also providing flexible user interaction capabilities, ultimately generating a fused image of far superior quality to existing technologies.

[0068] Example 2

[0069] The scheme in Example 1 will be further described below with specific examples and calculation formulas:

[0070] I. Data Preparation

[0071] The embodiments of this invention validate the effectiveness of the proposed method on several publicly available visible-infrared image fusion benchmark datasets, which provide the model with realistic and challenging experimental scenarios. The main datasets used include LLVIP, Roadscene, and MRSR datasets. LLVIP (Low-Light Vision Visible-Infrared Pairing Dataset) aims to address the challenges of computer vision tasks in low-light conditions, such as image fusion, pedestrian detection, and image-to-image translation. This dataset contains 30,976 images, or 15,488 pairs of strictly temporally and spatially aligned visible and infrared images, most of which were taken in very dark scenes and include pedestrian annotations. By providing complementary information from both visible and infrared modalities, LLVIP helps develop more robust vision algorithms in low-light environments. The RoadScene dataset actually encompasses two different datasets, both focused on environmental perception in the field of autonomous driving: the Road Scene Semantic Segmentation Dataset, specifically designed for semantic segmentation tasks in autonomous driving applications, aims to perform pixel-level classification and depiction of various elements in road scenes. It contains approximately 2,000 images collected from the internet, each with a resolution of 1920x1080 pixels, and semantic segmentation annotations for elements such as sky, buildings, lane lines, and pedestrians. The Road Scene Graph dataset goes beyond traditional object detection, focusing on understanding the relationships between objects in driving scenes, such as "a vehicle is waiting for a pedestrian." It represents objects in the driving environment as nodes, and the relationships between them as edges, thus forming a scene graph. This dataset is based on existing datasets such as Nuscenes (a multimodal dataset for autonomous driving) and CARLA (an open-world driving simulator), containing camera images and 6D bounding box information, aiming to improve risk detection, scene description, and model interpretability. The MSRS (Multispectral Road Scene) dataset is a novel multispectral dataset designed specifically for infrared and visible light image fusion tasks. Its main purpose is to provide high-quality, aligned infrared and visible light image pairs to facilitate the development and evaluation of advanced image fusion algorithms, especially under challenging conditions such as low light. The dataset contains 1,444 pairs of aligned infrared and visible light images, of which 715 pairs are daytime images and 729 pairs are nighttime images. All images have a spatial resolution of 480×640 pixels.

[0072] To comprehensively verify the model's restoration and fusion capabilities, this embodiment of the invention selected high-quality visible light images from the aforementioned dataset and applied artificially simulated degradation effects, including five types: low light, rain, haze, blur, and noise. To enhance the model's user interaction flexibility, this embodiment of the invention also generated hundreds of corresponding text prompts for these degraded images. Ultimately, the experiments in this embodiment of the invention used 21,809 image pairs for model training and 675 image pairs for performance testing.

[0073] II. Interactive Image Restoration and Fusion Network Structure

[0074] The interactive integrated repair and fusion network (AIR-Fusion) in this embodiment of the invention, such as... Figure 2 As shown, it mainly consists of two core parts: an interactive latent space diffusion module for repair and fusion, and a structural repair module for supplementing details.

[0075] This invention uses a pre-trained latent space diffusion model as its backbone network. Its core mechanism involves adding noise to a degraded visible light image, encoding it into a latent space representation using the VAE encoder of the diffusion model, and then progressively removing the degradation effects through denoising. Simultaneously, an efficient feature injection module (adapter) inspired by ControlNet (adding conditional control to text-to-image diffusion models) encodes an infrared image of another modality into the same latent space. Guided by user text commands, features from the visible light image and features from the infrared image are fused during the decoding phase of the U-Net (U-network) to estimate the noise to be removed. After removing the estimated noise, a restored and fused image is obtained. To address the detail loss problem caused by latent space compression, a structure restoration module is invoked at each inference step. This module maps the preliminary fusion result of the current step back to the pixel space, enhances its details using information from the original multimodal image, and then encodes it back into the latent space for the next denoising step, resulting in a final restored and fused image with complete structural information.

[0076] III. Evaluation Indicators and Agreements

[0077] To quantitatively evaluate the quality of the fused image, this embodiment of the invention employs a series of recognized evaluation metrics for comprehensive measurement. These metrics evaluate the fusion result from multiple dimensions, including information richness, visual fidelity, and structural fidelity. Specifically, they include: standard deviation (SD) and information entropy (EN), which measure the amount of information contained in the image; visual information fidelity (VIF), which evaluates the visual perceptual consistency between the fused result and the source image; and a gradient-based fusion performance metric (Q). AB / FThe study focuses on the transmission of edge and gradient information, as well as peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), which are commonly used to evaluate the degree of image distortion and structure preservation.

[0078] IV. Details on Model Usage

[0079] 1. Data expansion:

[0080] Due to limited data resources, high-quality visible light images were carefully selected as the foundation from several recognized benchmark datasets, including LLVIP (a visible-infrared paired dataset for low-light visual visitors), MSRS (a multispectral road scene dataset for practical infrared and visible light image fusion), and RoadScene (a new infrared and visible light aligned image dataset). Subsequently, a series of simulated, real-world common degradation effects were applied to these clear images using an algorithm, specifically covering five typical complex scenes: low light, rain, fog, blur, and noise. To train and validate the model's interactive inpainting capabilities, a large number of corresponding natural language inpainting instructions were carefully designed and generated for each artificially degraded image.

[0081] 2. Model optimization:

[0082] During training, this method optimizes only the parameters of the newly introduced adapter and structural repair modules, while keeping the backbone network parameters frozen. Parameter updates are performed using the Adam optimizer, with a constant learning rate of 1e-5. Since image fusion is an unsupervised task, this method constructs a comprehensive image fusion knowledge prior and uses it as pseudo-ground values ​​to guide model training. All experiments were performed on an NVIDIA RTX A6000 GPU.

[0083] The embodiments of the present invention have the following two key inventive points:

[0084] I. An interactive, integrated image restoration and fusion framework is proposed.

[0085] Technical Results: This framework innovatively unifies image inpainting and image fusion tasks within a single latent diffusion model. We propose a novel "inpainting as fusion" paradigm, aiming to fundamentally address a major industry pain point in existing image fusion technologies: when the source image suffers from degradation issues such as low light, noise, or blur, the quality of the fused image deteriorates drastically. By simultaneously repairing degradation information during the fusion process, this technique can generate robust fused images that remain clear and retain rich details even in complex imaging environments. Furthermore, this mechanism leverages the efficient text-image interaction capabilities within the latent space, allowing users to precisely guide the model's inpainting and fusion behavior through intuitive natural language commands such as "remove haze" and "increase brightness." This design not only endows the model with unprecedented user interactivity and task adaptability but also avoids the complexity of traditional methods that require designing dedicated networks for different degradation types, enabling a single framework to flexibly handle diverse and unpredictable real-world application scenarios.

[0086] II. A structural repair module specifically designed for fusion tasks is proposed.

[0087] Technical Results: This module specifically addresses the image detail loss problem commonly encountered in latent spatial diffusion models due to information compression. By learning to extract and supplement effective structural and texture information from multimodal source images, the SRM (Structure Restoration Module) significantly enhances the detail fidelity and overall visual quality of the final fused image. Extensive experiments on multiple public datasets such as LLVIP, MSRS, and RoadScene have validated that the method of this invention outperforms state-of-the-art techniques across various evaluation metrics.

[0088] In summary, this invention provides an interactive, integrated image inpainting and fusion framework. This framework creatively designs a unified latent spatial diffusion process, enabling simultaneous image inpainting and multimodal information fusion; it also introduces a natural language-based interaction mechanism, giving users flexible control over the inpainting task; finally, a unique structural inpainting module significantly enhances the detail representation of the fusion result. This invention overcomes the negative impact of source image degradation on fusion quality, and can generate fused images with high fidelity and rich detail in various complex scenes.

[0089] Example 3

[0090] The method proposed in this invention has been compared with several state-of-the-art image fusion methods on multiple public datasets, including LLVIP, MRSR, and Roadscene. These comparison methods include: U2Fusion (a unified unsupervised image fusion network), TarDAL (object-aware dual adversarial learning and multi-scene multimodal benchmark for object detection by fusing infrared and visible light), SwinFusion (cross-domain long-range learning for general image fusion via Swin transform), CDDFuse (correlation-driven bi-branch eigenvalue decomposition for multimodal image fusion), DDFM (denoising diffusion model for multimodal image fusion), EMMA (equivariant multimodal image fusion), and Text-IF (degradation-aware and interactive image fusion guided by semantic text).

[0091] The experimental results in fusion scenarios without degradation effects are shown in Table 1. On the LLVIP dataset, the method proposed in this embodiment achieves the best performance in four of the five evaluation metrics. Specifically, the method in this embodiment shows significant advantages in VIF (Visual Information Fidelity), Qabf (a gradient-based fusion performance metric), and PSNR (Peak Signal-to-Noise Ratio), which fully demonstrates that the method in this embodiment can generate fused images with rich information and excellent visual quality. The EN metric ranks second, differing from the best value by only 0.05, which also proves that even when dealing with ideal image fusion tasks without degradation effects, the method in this embodiment still has excellent fusion performance. On the MSRS and RoadScene datasets, which were not used for training, the method in this embodiment shows good generalization ability. It achieves the best performance in SD, Qabf, and PSNR metrics. Although it is slightly inferior to some of the comparison methods in EN and VIF metrics, the overall results are sufficient to prove that the method has robust generalization performance on unseen datasets. In summary, the experimental results show that the method in the embodiments of the present invention can not only effectively handle image fusion tasks without degradation effects and generate fused images with high visual quality, but also demonstrate excellent cross-dataset generalization ability.

[0092] Table 2 presents the quantitative evaluation results of fusion under different degradation scenarios. Analysis of the data shows that the method in this embodiment achieves optimal values ​​for all evaluation metrics in rainy, blurry, and low-light degradation scenarios. This fully demonstrates that the method in this embodiment possesses excellent de-degradation performance and efficient image fusion capabilities when dealing with these specific degradation problems. In the haze degradation scenario, although the visual information fidelity index of the method in this embodiment is not the best, its structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) indices both reach the highest level. This indicates that the method in this embodiment can effectively recover the structural information of the image and maintain high pixel-level fidelity in this scenario. For noisy scenarios, the overall performance of the method in this embodiment is slightly inferior to some of the comparison methods, but its PSNR index still ranks first. This shows that the method in this embodiment achieves a better balance between noise suppression and image detail preservation. It is worth noting that the roadscene dataset was not used in the training process of the method in this embodiment. Nevertheless, the fusion results of the method in this embodiment on this dataset still show strong competitiveness, which further confirms the generalization ability and robustness of the method in this embodiment.

[0093] The experimental results used in the embodiments of this invention are shown in Table 3. To further verify the visual quality and information preservation capability of the fusion result of the model of this invention, object detection experiments were conducted on the fused images. YOLOv5 was used as the backbone network for object detection. The experimental results show that the fusion result of the method of the embodiments of this invention exhibits the best detection performance.

[0094] Table 1

[0095]

[0096] Table 2

[0097]

[0098]

[0099] Table 3

[0100] mAP@0.50 89.6 89.0 89.6 89.3 91.8 89.9 90.3 92.3 mAP@0.50:0.95 66.4 67.1 66.7 66.4 67.3 66.6 69.0 71.7

[0101] Example 4

[0102] An interactive integrated image restoration and fusion device includes a processor and a memory. The memory stores program instructions, and the processor calls the program instructions stored in the memory to cause the device to perform the following method steps in Embodiment 1:

[0103] An interactive restoration and fusion module based on a multimodal feature injection module is constructed to inject infrared modal image information into a network with a pre-trained latent diffusion model as the backbone, so as to achieve integrated interactive multimodal image restoration and fusion.

[0104] A fused image structure restoration module is constructed to refine the structure and details of the restored fusion result using the original image information during the iterative denoising process of the latent diffusion model.

[0105] The interactive recovery fusion module based on the multimodal feature injection module is as follows:

[0106] Using a pre-trained latent diffusion model as the backbone network, the generative prior knowledge learned by the backbone network for inpainting visible light images is fully preserved, as well as the high degree of freedom of text interaction provided by the pre-trained text encoder.

[0107] The multimodal feature injection module takes the sum of infrared and visible light image information as input, extracts the fusion features of the two modalities through deep feature extraction, and injects them into the frozen backbone network to perform image restoration and fusion tasks.

[0108] By introducing prior knowledge of the fusion image, the problem of the diffusion model being unable to be trained due to the lack of real reference images in the fusion task is solved.

[0109] The multimodal feature injection module is as follows:

[0110] Visible light images and infrared images are respectively processed by corresponding image encoders to obtain corresponding latent space representations. The latent space information of the visible light image is concatenated with the pure noise information and then added to the shallow space information of the infrared image to obtain fused information. This information is then input into the multimodal feature injection module.

[0111] The structure of this module is a complete replica of the backbone network U-Net encoder. It maps the input image to an initial feature map through a convolutional layer, and then consists of multiple hierarchical downsampling modules stacked together. Each downsampling module contains several convolutional blocks with residual connections.

[0112] Each convolutional block consists of a convolutional layer, a normalization layer, and an activation function. A convolutional layer or pooling layer with a stride of 2 is used to halve the spatial resolution of the feature map while increasing the channel dimension.

[0113] This structure is repeatedly stacked from top to bottom to form a feature pyramid that can extract multi-scale features. Finally, each multi-scale feature map extracted from each layer in the encoder body is connected to an independent 1x1 zero convolutional layer. The output of the zero convolutional layer constitutes the final output and is used to inject into the corresponding layer of the backbone network U-Net decoder.

[0114] Among them, the fusion of prior image knowledge is as follows:

[0115] For each pair of multimodal source images, an advanced fusion algorithm is used to generate multiple candidate fusion results;

[0116] Design a quantitative evaluation mechanism that includes structural similarity, gradient, and visual fidelity to score the multiple candidate fusion results;

[0117] Based on the scoring results, a unique and optimal fusion result is adaptively selected for each pair of source images and used as the pseudo-real value for training the network.

[0118] Among them, the fused image structure repair module is an encoder-decoder architecture based on the U-Net morphology;

[0119] The system receives the initial fused image, the original visible light image, and the original infrared image output by the backbone network in the current denoising step as inputs. The three inputs are then concatenated along the channel dimension to form a composite input tensor containing the content to be repaired and structural reference information.

[0120] A 1x1 convolutional layer is used to process the composite input tensor to fuse and adjust the channel dimensions, thereby achieving integrated information processing starting from the shallowest network layer.

[0121] The encoder backbone consists of multiple stacked NAFNet modules that do not contain non-linear activation functions, used for hierarchical downsampling and deep feature extraction of the features after front-end fusion;

[0122] The decoder is used to upsample the depth features extracted by the encoder step by step to restore the spatial resolution of the image and output a high-fidelity fused image with structural correction and enhanced detail texture.

[0123] It should be noted that the device descriptions in the above embodiments correspond to the method descriptions in the embodiments, and the embodiments of the present invention will not be repeated here.

[0124] The execution entities of the aforementioned processor and memory can be devices with computing functions such as computers, microcontrollers, and single-chip microcomputers. In specific implementations, the embodiments of the present invention do not limit the execution entities and can select them according to the needs of actual applications.

[0125] Data signals are transmitted between the memory and the processor via a bus, which will not be elaborated upon in this embodiment of the invention.

[0126] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, the storage medium including a stored program, which, when the program is running, controls the device where the storage medium is located to execute the method steps in the above embodiments.

[0127] The computer-readable storage medium includes, but is not limited to, flash memory, hard disk, solid-state drive, etc.

[0128] It should be noted that the description of the readable storage medium in the above embodiments corresponds to the description of the method in the embodiments, and the embodiments of the present invention will not be repeated here.

[0129] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated.

[0130] A computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted through a computer-readable storage medium. A computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic or semiconductor, etc.

[0131] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0132] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An interactive integrated image restoration and fusion method, characterized in that, The method includes: An interactive restoration and fusion module based on a multimodal feature injection module is constructed to inject infrared modal image information into a network with a pre-trained latent diffusion model as the backbone, so as to achieve integrated interactive multimodal image restoration and fusion. A fused image structure restoration module is constructed to refine the structure and details of the restored fusion result using the original image information during the iterative denoising process of the latent diffusion model. The interactive recovery fusion module based on the multimodal feature injection module is as follows: Using a pre-trained latent diffusion model as the backbone network, the generative prior knowledge learned by the backbone network for inpainting visible light images is fully preserved, as well as the high degree of freedom of text interaction provided by the pre-trained text encoder. The multimodal feature injection module takes the sum of infrared and visible light image information as input, extracts the fusion features of the two modalities through deep feature extraction, and injects them into the frozen backbone network to perform image restoration and fusion tasks. By introducing prior knowledge of fused images, the problem of diffusion models being unable to be trained due to the lack of real reference images in fusion tasks can be solved; The multimodal feature injection module works as follows: the visible light image and the infrared image are respectively processed by the corresponding image encoder to obtain the corresponding latent space representation; the latent space information of the visible light image is concatenated with the pure noise information and then added with the latent space information of the infrared image to obtain fused information; this information is then input into the multimodal feature injection module. The fused image structure repair module is based on a U-Net-based encoder-decoder architecture. It receives the initial fused image, the original visible light image, and the original infrared image output by the backbone network in the current denoising step as inputs, and concatenates the three inputs along the channel dimension to form a composite input tensor containing the content to be repaired and structural reference information. A 1x1 convolutional layer is used to process the composite input tensor to fuse and adjust the channel dimensions, thereby achieving integrated information processing starting from the shallowest network layer. The encoder backbone consists of multiple stacked NAFNet modules that do not contain non-linear activation functions, used for hierarchical downsampling and deep feature extraction of the features after front-end fusion; The decoder is used to upsample the depth features extracted by the encoder step by step to restore the spatial resolution of the image and output a high-fidelity fused image with structural correction and enhanced detail texture.

2. The interactive integrated image restoration and fusion method according to claim 1, characterized in that, The structure of the multimodal feature injection module is a complete replica of the backbone network U-Net encoder. It maps the input image to an initial feature map through a convolutional layer, and then is composed of multiple hierarchical downsampling modules stacked together. Each downsampling module contains several convolutional blocks with residual connections. Each convolutional block consists of a convolutional layer, a normalization layer, and an activation function. A convolutional layer or pooling layer with a stride of 2 is used to halve the spatial resolution of the feature map while increasing the channel dimension. This structure is repeatedly stacked from top to bottom to form a feature pyramid that can extract multi-scale features. Finally, each multi-scale feature map extracted from each layer in the encoder body is connected to an independent 1x1 zero convolutional layer. The output of the zero convolutional layer constitutes the final output and is used to inject into the corresponding layer of the backbone network U-Net decoder.

3. The interactive integrated image restoration and fusion method according to claim 1, characterized in that, The prior knowledge of the fused image is: For each pair of multimodal source images, an advanced fusion algorithm is used to generate multiple candidate fusion results; Design a quantitative evaluation mechanism that includes structural similarity, gradient, and visual fidelity to score the multiple candidate fusion results; Based on the scoring results, a unique and optimal fusion result is adaptively selected for each pair of source images and used as the pseudo-real value for training the network.

4. An interactive integrated image restoration and fusion device, characterized in that, The device includes a processor and a memory, the memory storing program instructions, the processor invoking the program instructions stored in the memory to cause the device to perform the method according to any one of claims 1-3.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1-3.

Images