A multi-degraded image enhancement system based on cross-modal co-evolution

Abstract

This invention discloses a multi-degraded image enhancement system based on cross-modal collaborative evolution, comprising a method and system for multi-degraded image enhancement. The invention addresses the limitations of single-modal information boundaries, the restricted scope of "all-in-one" models in multi-degraded image enhancement, and performance degradation inherent in existing image inpainting techniques. This invention constructs a multi-modal image inpainting model library by integrating multi-source information such as infrared, near-infrared, depth, and text; it achieves intelligent identification of image degradation types and degrees, automatic reasoning of restoration strategies, and dynamic scheduling by introducing a collaborative scheduling mechanism between visual language models and large language models; and it improves the system's adaptability in complex scenarios by designing a reflective-rescheduling closed-loop restoration process that automatically rolls back and replans when restoration fails to meet expectations. This invention belongs to the fields of computer vision and artificial intelligence technology.

Description

Technical Field

[0001] This invention relates to a method and system for enhancing multiple degraded images, belonging to the fields of computer vision and artificial intelligence technology, and is applicable to application scenarios such as medical imaging, security monitoring, satellite remote sensing, and autonomous driving that require the restoration and enhancement of complex degraded images. Background Technology

[0002] Image enhancement is a core research area in computer vision, aiming to recover clear and realistic visual content from degraded images. However, existing image enhancement methods perform well in single degraded scenes (such as noise, haze, and low lighting), but their performance drops significantly in irregularly damaged scenes with severe information loss or complex degraded scenes. The main problems include: 1. Existing image restoration methods are mostly based on single-modal information (such as RGB images) for restoration. Their restoration process is highly dependent on the pixel information of the image itself, resulting in a lack of sufficient information sources during the restoration process.

[0003] 2. Existing mainstream multi-degrade image restoration techniques, such as all-in-one models, have limitations in terms of task scope and performance.

[0004] 3. Existing score-based image quality assessment methods are limited to complex mathematical analysis and weighting; the information provided by a single score is insufficient to reasonably evaluate image quality. In summary, in order to find an optimal strategy for restoring multi-degraded images while balancing computation time and restoration performance, there is an urgent need for a multi-degraded image enhancement system that can integrate multimodal information and has intelligent decision-making and adaptive adjustment capabilities.

[0005] A patent application (CN120852200A, filed on July 24, 2025) discloses an ultrasound image enhancement method, system, and medium based on multimodal image transfer. The method includes acquiring multimodal image data; spatially aligning ultrasound image data and CT / MRI image data to obtain registered CT / MRI image data; converting the registered CT / MRI image data into pseudo-ultrasound image data with an ultrasound style; and using a trained dual-path anatomy-texture fusion enhancement network to enhance the acquired ultrasound image data, fusing them to generate enhanced ultrasound image data. This invention integrates physical imaging models and deep learning-based cross-modal image enhancement technology. By constructing a joint representation space from CT / MRI to ultrasound, it transfers high-resolution anatomical structure information to the ultrasound imaging system, significantly improving the spatial resolution and tissue contrast of ultrasound images. It is suitable for scenarios such as intraoperative real-time navigation, interventional therapy, and clinical diagnosis.

[0006] However, the aforementioned patented technologies still cannot solve the problems of the limitations of single-modal information boundaries, the limited scope of "all-in-one" model tasks in the restoration of multiple degraded images, and performance degradation in existing image restoration technologies. Summary of the Invention

[0007] To address the limitations of single-modal information boundaries, the restricted scope of "all-in-one" model tasks, and performance degradation in existing image restoration technologies, this invention proposes a multi-degraded image enhancement system based on cross-modal collaborative evolution.

[0008] The technical solution adopted by the present invention to solve the above problems is as follows: The steps of the multi-degraded image enhancement method based on cross-modal cooperative evolution described in the present invention include: Step 1: Construct a multimodal image restoration model library; systematically test and integrate high-performance single-modal and multimodal restoration models for various typical degradation types; Step 2: Identify the type and severity of image degradation in the input image based on a visual language model; Step 3: Acquire and intelligently schedule image restoration experience, build a priori restoration experience database, and retrieve the experience database based on a large language model to generate the optimal restoration order and model combination; Step 4: Design Reflection - Rescheduling Image Inpainting Mechanism. After the inpainting strategy is executed, the output image is evaluated. If the inpainting does not meet expectations, failure information is automatically fed back and a new inpainting plan is formulated, forming a closed-loop optimization process.

[0009] Furthermore, the steps in step 1 for constructing the multimodal image inpainting model library include: Step 101: Under a unified dataset and evaluation system, conduct system performance testing and verification for advanced models with different degradation types; Step 102: Regarding the evaluation index system, a combination of traditional image quality evaluation index scores with subjective visual effects is adopted. Step 103: In terms of single-modal repair model integration, integrate multiple models with better test results, such as SwinIR, RefineDNet, and RRDNet; Step 104: In terms of multimodal restoration, the DVN model is integrated to achieve visible light-near infrared fusion denoising, the VIFNet model is used to complete visible light-infrared image fusion dehazing, and the LEDN model is integrated to guide low-light enhancement using depth images.

[0010] Furthermore, step 2 specifically includes: Step 201: By combining eight degradation types, a complex dataset containing 16 common real-world degradation combinations was generated, and a severity level of five levels from very low to very high was randomly generated for each degradation. Step 202: Use the dataset to fine-tune the visual language model DepictQA, enabling it to accurately identify all degradation types in the input image and assess their severity, providing a key basis for subsequent repair tool selection and workflow scheduling.

[0011] Furthermore, step 3 specifically includes: Step 301: On an image dataset containing multiple degradation types, iterate through various image processing strategies and record the output results corresponding to each strategy in detail. Step 302: Use DepictQA to assist in evaluating the repair effects of different strategies, identify successful repair strategies, and statistically summarize the key indicators of repair success rate into a structured experience document. Step 303: Retrieve the experience base based on the large language model, and use the preset prompt words to perform reasoning to generate the optimal repair order and model combination.

[0012] Furthermore, step 4 specifically includes: Step 401: After the repair strategy is executed, the output image is evaluated in real time using the DepictQA model. If the repair does not meet expectations, it is judged as a failure and the rollback mechanism is triggered. Step 402: The system rolls back to the previous processing stage and feeds back the failure information to the large language model for reflection and strategy rescheduling, generating an adjusted repair plan to achieve dynamic optimization of the repair strategy and adaptive adjustment of the system.

[0013] The multi-degraded image enhancement system based on cross-modal collaborative evolution described in this invention includes a multi-modal image restoration model library, a degradation identification and evaluation module, an intelligent scheduling and reasoning module, and a restoration execution and feedback module; The multimodal image restoration model library integrates dedicated processing models for typical degradation factors. After systematically investigating cutting-edge single-modal processing methods, it further expands to the selection and integration of multimodal image restoration models. Each selected model has been systematically tested on multiple standard datasets, and its objective indicators such as PSNR and SSIM under different degradation levels have been fully recorded. Subjective evaluation data is obtained by combining the visual quality evaluation model to form a multi-dimensional evaluation system. The degradation identification and evaluation module constructs a dataset containing various combinations of degradation types and fine-tunes the DepictQA-Wild visual language model to intelligently identify the degradation type of the input image and assess its severity. The intelligent scheduling and reasoning module, based on a large language model, constructs a decision-making system with experience learning capabilities. The system first explores the policy space and conducts full-sequence repair tests on 16 common degradation combinations, recording the success rate and failure rate of different processing orders to form a structured experience document. During the actual repair process, the large language model dynamically retrieves the experience knowledge base based on the output results of the degradation identification module and infers the optimal repair order and model combination. This module also introduces a semantic reasoning mechanism. When encountering an unseen degradation combination, the large language model can perform analogical reasoning based on similar cases to generate a reasonable repair solution. The repair execution and feedback module is responsible for executing repair tasks and achieving dynamic optimization. According to the repair plan generated by the intelligent scheduling module, the system sequentially calls the corresponding tools in the model library for step-by-step processing. After each repair step is completed, the DepictQA model evaluates the quality of the intermediate results. If a repair step does not meet expectations, the system automatically triggers the rollback mechanism, returns to the previous processing node, and re-plans the strategy.

[0014] The beneficial effects of this invention are: 1. This invention overcomes the limitations of single-modality information through multimodal information fusion, and innovatively constructs a cross-modal collaborative enhancement framework. It enhances semantic guidance through text modality and optimizes decision-making efficiency by leveraging the reasoning capabilities of LLM; simultaneously, it fully utilizes the complementary characteristics of multimodal image data to significantly improve image restoration quality in complex degraded scenarios.

[0015] 2. This invention introduces an intelligent scheduling mechanism that combines LLM and VLM, which can intelligently diagnose the type and degree of image degradation and dynamically select or combine the optimal repair algorithm from the model toolbox based on the diagnosis results. This effectively reduces reliance on human experience and improves the system's processing efficiency and intelligence level.

[0016] 3. This invention integrates multiple complementary repair models and combines them with a feedback-based rollback mechanism to adjust the processing strategy in real time when the repair effect does not meet expectations, ensuring that the system has adaptive enhancement capabilities and excellent robustness in diverse real-world scenarios.

[0017] 4. This invention employs a highly flexible modular architecture design, constructing an scalable model library system capable of continuously integrating new repair algorithms and degradation types. This design endows the system with excellent maintainability, scalability, and long-term practical value, providing a solid foundation for subsequent technology upgrades and application expansion. Attached Figure Description

[0018] Figure 1 This is an overall architecture diagram of the present invention; Figure 2 This is a flowchart of the intelligent scheduling and reflection-rescheduling image restoration mechanism. Detailed Implementation

[0019] Specific implementation method one: as follows Figure 1 and Figure 2 As shown, the steps of a multi-degraded image enhancement method based on cross-modal co-evolution include: Step 1: Construct an image restoration model library; For various typical degradation types (denoising, dehazing, deraining, deblurring, low-light enhancement, super-resolution, JPEG artifact removal, etc.), systematically test and integrate high-performance single-modal restoration models, and construct a reasonable evaluation index system to accurately reflect the actual restoration effect of the models; Constructing a reasonable evaluation indicator system includes: The restoration effect is evaluated by combining traditional image quality assessment indicators with subjective visual effects. The traditional indicators include PSNR, SSIM, and NIQE for quantitative evaluation. At the same time, the open-source visual language question answering model DepictQA-Wild is introduced to assist in judging the visual effect of image enhancement, so as to make up for the shortcomings of human eye judgment being too subjective and having a high degree of uncertainty. Step 2: Introduce multimodal guidance information to assist image enhancement. By utilizing multimodal data such as near-infrared, depth, and infrared, a cross-modal feature fusion mechanism is established to expand the processing capability and robustness of the image enhancement system in irregular damage scenarios with severe information loss. Introducing multimodal guidance information to assist image enhancement includes: By utilizing multimodal guidance information such as near-infrared, depth, and infrared, a high-performance multimodal image restoration model is integrated, and a cross-modal feature fusion mechanism is constructed to enhance the image's detail restoration and structural integrity through multi-source information complementarity. Step 3: Image restoration experience acquisition and intelligent scheduling; explore restoration strategies for actual degraded images, record and statistically analyze the restoration effects of different strategies, and construct a priori restoration experience database; retrieve this experience database based on a large language model, and use preset prompt words to infer the optimal restoration order and model combination to achieve intelligent scheduling of the restoration process; Image restoration experience acquisition and intelligent scheduling include: traversing various image processing strategies on image datasets containing multiple degradation types, fully recording the output results corresponding to each strategy; using DepictQA to assist in evaluating the restoration effect of different strategies, identifying successful restoration strategies applicable to specific degradation conditions, and statistically summarizing key indicators such as restoration success rate, organizing them into a structured experience document as the prior knowledge basis for subsequent intelligent scheduling. Step 4: Design Reflection - Rescheduling Image Inpainting Mechanism. After the inpainting strategy is executed, the output image is evaluated. If the inpainting does not meet expectations, the system will automatically provide failure information, generate adjustment suggestions, and re-formulate the inpainting plan, starting a new round of inpainting process to form a closed-loop optimization process. The reflection-rescheduling mechanism design includes: after the execution phase, the output image is evaluated in real time using the DepictQA model. If the repair does not meet expectations, it is judged as a failure and a rollback mechanism is triggered. The system rolls back to the previous processing phase and feeds back the failure information to the large language model for reflection and strategy rescheduling, generating an adjusted repair plan to achieve dynamic optimization of the repair strategy and adaptive adjustment of the system.

[0020] Specific implementation method two: such as Figure 1 and Figure 2 As shown, a multi-modal image enhancement system based on multimodal information fusion includes a multimodal image restoration model library, a degradation identification and evaluation module, an intelligent scheduling and reasoning module, and a restoration execution and feedback module. The multimodal image restoration model library integrates dedicated processing models for typical degradation factors. After systematically investigating cutting-edge single-modal processing methods, it further expands to the selection and integration of multimodal image restoration models. Each selected model has been systematically tested on multiple standard datasets, and its objective indicators such as PSNR and SSIM under different degradation levels have been fully recorded. Subjective evaluation data is obtained by combining the visual quality evaluation model to form a multi-dimensional evaluation system. In single-modal inpainting, the system employs advanced models such as SwinIR, Restormer, and MPRNet to handle denoising and super-resolution tasks; specialized models such as RefineDNet and DehazeFormer are used for image dehazing; and models such as RRDNet are introduced to enhance low-light performance. In multi-modal inpainting, the system innovatively integrates cross-modal information: the DVN model is used for visible-near-infrared fusion denoising, effectively avoiding artifacts through depth structure inconsistency priors; the VIFNet model is used for visible-infrared image fusion dehazing, significantly improving detail recovery capabilities under dense fog conditions; the LEDN model is integrated, using depth images to guide low-light enhancement, and a depth boundary awareness module to enhance edge detail representation. This model library adopts a modular architecture design, possessing good scalability and supporting the dynamic integration and continuous updating of new inpainting algorithms. The degradation identification and evaluation module constructs a dataset containing various combinations of degradation types and fine-tunes the DepictQA-Wild visual language model to intelligently identify the degradation type of the input image and assess its severity. The intelligent scheduling and reasoning module, based on a large language model, constructs a decision-making system with experience learning capabilities. The system first explores the policy space and conducts full-sequence repair tests on 16 common degradation combinations, recording the success rate and failure rate of different processing orders to form a structured experience document. During the actual repair process, the large language model dynamically retrieves the experience knowledge base based on the output results of the degradation identification module and infers the optimal repair order and model combination. This module also introduces a semantic reasoning mechanism. When encountering an unseen degradation combination, the large language model can perform analogical reasoning based on similar cases to generate a reasonable repair solution. The repair execution and feedback module is responsible for executing repair tasks and achieving dynamic optimization. According to the repair plan generated by the intelligent scheduling module, the system sequentially calls the corresponding tools in the model library for step-by-step processing. After each repair step is completed, the DepictQA model evaluates the quality of the intermediate results. If a repair step does not meet expectations, the system automatically triggers the rollback mechanism, returns to the previous processing node, and re-plans the strategy.

[0021] Regarding the construction of a multimodal image inpainting model library, although many existing models claim to be optimal on specific tasks, the generalizability of their conclusions still needs careful verification due to differences in experimental settings, datasets, and evaluation metrics. Therefore, we further tested the performance of existing high-performing models on a unified dataset and evaluation system for different types of degradation.

[0022] In addition, current image enhancement systems widely employ single-modal inpainting techniques, such as diffusion models and variational autoencoders. However, these methods still face fundamental limitations when dealing with complex scenes: their inpainting process heavily relies on the image's own pixel information, which may lead to a lack of sufficient information sources or an inability to fully capture all details and features in the image. In low-resolution or irregularly damaged scenes with severely missing information, artifact generation, detail blurring, and structural distortion are prone to occur. MIR (Multimodal Image Inpainting) is a technique that combines multiple modalities (such as visual, text, or other forms of data) for image inpainting. Its goal is to utilize information from different sources to improve the quality and accuracy of inpainting, effectively addressing the problem of information loss when using only a single modality image for inpainting, and significantly enhancing the applicability and processing capabilities of image inpainting technology in complex scenes. In terms of multimodal inpainting model integration, the system fully leverages the complementary characteristics of cross-modal information. The DVN model predicts the depth structure maps of RGB and near-infrared images through a deep structure extraction module, and dynamically adjusts the feature fusion weights based on depth inconsistency patterns, effectively avoiding artifacts caused by structural inconsistencies. VIFNet adopts a three-stage visible-infrared fusion architecture. In the depth feature extraction stage, it extracts multi-scale structural features through an encoder-decoder. In the feature fusion stage, it dynamically adjusts the weights using an inconsistency fusion strategy, and finally restores the fog-free image through supervised learning techniques. The LEDN model innovatively utilizes depth images to assist in low-light enhancement. It extracts object edge information through a depth boundary perception module, uses parallel image encoders and depth encoders for multi-scale feature extraction, and achieves cross-modal feature fusion through a multimodal Fourier convolution module. The introduction of these multimodal models greatly improves the robustness of image inpainting enhancement.

[0023] The implementation of the degradation identification and evaluation module mainly includes two aspects: low-quality dataset generation and DepictQA model fine-tuning.

[0024] Low-quality dataset generation: Complex restoration tasks of varying difficulty are created by combining eight different degradation types. Specifically, 16 degradation combinations were designed for exploration and testing. The combinations were intentionally designed to incorporate common degradation types from real-world scenarios (such as fog and rain, low light and noise). For each degradation type, common degradation algorithms were used, such as bicubic interpolation to downsample the image for low resolution and adding Gaussian or Poisson noise at random scales for noise. The degree of degradation for each type was also randomly generated, categorized into five levels based on severity: very low, low, medium, high, and very high.

[0025] DepictQA Model Fine-tuning: By traversing each degraded image and training the VLM model DepictQA based on the recorded degrade type and corresponding degree of degrade information, the fine-tuned DepictQA can identify the degradation in the image and assess the severity of all types of degradation. This plays a crucial role in determining the degrade type of a given input image, providing feedback on the processing effect of each image processing tool, and selecting the best image restoration tool and restoration order from the toolbox.

[0026] The core of the intelligent scheduling and reasoning module lies in building an experience-based learning and reasoning system based on a large language model. The system first accumulates restoration experience through policy space exploration, conducting full-sequence restoration tests on 16 common degradation combinations. Specifically, the system traverses all possible image processing sequence combinations, including all possible permutations of task execution sequences and model combinations, calls the corresponding enhancement tools to apply to the degraded image, and uses a fine-tuned DepictQA to evaluate the restoration results. If the restored image still exhibits moderate, high, or very high degradation, the processing plan is deemed a failure. The system then calculates the failure rate of each restoration sequence and constructs an experience document containing the optimal processing order for each degradation combination. During actual reasoning, the system calls the large language model interface and uses structured prompts to automatically infer the optimal restoration process and order for the current image based on existing restoration experience. When faced with new degradation combinations, the large language model can also perform analogical reasoning based on similar cases to generate reasonable restoration solutions.

[0027] Repair Execution and Feedback Module: During the repair process based on LLM recommendations, the system will introduce a model rollback and reflection mechanism: when the repair result does not meet expectations, the mechanism can automatically roll back to the previous steps and replan the repair strategy based on the failure feedback, thereby forming a "evaluation-repair-reflection" closed loop with self-optimization capabilities, improving the success rate and robustness of the repair system.

[0028] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent substitutions, and improvements made to the above embodiments without departing from the scope of the present invention, based on the technical essence of the present invention and within the spirit and principles of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. A multi-degraded image enhancement method based on cross-modal co-evolution, characterized in that, The specific steps include: Step 1: Construct a multimodal image restoration model library; systematically test and integrate high-performance single-modal and multimodal restoration models for various typical degradation types; Step 2: Identify the type and severity of image degradation in the input image based on a visual language model; Step 3: Acquire and intelligently schedule image restoration experience, build a priori restoration experience database, and retrieve the experience database based on a large language model to generate the optimal restoration order and model combination; Step 4: Design Reflection - Rescheduling Image Inpainting Mechanism. After the inpainting strategy is executed, the output image is evaluated. If the inpainting does not meet expectations, failure information is automatically fed back and a new inpainting plan is formulated, forming a closed-loop optimization process.

2. The multi-degraded image enhancement method based on cross-modal co-evolution according to claim 1, characterized in that, Step 1, which involves building a multimodal image restoration model library, includes the following steps: Step 101: Under a unified dataset and evaluation system, conduct system performance testing and verification for advanced models with different degradation types; Step 102: Regarding the evaluation index system, a combination of traditional image quality evaluation index scores with subjective visual effects is adopted. Step 103: In terms of single-modal repair model integration, integrate multiple models with better test results, such as SwinIR, RefineDNet, and RRDNet; Step 104: In terms of multimodal restoration, the DVN model is integrated to achieve visible light-near infrared fusion denoising, the VIFNet model is used to complete visible light-infrared image fusion dehazing, and the LEDN model is integrated to guide low-light enhancement using depth images.

3. The multi-degraded image enhancement method based on cross-modal co-evolution according to claim 1, characterized in that, Step 2 specifically includes: Step 201: By combining eight degradation types, a complex dataset containing 16 common real-world degradation combinations was generated, and a severity level of five levels from very low to very high was randomly generated for each degradation. Step 202: Use the dataset to fine-tune the visual language model DepictQA, enabling it to accurately identify all degradation types in the input image and assess their severity, providing a key basis for subsequent repair tool selection and workflow scheduling.

4. The multi-degraded image enhancement method based on cross-modal co-evolution according to claim 1, characterized in that, Step 3 specifically includes: Step 301: On an image dataset containing multiple degradation types, iterate through various image processing strategies and record the output results corresponding to each strategy in detail. Step 302: Use DepictQA to assist in evaluating the repair effects of different strategies, identify successful repair strategies, and statistically summarize the key indicators of repair success rate into a structured experience document. Step 303: Retrieve the experience base based on the large language model, and use the preset prompt words to perform reasoning to generate the optimal repair order and model combination.

5. The multi-degraded image enhancement method based on cross-modal co-evolution according to claim 1, characterized in that, Step 4 specifically includes: Step 401: After the repair strategy is executed, the output image is evaluated in real time using the DepictQA model. If the repair does not meet expectations, it is judged as a failure and the rollback mechanism is triggered. Step 402: The system rolls back to the previous processing stage and feeds back the failure information to the large language model for reflection and strategy rescheduling, generating an adjusted repair plan to achieve dynamic optimization of the repair strategy and adaptive adjustment of the system.

6. A system for implementing any one of the multi-degraded image enhancement methods of claims 1 to 5, characterized in that, It includes a multimodal image restoration model library, a degradation identification and evaluation module, an intelligent scheduling and inference module, and a restoration execution and feedback module; The multimodal image restoration model library integrates dedicated processing models for typical degradation factors. After systematically investigating cutting-edge single-modal processing methods, it further expands to the selection and integration of multimodal image restoration models. Each selected model has been systematically tested on multiple standard datasets, and its objective indicators such as PSNR and SSIM under different degradation levels have been fully recorded. Subjective evaluation data is obtained by combining the visual quality evaluation model to form a multi-dimensional evaluation system. The degradation identification and evaluation module constructs a dataset containing various combinations of degradation types and fine-tunes the DepictQA-Wild visual language model to intelligently identify the degradation type of the input image and assess its severity. The intelligent scheduling and reasoning module, based on a large language model, constructs a decision-making system with experience learning capabilities. The system first explores the policy space and conducts full-sequence repair tests on 16 common degradation combinations, recording the success rate and failure rate of different processing orders to form a structured experience document. During the actual repair process, the large language model dynamically retrieves the experience knowledge base based on the output results of the degradation identification module and infers the optimal repair order and model combination. This module also introduces a semantic reasoning mechanism. When encountering an unseen degradation combination, the large language model can perform analogical reasoning based on similar cases to generate a reasonable repair solution. The repair execution and feedback module is responsible for executing repair tasks and achieving dynamic optimization. According to the repair plan generated by the intelligent scheduling module, the system sequentially calls the corresponding tools in the model library for step-by-step processing. After each repair step is completed, the DepictQA model evaluates the quality of the intermediate results. If a repair step does not meet expectations, the system automatically triggers the rollback mechanism, returns to the previous processing node, and re-plans the strategy.

Images