Composite degraded remote sensing image restoration method based on perception-planning-execution mechanism

Abstract

The invention discloses a composite degraded remote sensing image restoration method based on a perception-planning-execution mechanism, which comprises the following steps of: firstly, perceiving a degradation type existing in an input image by using a multi-modal large language model (MLLM), and then planning a proper degraded image restoration path based on an identification result large language model (LLM). And finally, selectively calling related tools from the image recovery tools by the actuator according to the image recovery path to finish the recovery of the degraded remote sensing image. According to the method, a complex degraded image recovery task is decomposed, and the degraded remote sensing image is recovered for multiple times, so that the remote sensing image with composite degradation is recovered, the existing remote sensing image recovery tools for single degradation are directly integrated and called, and the tools do not need to be trained for a specific data set; and meanwhile, an existing image restoration tool can be easily added into a composite degraded remote sensing image restoration task, so that a new degradation type is processed.

Description

Technical Field

[0001] This invention belongs to the field of image restoration technology, and more specifically, relates to a composite degraded remote sensing image restoration method based on a perception-planning-execution mechanism. Background Technology

[0002] Remote sensing images acquired through various satellite and airborne platforms provide crucial data for fields such as Earth environmental observation, urban planning, agriculture, and disaster management. However, due to environmental physical factors or sensor limitations, remote sensing images are often affected by various forms of degradation. Common types of degradation include low light, fog, noise, blurring, and low resolution. These degradations significantly reduce image quality, adversely affecting the effectiveness of human perception and downstream analysis tasks. Therefore, remote sensing image restoration has become a key technology for reconstructing high-quality remote sensing images from degraded images.

[0003] Existing methods for restoring degraded remote sensing images have achieved significant success in addressing single degradation problems using specialized, task-specific models, such as single models for dehazing and low-light enhancement. However, remote sensing images acquired in real-world scenarios are often affected by dynamic and complex environmental conditions, leading to unpredictable compound degradation. Models designed for single degradation problems cannot effectively handle these compound degradation issues.

[0004] Some methods attempt to address specific combinations of degradation types. For example, Zhang et al. developed a unified framework to handle super-resolution and image blurring problems. Cao et al. proposed a method that simultaneously considers haze and noise in the same scene. However, in practical applications, training a separate model for each specific degradation combination significantly increases computational complexity and resource overhead. In recent years, to improve the generalization ability of models, some integrated image restoration methods have been proposed. These methods attempt to train a single model on mixed datasets to solve multiple independent image degradation problems.

[0005] However, real-world degradation patterns are complex and diverse. When faced with complex degradation problems, these methods cannot address multi-task optimization during model training or model scalability issues in practical applications. This results in poor performance on remote sensing image restoration tasks in complex degradation scenarios, falling far short of real-world application requirements.

[0006] Existing methods for restoring degraded remote sensing images have the following drawbacks:

[0007] 1. It does not consider the problem of image composite degradation in complex scenes, and cannot effectively restore remote sensing images with composite degradation;

[0008] 2. The model has low scalability and cannot quickly adapt to new types of image degradation;

[0009] 3. The recovery process is uncontrollable, and it is impossible to manually plan and intervene in the recovery process, which reduces the practicality of the algorithm. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of the prior art and provide a composite degraded remote sensing image restoration method based on a perception-planning-execution mechanism. Based on the perception-planning-execution paradigm, different image restoration tools are automatically executed step by step to intelligently complete the composite degraded remote sensing image restoration task. At the same time, by constructing a novel structural design, existing image restoration models can be quickly integrated without additional training, thereby adapting to new restoration tasks and enhancing scalability.

[0011] To achieve the above-mentioned objectives, the present invention provides a composite degraded remote sensing image restoration method based on a perception-planning-execution mechanism, characterized by comprising the following steps:

[0012] (1) Fine-tuning of multimodal large language model

[0013] Degraded remote sensing images and corresponding degradation type labels are provided to form question-and-answer data for fine-tuning a multimodal large language model, enabling the multimodal large language model to identify the degradation types present in the input degraded remote sensing images;

[0014] (2) Empirical exploration of degraded image restoration path based on large language model

[0015] Implementation exploration: Composite degraded remote sensing images with multiple degradation types are restored according to different image restoration paths, where the image restoration path is the order of the image restoration types, and each image restoration type is an image restoration model for a single degradation type;

[0016] Execution log: Compare the image recovery success rates of different image recovery paths and record the image recovery path with the highest success rate;

[0017] Prior knowledge extraction: Based on the image restoration path with the highest success rate recorded, and using the constructed question-and-answer data, the prior knowledge is extracted from the experience knowledge used to guide the planning of the image restoration path.

[0018] (3) Restoration of composite degraded remote sensing images

[0019] First, the composite degraded remote sensing image is fed into a multimodal large language model for perception, identifying the degradation types present in the composite remote sensing image. Then, the large language model plans an image restoration path based on the perceived degradation types, and finally, the composite degraded remote sensing image restoration is performed.

[0020] In the image restoration path, an image restoration tool is randomly selected from the toolbox corresponding to the first image restoration type to restore the composite degraded remote sensing image. Then, a multimodal large language model is used for evaluation to determine whether the image restoration type has been successfully restored. If successful, proceed directly to the next step: randomly select an image restoration tool from the toolbox corresponding to the second image restoration type to restore the image. If unsuccessful, select another image restoration tool from the same toolbox to restore the image. Repeat this process until successful, then proceed to the next step: randomly select an image restoration tool from the toolbox corresponding to the second image restoration type to restore the image. This continues until all image restoration types in the image restoration path have been restored, resulting in the restored remote sensing image.

[0021] In the process of restoring composite degraded remote sensing images, if the restoration of a certain image restoration type fails, that is, all restored composite degraded remote sensing images fail to meet the restoration index in the corresponding degradation type, then return to step (2). When using the large language model to replan the execution path, the system will prompt the large language model of the currently failed image restoration path and require it to replan a new image restoration path based on its own reasoning, according to prior knowledge and the image restoration path that failed this time.

[0022] The objective of this invention is achieved as follows.

[0023] The present invention provides a composite degraded remote sensing image restoration method based on a perception-planning-execution mechanism. First, a multimodal large language model (MLLM) is used to perceive the degradation type in the input image. Then, based on the recognition results, the large language model (LLM) plans a suitable degraded image restoration path. Finally, the executor selectively calls relevant tools from the image restoration tools according to the image restoration path to complete the restoration of the degraded remote sensing image.

[0024] The present invention has the following beneficial effects:

[0025] 1. Compared with methods for restoring single degraded images, this invention restores remote sensing images with complex degradation by decomposing the task of restoring complex degraded images and performing multiple restorations on the degraded remote sensing images.

[0026] 2. Compared to remote sensing image restoration methods trained on specific datasets, this invention directly integrates and utilizes existing remote sensing image restoration tools designed for single types of degradation, which do not require training on specific datasets. Furthermore, this structural design allows for the easy addition of existing image restoration tools to complex degradation remote sensing image restoration tasks, thereby handling new degradation types. Attached Figure Description

[0027] Figure 1This is a schematic diagram of the structure of a remote sensing image restoration system designed under the composite degraded remote sensing image restoration method based on the perception-planning-execution mechanism of the present invention;

[0028] Figure 2 This is a flowchart of a specific implementation of the composite degraded remote sensing image restoration method based on the perception-planning-execution mechanism of the present invention;

[0029] Figure 3 This is a schematic diagram of the empirical exploration process;

[0030] Figure 4 These are the restoration results images from different methods. For the first row of images, the images from left to right contain the following degradation types: (low-light, low-resolution), (low-light, noise), (haze, low-resolution), (haze, blur), (low-light, blur), and (blur, noise, low-resolution); where: LL = low-light, LR = low-resolution, N = noise, H = haze, and B = blur.

[0031] Figure 5 This is a comparison chart of recovery results in a real-world scenario. Detailed Implementation

[0032] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0033] Figure 1 This is a schematic diagram of the structure of a remote sensing image restoration system designed under the composite degradation remote sensing image restoration method based on the perception-planning-execution mechanism of the present invention.

[0034] In this embodiment, the restoration of composite degraded remote sensing images is divided into automatic and manual modes. In automatic mode, the degradation type present in the composite degraded remote sensing image is first detected by a multimodal large language model (MLLM). Then, the large language model (LLM) is used to plan a suitable image restoration path. Finally, the executor calls the restoration tools in the toolbox corresponding to the image restoration type according to the image restoration path to perform image restoration. In manual mode, the user defines the restoration process, and the multimodal large language model (MLLM) provides system interaction, determining the degradation type and performing image restoration according to the user's instructions. Thus, this invention supports user-defined restoration, allowing users to determine the image degradation restoration type and restoration order, enhancing the system's practicality.

[0035] Figure 2 This is a flowchart of a specific implementation of the composite degraded remote sensing image restoration method based on the perception-planning-execution mechanism of the present invention.

[0036] In this embodiment, as Figure 2 As shown, the composite degraded remote sensing image restoration method based on the perception-planning-execution mechanism of the present invention includes the following steps:

[0037] Step S1: Fine-tuning of the multimodal large language model

[0038] Degraded remote sensing images and corresponding degradation type labels are provided to form question-and-answer data for fine-tuning a multimodal large language model, enabling the model to identify the degradation types present in the input degraded remote sensing images.

[0039] In this embodiment, image evaluation based on a multimodal large language model (MLLM) is used as the perceptual component. To mitigate domain disparities, a low-rank adaptive technique is employed for fine-tuning. The fine-tuned model is represented as follows:

[0040]

[0041] in, Θ represents the weights of the multimodal large language model to be fine-tuned. q ,Θ k ,Θ v ,Θ o These represent the parameters of the query, key, value, and output projection matrix in the model's self-attention mechanism. During fine-tuning, degraded remote sensing images and corresponding degradation type labels constitute question-answering data for fine-tuning the multimodal large language model. For example, the question is: [Degraded Image]#How severe is the degradation of this remote sensing image? Please choose one word from the following options: very low, low, moderate, high, very high. The answer is: [Degradation Level]. During inference, the multimodal large language model considers images with a degradation level of moderate or higher to be subject to further processing.

[0042] Step S2: Empirical exploration of degraded image restoration path based on large language model

[0043] Due to the complex relationships between composite degradation types and the mutual influence between different model tools, different image restoration paths can lead to different results. Finding a suitable restoration path is itself a complex decoupling problem. The powerful reasoning capabilities of large language models can help us handle this complex restoration path planning problem. In this embodiment, the popular GPT-4 is used as the large language model (LLM) component of this invention.

[0044]

[0045]

[0046] Table 1

[0047] Table 1 illustrates the impact of different inputs on LLM decision-making. As shown in Table 1, when posing questions to the LLM, even when only the order of questions involving degraded types is changed, the LLM's response varies. This demonstrates that GPT-4 cannot reliably solve these problems because it merely guesses. LLMS possesses only general planning knowledge but cannot truly provide an executable plan for our complex system. To address this issue, we need to provide the LLM with some prior knowledge of image restoration, thereby leveraging the LLM's reasoning capabilities to plan reasonable restoration paths.

[0048] To obtain comprehensive prior knowledge and minimize subjective human experience, this invention designs an empirical exploration process. For example... Figure 3 As shown. This invention uses LLM to evaluate the success of image restoration. We calculate the success rate of each operation sequence and summarize it into the system described in the patent, which learns from the execution history to guide subsequent reasoning processes. Specifically, we execute all possible restoration paths using corresponding tools and evaluate them using a fine-tuned multimodal large model. After obtaining this data, we prompt the LLM to summarize it and extract empirical knowledge to guide path planning. For example, when we prompt the LLM again to arrange the order of noise removal and dehazing, the LLM can reasonably provide a consistent restoration process with the help of prior knowledge. Specifically, the empirical exploration of degraded image restoration paths based on large language models includes the following steps:

[0049] Implementation exploration: Composite degraded remote sensing images with multiple degradation types are restored according to different image restoration paths, where the image restoration path is the order of the image restoration types, and each image restoration type is an image restoration model for a single degradation type.

[0050] Execution log: Compare the image recovery success rates of different image recovery paths and record the image recovery path with the highest success rate.

[0051] Prior knowledge extraction: Based on the image restoration path with the highest success rate recorded, and using the constructed question-and-answer data, the prior knowledge is extracted from the experience knowledge used to guide the planning of image restoration paths.

[0052] Step S3: Restoration of composite degraded remote sensing images

[0053] First, the composite degraded remote sensing image is fed into a multimodal large language model for perception, identifying the degradation types present in the composite remote sensing image. Then, the large language model plans an image restoration path based on the perceived degradation types, and finally, the composite degraded remote sensing image restoration is performed.

[0054] In the image restoration path, an image restoration tool is randomly selected from the toolbox corresponding to the first image restoration type to perform image restoration on the composite degraded remote sensing image. Then, a multimodal large language model is used for evaluation to assess whether the image restoration type has been successfully restored. If successful, proceed directly to the next step: randomly select an image restoration tool from the toolbox corresponding to the second image restoration type to perform image restoration. If unsuccessful, select another image restoration tool from the same toolbox to perform restoration, and repeat until successful, then proceed to the next step: randomly select an image restoration tool from the toolbox corresponding to the second image restoration type to perform image restoration, and so on until all image restoration types in the image restoration path have been restored, resulting in the restored remote sensing image. This step is called the "thinking" process.

[0055] In the process of restoring composite degraded remote sensing images, if the restoration of a certain image restoration type fails, that is, all restored composite degraded remote sensing images fail to meet the restoration index for the corresponding degradation type, then return to step S2. When replanning the execution path using the large language model, the system will prompt the large language model with the currently failed image restoration path and require it to replan a new image restoration path based on prior knowledge and the failed image restoration path, according to its own reasoning.

[0056] In this embodiment, as Figure 1 As shown, the actuator performs the degraded image restoration task according to the degraded image restoration path. The actuator includes a set of image restoration (IR) tools and restoration operations. To ensure the robustness and stability of the system under various degradation conditions, two key mechanisms are introduced into the actuator: a think-and-go mechanism and a rollback mechanism.

[0057] The executor will use MLLM to evaluate the recovery results of the IR tool. If the currently called IR tool does not perform well (the degree of degradation is greater than low), the executor will intelligently call other suitable IR tools.

[0058] Rollback. If all tools in a toolbox fail to restore the image, we prioritize images where the established restoration order is unsuitable for the current image. The system state is rolled back to the previous restoration stage. Failed processes are logged, and the system prompts the Large Language Model (LLM) to incorporate these failures into the new restoration plan, thus avoiding the repetition of ineffective strategies and adaptively replanning restoration steps. In more extreme cases, if all restoration processes and image restoration tools fail to yield satisfactory results, the execution module will compare all attempts using a Multimodal Large Language Model (MLLM) and select the best-performing image as the final output.

[0059] experiment

[0060] Based on five typical degradation types—darkness, motion blur, low resolution, noise, and haze—we considered two and three types of combined degradation according to common natural conditions, constructing a total of 12 combined degradation scenarios. For example, low-light conditions often include noise, and blur and low resolution frequently coexist.

[0061] Degradation quantity Have any empirical explorations been conducted? Composite degradation type 2 yes haze + motion blur 2 yes Motion blur + low resolution 2 yes haze+noise 2 yes noise + low resolution 2 yes dark+noise 2 yes dark + low resolution 2 no motion blur + dark 2 no haze + low resolution 2 no motion blur+noise 3 no haze + noise + low resolution 3 no motion blur+noise+low resolution 3 no dark + motion blur + noise

[0062] Table 2

[0063] Currently, we have not found a method for restoring remote sensing images with multiple composite degradations. Therefore, we use the state-of-the-art all-in-one approach for result comparison. It should be noted that our purpose is not to prove that any particular method of this invention is superior, but rather to demonstrate the capabilities of our framework: by integrating multiple image restoration models focused on a single task, this invention can not only handle mixed degradation problems, but its restoration effect is comparable to integrated restoration solutions in recent years. At the same time, the framework also has high scalability and can seamlessly integrate existing mature restoration models, thereby further improving the overall system performance.

[0064] This invention can effectively address the complex degradation problems existing in remote sensing images, achieving excellent image restoration results. The image restoration effect achieved by this invention is as follows: Figure 4 As shown in the figure, the first row is the input degraded image, the last row is the restoration result obtained by our invention, and the rest are restoration results obtained by comparison methods.

[0065] For performance evaluation, we used six commonly used quantitative metrics: PSNR, SSIM, LIPLS, MANIQA, CLIPQA, and MUSIQ. The first three are full-reference metrics, while the latter three are no-reference metrics. Specifically, PSNR is used to assess pixel-level distortion, SSIM measures structural similarity, and LIPLS, MANIQA, CLIPQA, and MUSIQ are learning-based perceptual quality assessment metrics designed to better align with human visual preferences and subjective quality judgments. Among these metrics, a lower LIPLS value indicates better restoration, while higher values ​​for the other five metrics indicate superior image restoration quality.

[0066] method years PSNR SSIM LPIPS MANIQA CLIPQA MUSIQ AirNet cvpr2022 19.8308 0.5973 0.3959 0.2579 0.3544 51.5090 promptir NIPS 2023 20.2363 0.5620 0.4575 0.2078 0.3205 39.2085 daclip ICLR2024 19.1690 0.6203 0.3945 0.2589 0.3998 43.8691 RAM-p ECCV2024 20.5311 0.6076 0.4089 0.2341 0.3796 43.6778 RAM-s ECCV2024 20.6347 0.5985 0.4764 0.2122 0.2968 39.5111 Ada4dir IF2025 20.5860 0.5821 0.5062 0.1851 0.2729 34.9212 This invention 20.6559 0.6279 0.3699 0.2984 0.4015 57.6546

[0067] Table 3

[0068] Table 3 presents a quantitative comparison on the synthetic dataset. The present invention achieved the best recovery results on three full-reference metrics and three non-reference metrics based on perceived quality.

[0069] Real-world test results

[0070] In real-world scenarios, image degradation can be more complex and varied. To more comprehensively verify the generalization of this invention on real degraded remote sensing images, we collected real degraded images from datasets goo and MUPSW and conducted verification.

[0071] Methods Venue MANIQA↑ CLIPQA↑ MUSIQ↑ ANFNet ECCV'2022 0.2059 0.2968 28.6839 Restormer CVPR'2022 0.2062 0.2923 28.6533 LedNet ECCV'2022 0.2034 0.2852 29.3074 End TCSVT′2024 0.2080 0.2799 29.1304 AirNet CVPR'2022 0.2123 0.2872 28.6566 Promptir NIPS 2023 0.2058 0.2953 28.6181 Daclip ICLR'2024 0.2047 0.2938 28.6335 RAM-p ECCV'2024 0.2048 0.3009 28.7480 RAM-s ECCV'2024 0.2105 0.2793 29.0502 Ada4dir IF'2025 0.2009 0.2903 28.8680 This invention 0.2935 0.5278 58.0354

[0072] Table 4

[0073] Table 4 shows the quantitative comparison results of three no-reference indicators in real-world degradation scenarios. Figure 5 The table shows a comparison of recovery results in real-world scenarios. Figure 5 As we can see, the present invention (Ours) has achieved the best recovery results in terms of both evaluation metrics and visual aspects.

[0074] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A composite degraded remote sensing image restoration method based on a perception-planning-execution mechanism, characterized in that, Includes the following steps: (1) Fine-tuning of multimodal large language model Degraded remote sensing images and corresponding degradation type labels are provided to form question-and-answer data for fine-tuning a multimodal large language model, enabling the multimodal large language model to identify the degradation types present in the input degraded remote sensing images; (2) Empirical exploration of degraded image restoration path based on large language model Implementation exploration: Composite degraded remote sensing images with multiple degradation types are restored according to different image restoration paths, where the image restoration path is the order of the image restoration types, and each image restoration type is an image restoration model for a single degradation type; Execution log: Compare the image recovery success rates of different image recovery paths and record the image recovery path with the highest success rate; Prior knowledge extraction: Based on the image restoration path with the highest success rate recorded, and using the constructed question-and-answer data, the prior knowledge is extracted from the experience knowledge used to guide the planning of the image restoration path. (3) Restoration of composite degraded remote sensing images First, the composite degraded remote sensing image is fed into a multimodal large language model for perception, identifying the degradation types present in the composite remote sensing image. Then, the large language model plans an image restoration path based on the perceived degradation types, and finally, the composite degraded remote sensing image restoration is performed. In the image restoration path, an image restoration tool is randomly selected from the toolbox corresponding to the first image restoration type to restore the composite degraded remote sensing image. Then, a multimodal large language model is used for evaluation to determine whether the image restoration type has been successfully restored. If successful, proceed directly to the next step: randomly select an image restoration tool from the toolbox corresponding to the second image restoration type to restore the image. If unsuccessful, select another image restoration tool from the same toolbox to restore the image. Repeat this process until successful, then proceed to the next step: randomly select an image restoration tool from the toolbox corresponding to the second image restoration type to restore the image. This continues until all image restoration types in the image restoration path have been restored, resulting in the restored remote sensing image. In the process of restoring composite degraded remote sensing images, if the restoration of a certain image restoration type fails, that is, all restored composite degraded remote sensing images fail to meet the restoration index in the corresponding degradation type, then return to step (2). When using the large language model to replan the execution path, the system will prompt the large language model of the currently failed image restoration path and require it to replan a new image restoration path based on its own reasoning, according to prior knowledge and the image restoration path that failed this time.

2. The composite degraded remote sensing image restoration method based on a perception-planning-execution mechanism according to claim 1, characterized in that, Fine-tuning of a multimodal large language model using low-rank adaptive techniques.

3. The composite degraded remote sensing image restoration method based on the perception-planning-execution mechanism according to claim 1, characterized in that, GPT-4 was used as the large language model.

4. The composite degraded remote sensing image restoration method based on the perception-planning-execution mechanism according to claim 1, characterized in that, It also includes a manual mode, where the user customizes the recovery process, and a multimodal large language model provides system interaction to determine the degradation type and restore the degraded image according to the user's instructions.

Images