A multi-modal large model image enhancement method for spatial dim and weak targets

Abstract

The application provides a multi-modal large model image enhancement method for space dark and weak targets, which improves the quality of low signal-to-noise ratio images by fine-tuning a multi-modal large model. The method adopts a graphic-text joint training framework, first constructs a training data set containing original images, enhanced images and prompt information; then expands the input structure of the pre-trained large language model to adapt to the graphic-text dual mode, converts the image enhancement into a graphic-text conditional generation task, and makes the model learn the enhancement semantic rules and structural features through instruction fine-tuning. In the inference stage, the user inputs the original image and the enhancement prompt, and the model can generate the enhancement result, realize the structure recovery, brightness improvement and edge enhancement of the dark and weak area, and significantly improve the visibility and clarity of the small target. Compared with the traditional method, the method has stronger context adaptability and higher target enhancement precision, and can be applied to image processing tasks in complex environments such as space remote sensing, infrared monitoring and small target recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of aerospace technology, and in particular to a multi-modal large model image enhancement method for space dim and weak targets. BACKGROUND

[0002] In the process of space target observation and remote sensing imaging, it is often difficult to identify and detect dim and weak targets. Because of the long imaging distance, low signal-to-noise ratio, small target size, and complex observation environment, the original images obtained often have defects such as insufficient brightness, blurred details, low contrast, and serious noise interference. These problems will lead to a decrease in the accuracy of subsequent target detection, identification, and tracking, thereby affecting aerospace missions, space situation awareness, and infrared monitoring applications.

[0003] Existing image enhancement methods mainly have two categories: one is traditional image processing algorithms, such as histogram equalization, Retinex method, filtering, and multi-scale transformation. These methods are simple to implement, but the enhancement effect depends on manually designed rules, and in complex background and low signal-to-noise ratio environments, dim and weak targets are not adaptable, and details are easily lost or over-enhanced. The second category is deep learning-based image enhancement methods, which usually learn the mapping relationship through convolutional neural networks (CNN) and generative adversarial networks (GAN). This type of method has good results in natural image enhancement, but still has problems such as insufficient training data, limited semantic understanding of the model, and insufficient generalization of the enhancement results in the space dim and weak target scenario.

[0004] In recent years, multi-modal large models have shown strong capabilities in cross-modal understanding and generation. Such models not only can handle large amounts of image and text data, but also can learn complex semantic constraints and generation rules through instruction fine-tuning. Compared with traditional single-modal models, they have more advantages in context understanding and task transfer. In the image enhancement task, the introduction of semantic guidance and cross-modal alignment mechanisms of large models can make the enhancement process no longer limited to low-level feature adjustment, but can combine scene description, target type, and task requirements to make more targeted enhancements, thereby significantly improving the visibility and detection reliability of dim and weak targets.

[0005] In the development of multi-modal large models, the prompt word (Prompt) technology has gradually become an important mechanism to drive model generation and reasoning. The basic principle of prompt words is to guide the model output to meet the expected results through pre-designed text or image input. This way, specific tasks can be controlled more accurately. Multi-modal models will encounter a large amount of image-text data during training, learning the relationship between different modalities. The core of the prompt word technology is to use these relationships to activate the existing knowledge of the model through the design of appropriate prompt inputs, thereby completing the task. Compared with traditional input methods, the prompt word method is more flexible and easier to extend, suitable for different downstream applications.

[0006] Currently, there are many representative multi-modal large models at home and abroad. Qwen2.5-VL is a model released by Ali DAMO Institute, which combines visual encoder and language model, can process mixed input of text and image, is good at image question answering and description task, and performs outstandingly in Chinese scene. DeepSeek-VL2 is a lightweight visual language model, which adopts VisionTransformer architecture, balances performance and speed, and is suitable for deployment on devices with limited resources. InternVL2.5 comes from Shanghai Artificial Intelligence Laboratory, which integrates multiple visual architectures and strengthens the collaborative reasoning of text and image through optimization mechanism, especially suitable for chart and document scenes. LLaMA-3.2V is developed based on Meta's LLaMA-3 language model, has smooth text and image interaction ability, and approaches the level of GPT-4V in English scene. The Phi-3.5 model of Microsoft continues the characteristics of small models with high performance, and performs excellently in image question answering and mathematical visual tasks, indicating that its training data quality and generalization ability are strong.

[0007] The core of multi-modal large model is the cross-modal alignment mechanism between visual encoder and language model. The common structure is composed of ViT and LLaMA, and the fusion of visual and language features is realized through Q-Former module. During inference, the remote sensing image is first encoded into visual token by ViT, and the natural language prompt is encoded into semantic token by LLaMA. Q-Former interacts visual and text information through multi-head attention mechanism to generate unified multi-modal features, and finally outputs the result.

[0008] The combination of prompt word technology and multi-modal model enables the model to better understand and utilize multi-source information. In the text and image related task, it can realize efficient and natural interaction and reasoning, and also provides new possibilities for image enhancement and target recognition.

[0009] Therefore, it is urgent to propose an image enhancement method combining the semantic understanding ability of multi-modal large model and the visual feature extraction ability of deep learning. SUMMARY

[0010] In view of the above problems, the present application provides a multi-modal large model image enhancement method for spatial dim and weak targets, which constructs a multi-modal training data set and fine-tunes the large model, so that it can effectively enhance the spatial dim and weak targets under complex background and low signal-to-noise ratio conditions, thereby providing new technical support for space remote sensing, infrared monitoring and space situation awareness.

[0011] To solve the above problems, the technical scheme adopted by the present application is:

[0012] A multi-modal large model image enhancement method for space dim and weak targets, comprising the following steps:

[0013] Step 1: data construction and preprocessing, collecting space images containing dim and weak targets in aerospace remote sensing and infrared monitoring scenes as original images, using traditional image enhancement algorithms to generate corresponding enhanced images as pseudo labels or reference results, and writing semantic prompt information for each pair of original image and enhanced image to form multi-modal training samples of original image-enhanced image-semantic prompt;

[0014] Step 2: model design and fine-tuning, selecting a pre-trained multi-modal large model, extending its image input module, and using a convolutional neural network or a visual transformer as a feature extractor to convert image representation into a vector representation aligned with text embedding;

[0015] Step 3: model training and optimization, converting the image enhancement task into a text and image conditional generation task, taking the original image and semantic prompt as input, and taking the target enhanced image as output supervision signal, training the model in an instruction fine-tuning manner, introducing a prompt learning mechanism to optimize the learnable prompt vector during training, and introducing a binary cross-entropy loss function to constrain the convergence direction of prompt learning, taking DCE-Net as a lightweight baseline model, and training and optimizing by fusing zero reference loss function;

[0016] Step 4: inference and application, receiving user input of original images to be enhanced and natural language prompts, generating enhanced images through the trained model to improve the visibility and structural clarity of dim and weak targets.

[0017] Preferably, the traditional image enhancement algorithm in step 1 includes at least one of histogram equalization algorithm, Retinex algorithm, and convolutional neural network enhancement algorithm.

[0018] Preferably, the semantic prompt information in step 1 includes at least one of target type, image scene description, and enhancement requirement keyword, which is used to guide the model to realize local reinforcement and background suppression.

[0019] Preferably, the instruction fine-tuning in step 3 adopts a combination of supervised training and cross-modal contrast learning, which improves the multi-modal alignment capability of the model through the consistency constraint of image and text representation.

[0020] Preferably, in the binary cross-entropy loss function in step 3, the positive sample is an image processed by 4x4 averaging, and the label is marked as 0; the negative sample is an image processed by 1:4 downsampling, and the label is marked as 1; the model calculates the probability distribution based on the cosine similarity between the prompt and the image, and distinguishes images of different qualities through the loss function.

[0021] Preferably, the zero-reference loss function in step 3 comprises at least one of exposure control loss, spatial consistency loss, color constancy loss, and illumination smoothing loss.

[0022] Preferably, the exposure control loss promotes exposure correction for underexposed and overexposed regions by setting a desired region average intensity of 0.6.

[0023] The spatial consistency loss ensures the spatial coherence of the enhanced image by maintaining the relative difference between adjacent image regions.

[0024] The color constancy loss follows the gray world assumption to constrain the difference between color channels.

[0025] The illumination smoothing loss constrains the monotonic relationship between adjacent image pixels and acts on the intermediate curve parameter map.

[0026] Preferably, the method has weak supervision and unsupervised characteristics, and positive and negative samples are generated through image processing methods such as average pooling and downsampling, without the need for manual accurate labeling, and the matching of the enhanced result and the task target is ensured through the cross-modal semantic alignment mechanism.

[0027] Preferably, the enhanced image significantly improves the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) indicators compared to the original image, and the detection recall rate in typical dark and weak target scenes is improved by no less than 5%.

[0028] Preferably, the trained model supports running on a GPU platform and can process images in batch mode in offline mode or achieve quasi-real-time enhancement in online mode, and is suitable for aerospace remote sensing, infrared monitoring, and dark and weak target detection scenes, and can improve the visibility and detectability of dark and weak targets in complex backgrounds and low signal-to-noise ratio conditions.

[0029] The beneficial effects of the present application are:

[0030] By converting the image enhancement task into a picture-text conditional generation task, the large language model is fine-tuned in instructions, so that the model can generate enhanced image results according to the input image and language prompts. The enhancement process combines image content understanding, semantic guidance, and visual instruction following, and the model can perform structure recovery, brightness enhancement, and edge enhancement on dark and weak regions. In the inference stage, the user only needs to input the spatial target image and the prompt, and the model can output the enhanced image. Compared with traditional image enhancement methods, this method has stronger context adaptability and higher target enhancement accuracy, and it can be applied to image processing tasks in complex environments such as aerospace remote sensing, infrared monitoring, and small target recognition. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1A flowchart of a multi-modal image enhancement method for space dim and weak target of the present application. DETAILED DESCRIPTION

[0032] To make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below. In the following description, a lot of specific details are set forth in order to fully understand the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0033] Reference Figure 1 A multi-modal large model image enhancement method for space dim and weak target, comprising the following steps:

[0034] Step 1: Collect space images in space remote sensing, infrared monitoring and other scenes, select samples containing dim and weak targets, and generate corresponding enhanced images using traditional image enhancement algorithms (such as histogram equalization, Retinex, convolutional neural network enhancement) as pseudo-labels or reference results.

[0035] At the same time, semantic prompt information is written for each pair of original image and enhanced image. The prompt content can include scene description (such as "low light space target"), target type (such as "small satellite", "weak small light point"), enhancement requirement (such as "improve brightness", "enhance edge") and the like, forming multi-modal training samples of original image-enhanced image-semantic prompt.

[0036] The semantic prompt information includes but is not limited to target type, image scene description, and enhancement requirement keywords, which are used to guide the model to achieve local reinforcement and background suppression in the enhancement process.

[0037] Step 2: Select an existing pre-trained multi-modal large model (such as an open source model with image and text input), and extend the image input module based on it to enable it to receive and encode space image features. This module can use convolutional neural network (CNN) or visual transformer (ViT) as a feature extractor to convert image representation into a vector representation aligned with text embedding.

[0038] Step 3: Convert the image enhancement task into a text and image conditional generation task, and train the model using instruction fine-tuning. Specifically, the original image and semantic prompt are input, and the target enhanced image is output as the supervision signal, so that the model learns to generate enhanced results from joint input of image and text.

[0039] The model fine-tuning adopts a combination of supervised training and cross-modal contrast learning, which improves the multi-modal alignment capability of the model through the consistency constraint of image and text representation.

[0040] In the model fine-tuning training process, the present application further introduces prompt learning and semantic guidance mechanism on the basis of traditional text-image conditional generation framework to enhance the visibility and semantic consistency of dark and weak targets.

[0041] Firstly, the prompt learning method is adopted to optimize a set of learnable prompt vectors (prompt pair) during the training process, so that the semantic information can be dynamically adjusted according to the input image features. Unlike the static artificially set prompts, this mechanism can better capture the prior features of the image and improve the adaptability of the model to dark and weak targets in complex backgrounds by iteratively updating the prompt representation.

[0042] Secondly, the binary cross-entropy loss function (BCE) is introduced in the training to distinguish the positive and negative sample image quality, thereby constraining the convergence direction of prompt learning. The loss function is defined as follows:

[0043] (1)

[0044] Where y represents the sample label, and when the image is a positive sample, it is recorded as 0, that is, the image after 4x4 average processing; when the image is a negative sample, it is recorded as 1, that is, the image after 1:4 downsampling, and ŷ is the probability distribution calculated by the model based on the cosine similarity between the prompt and the image. Through this loss function, the model can learn to distinguish images of different quality during training, ensuring that the prompt optimization direction conforms to the enhancement target.

[0045] In existing research, the curve-based enhancement method has achieved good results in the zero-reference low-light enhancement task. In the design of this method, DCE-Net is introduced as a lightweight baseline model. This model describes the image enhancement process as pixel-by-pixel curve prediction, and directly acts on the input image :

[0046] (2)

[0047] Where, represents the enhancement result of pixel point at the th iteration, represents a set of pixel-by-pixel parameter mappings containing parameters with a value range of , and acts on the input image in an iterative manner.

[0048] This method employs a set of zero-reference loss functions for low-light image enhancement. The following is a brief introduction to the types of losses used: Exposure Control Loss. Spatial consistency loss Color constancy loss and lighting smoothing loss .

[0049] Among them, exposure control loss By setting a desired regional average intensity This facilitates exposure correction for underexposed and overexposed areas.

[0050] (3)

[0051] in, It is the first time to enhance the image. indivual Image patch Yes and no overlap Number of image patches This is the desired average brightness, which is empirically set to 0.6.

[0052] Spatial consistency loss By maintaining the relative differences between adjacent image regions, we ensure enhanced spatial coherence of the image:

[0053] (4)

[0054] in, and These represent the original image and the enhanced image, respectively. It refers to the number of local areas. Indicates area The 4-neighborhood centered on the center.

[0055] Color loss Following the gray-world hypothesis, this is used to constrain the differences between the various color channels:

[0056] (5)

[0057] in, It is an image enhancement Middle color channel The average brightness.

[0058] Light smoothing loss Constrain the monotonic relationship between adjacent image pixels and apply it to the intermediate curve parameter map. :

[0059] (6)

[0060] wherein, is a channel-wise intermediate curve parameter for enhancing the image . denotes a gradient operation.

[0061] The enhancement process integrates the visual feature extraction network for capturing low-level structural information of the image and the multi-modal large model for providing semantic constraints and enhancement generation.

[0062] In addition, the method has weak supervision and unsupervised characteristics. In the absence of large-scale labeled data, the model can still be optimized through prompt learning and semantic guidance. For example, the generation of positive and negative samples only depends on simple image processing methods (average pooling, downsampling, etc.), without the need for accurate manual labeling; and the cross-modal semantic alignment mechanism ensures the matching of the enhancement results and the task target, so that good results can still be achieved under limited data conditions. This makes it have strong practical value in fields such as aerospace remote sensing and infrared monitoring where data acquisition is difficult.

[0063] Step 4: In the inference stage, the user inputs the original image to be enhanced and the natural language prompt (such as "please enhance the clarity of the weak small light points"), and the model can generate the enhanced image. The enhancement result has a significant improvement in peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and other indicators compared to the original image, and performs better in the downstream weak target detection algorithm with an increase in recall rate and a decrease in false detection rate. The method supports GPU platform operation and can process images in batch mode in offline mode or achieve quasi-real-time enhancement in online mode, which is suitable for image preprocessing tasks in ground receiving centers.

[0064] The image enhancement result has an improvement in peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and other indicators compared to the original image, and the detection recall rate in typical dark weak target scenes is improved by no less than 5%.

[0065] By converting the image enhancement task into a picture-text conditional generation task, the large language model is fine-tuned on the instruction, enabling the model to generate enhanced image results according to the input image and language prompts. The enhancement process combines image content understanding, semantic guidance and visual instruction following, and the model can restore the structure, enhance the brightness and enhance the edges of the dark and weak areas. In the inference stage, the user only needs to input the spatial target image and the prompt, and the model can output the enhanced image. Compared with traditional image enhancement methods, this method has stronger context adaptability and higher target enhancement accuracy. It can be applied to image processing tasks in complex environments such as space remote sensing, infrared monitoring and weak target recognition, and can be applied to space remote sensing, infrared monitoring and dark and weak target detection scenes to improve the visibility and detectability of dark and weak targets in complex backgrounds and low signal-to-noise ratio conditions.

[0066] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A multimodal large-scale image enhancement method for spatially faint targets, characterized in that, Includes the following steps: Step 1: Data construction and preprocessing. Collect spatial images containing faint targets in aerospace remote sensing and infrared monitoring scenarios as original images. Use traditional image enhancement algorithms to generate corresponding enhanced images as pseudo-labels or reference results. At the same time, write semantic prompts for each pair of original images and enhanced images to form a multimodal training sample of original image-enhanced image-semantic prompt. The semantic prompt information includes at least one of target type, image scene description, and enhancement requirement keywords, which is used to guide the model to achieve local enhancement and background suppression. Step 2: Model design and fine-tuning. Select a pre-trained multimodal large model, expand its image input module, and use a convolutional neural network or visual transformer as a feature extractor to convert the image representation into a vector representation aligned with the text embedding. Step 3: Model training and optimization. The image enhancement task is transformed into a text-image conditional generation task. The original image and semantic prompts are used as inputs, and the target enhanced image is used as the output supervision signal. The model is trained by instruction fine-tuning. During the training process, a prompt learning mechanism is introduced to optimize the learnable prompt vector. At the same time, a binary cross-entropy loss function is introduced to constrain the convergence direction of prompt learning. DCE-Net is used as a lightweight baseline model, and a zero-reference loss function is fused for training and optimization. The instruction fine-tuning adopts a combination of supervised training and cross-modal contrastive learning, and improves the model's multimodal alignment capability through consistency constraints between image and text representations; In the binary cross-entropy loss function, positive samples are images processed by 4×4 averaging, with a label of 0; negative samples are images downsampled by 1:4, with a label of 1; the model calculates the probability distribution based on the cosine similarity between the prompt and the image, and uses this loss function to distinguish images of different quality. The zero-reference loss function includes at least one of exposure control loss, spatial consistency loss, color constancy loss, and illumination smoothing loss; Step 4: Reasoning and Application. Receive the original image to be enhanced and natural language prompts from the user, and generate the enhanced image through the trained model to improve the visibility and structural clarity of dark targets.

2. The multimodal large-scale image enhancement method for spatially dim targets according to claim 1, characterized in that, The traditional image enhancement algorithms mentioned in step 1 include at least one of histogram equalization algorithm, Retinex algorithm, and convolutional neural network enhancement algorithm.

3. The multimodal large-scale image enhancement method for spatially dim targets according to claim 1, characterized in that, The exposure control loss facilitates exposure correction for underexposed and overexposed areas by setting the average intensity of the desired area to 0.

6. The spatial consistency loss ensures enhanced spatial coherence of the image by maintaining the relative differences between adjacent image regions; The color constancy loss follows the gray-world assumption, constraining the differences between each color channel; The illumination smoothing loss constrains the monotonic relationship between adjacent image pixels and applies to the intermediate curve parameter map.

4. The multimodal large-scale image enhancement method for spatially dim targets according to claim 1, characterized in that, The proposed method features both weak supervision and unsupervised learning. Positive and negative samples are generated through image processing methods such as average pooling and downsampling, eliminating the need for precise manual annotation. A cross-modal semantic alignment mechanism ensures the matching between the enhanced results and the task objectives.

5. The multimodal large-scale image enhancement method for spatially dim targets according to claim 1, characterized in that, The enhanced image significantly improves the peak signal-to-noise ratio and structural similarity index compared to the original image, and the recall rate is improved by no less than 5% in typical dim target scenes.

6. The multimodal large-scale image enhancement method for spatially faint targets according to claim 1, characterized in that, The trained model supports running on GPU platforms and can process images in batches in offline mode or achieve near real-time enhancement in online mode. It is suitable for aerospace remote sensing, infrared monitoring, and low-light target detection scenarios, improving the visibility and detectability of low-light targets under complex backgrounds and low signal-to-noise ratio conditions.

Images