Infrared-visible light image fusion method, device and system

Abstract

The invention discloses an infrared-visible light image fusion method, device and system, and relates to the technical field of computer vision and cross-modal intelligent fusion. The method comprises the following steps: constructing a positive and negative semantic prompt text for a downstream task; inputting the infrared image and the visible light image into a preset fusion network model to obtain a fusion image, obtaining a visual fusion index based on semantic prompt, and performing normalization processing on the visual fusion index; based on the normalized visual fusion index, calculating to obtain a fusion score of the fusion image under the positive and negative semantic prompt texts, and taking the positive and negative semantic prompt texts corresponding to the maximum value of the fusion score as an optimal prompt pair; and obtaining an optimal fusion image based on the optimal prompt pair. According to the method, double-index collaborative design of fusion effects and task requirements and a semantic guidance mechanism with dynamic semantic selection can be realized, and generalization ability in a complex scene is achieved.

Description

Technical Field

[0001] This application relates to the field of computer vision and cross-modal intelligent fusion technology, and in particular to a method, apparatus and system for fusing infrared-visible light images. Background Technology

[0002] Infrared-Visible Image Fusion (IVIF) is a core research area in computer vision and cross-modal intelligent perception. With breakthroughs in deep learning technology and improved hardware computing power, deep learning-based IVIF methods have become mainstream. Researchers have leveraged the powerful feature learning capabilities of neural networks to propose various targeted improvement models, such as DenseFuse, CCAFusion, and FusionGAN. Furthermore, the combination of attention mechanisms and deep learning has become an important optimization direction. In recent years, the Transformer architecture has also been introduced into this field, using self-attention mechanisms to mine long-distance feature dependencies and further improve cross-modal feature alignment. Meanwhile, U-Net and its improved encoder-decoder architecture have become mainstream benchmark models for fusion tasks due to their excellent feature extraction and detail recovery capabilities, often combined with residual connections and multi-scale fusion modules to optimize feature transfer efficiency.

[0003] However, existing technologies still face many core challenges that need to be addressed, making it difficult to meet the application needs of complex real-world scenarios: (1) Imbalance between vision and task performance: Existing methods focus on improving the objective visual indicators of the fusion graph or preserving single modal features, lacking semantic collaborative design with downstream tasks, resulting in the contradiction of "excellent fusion effect but mediocre detection / segmentation performance" or "acceptable task performance but degraded fusion quality", making it difficult to achieve optimal synergy between the two indicators; (2) Insufficient design of semantic guidance mechanism: Although some methods attempt to introduce semantic constraints, they mostly adopt single prompts or simple semantic matching strategies, failing to fully combine scene sub-tasks (vehicle detection, lane marking recognition) and modal advantages to design refined constraints, resulting in the fusion features being out of sync with task requirements and weak cross-scene generalization ability.

[0004] The invention patent with authorization announcement number CN119359559B proposes a semantically aware visible light and infrared image fusion method. This method converts the infrared image into a single-channel grayscale image and the visible light image into YCBCR format, using only the Y channel to obtain a Y-channel image. The grayscale image and the Y-channel image are then input into a preset fusion network model to obtain a single-channel fused image. While this method can aggregate local and global information, it lacks a coordinated design for fusion effect and task performance, and its semantic guidance mechanism is insufficient, failing to meet the generalization requirements for complex scenarios. Summary of the Invention

[0005] Based on this, this application addresses the problem of the lack of "vision-task" collaborative design and semantic guidance mechanisms in the prior art, and provides an infrared-visible light image fusion method, apparatus, and system, the specific technical solution of which is as follows: In a first aspect, this application proposes a method for fusing infrared and visible light images, comprising the following steps: S1: Construct positive and negative semantic prompt text for downstream tasks; S2: Input the infrared image and the visible light image into a preset fusion network model to obtain a fused image. Calculate a visual fusion index based on semantic prompts using the infrared image, the visible light image, and the fused image. Then, normalize the visual fusion index. S3: Calculate the fusion score of the fused image under the positive and negative semantic prompt text based on the visual fusion index processed by normalization, and take the positive and negative semantic prompt text corresponding to the maximum value of the fusion score as the optimal prompt pair; S4: The optimal prompt is used as the training set for the preset fusion network model. The infrared image and the visible light image are input into the trained preset fusion network model to obtain the optimal fused image.

[0006] Furthermore, the preset fusion network model mentioned in step S2 includes the PromptFusion model. The PromptFusion model adopts a two-stage improved U-net architecture that includes a feature decomposition stage and a semantic fusion stage. The feature decomposition stage includes a high-frequency encoder and a low-frequency encoder, and both the encoder and the decoder adopt a three-layer stacked structure of convolutional layer, BatchNorm layer and ReLU layer.

[0007] Furthermore, the semantic prompts in step S2 include the positive and negative semantic prompt texts.

[0008] Furthermore, the visual fusion metrics mentioned in step S2 include information entropy, standard deviation, spatial frequency, mutual information, visual information fidelity, and structural similarity index, which are calculated as follows: Information entropy:

[0009] Where L represents the total number of gray levels in the fused image; Indicates grayscale value The probability of a pixel appearing in the fused image. ,in, The grayscale value is i The number of pixels; The total number of pixels in the fused image. MThis represents the number of pixels on the longer side of the fused image. N This represents the number of pixels on the wide side of the fused image; Standard deviation:

[0010] For the fused image in coordinates The pixel grayscale value at that location; The average grayscale value of the fused image; Spatial frequency:

[0011] in, and These represent the horizontal and vertical spatial frequencies, respectively, used to measure the drastic changes in grayscale in the horizontal and vertical directions. The expressions are as follows:

[0012]

[0013] Mutual information:

[0014] in This represents the mutual information between the visible light image and the fused image; The mutual information between the infrared image and the fused image is expressed as follows:

[0015]

[0016] in This represents the joint probability density of the visible light image and the fused image. The joint probability density of the infrared image and the fused image. , , These are the edge probability densities of the visible light image, the infrared image, and the fused image, respectively. Visual information fidelity:

[0017] K is the number of sub-bands in the visible light image or the infrared image after wavelet transform. The variance of the information in the k-th sub-band of the visible light image or the infrared image is used to measure the effective visual information. The noise variance of the visible light image or the infrared image in the kth sub-band is used to measure distortion information.

[0018] Structural similarity index: Calculated by summing the structural similarity indices of the visible light image and the fused image, and the structural similarity indices of the infrared image and the fused image, and taking the average.

[0019] in This represents the structural similarity index between the visible light image and the fused image. The structural similarity index between the infrared image and the fused image is represented by the following expression:

[0020] Wherein, X represents the visible light image or the infrared image; Y represents the fused image; The local average grayscale value of the visible light image or the infrared image; The local average grayscale value of the fused image; The local variance of the visible light image or the infrared image; The local variance of the fused image; The local covariance of the visible light image and the fused image, or the infrared image and the fused image; It is a constant.

[0021] Furthermore, the specific content of the normalization process in step S2 is as follows: No. i Infrared image, visible light image and fused image are processed J The set of visual fusion metrics obtained from positive and negative semantic cue texts is denoted as . , , , , ,

[0022] The normalized expression for information entropy is:

[0023] Similarly, the standard deviation, spatial frequency, mutual information, visual information fidelity, and structural similarity index obtained by normalization are denoted as follows: .

[0024] Furthermore, the calculation method for the fusion score in step S3 is as follows: No. i The infrared image, visible light image, and fused image are processed by the first... j The formula for calculating the fusion score of positive and negative semantic cue texts is as follows:

[0025] in , , , , , These represent the calculated weights of the normalized information entropy, standard deviation, spatial frequency, mutual information, visual information fidelity, and structural similarity index, respectively.

[0026] Secondly, this application proposes an infrared-visible light image fusion device, including a text construction module, an image fusion module, a semantic prompt selection module (SPS module), and an optimal training module. The text construction module is used to construct positive and negative semantic prompt texts for downstream tasks. The image fusion module is used to input infrared and visible light images into a preset fusion network model to obtain a fused image. The semantic prompt selection module is used to calculate a visual fusion index based on semantic prompts from the infrared image, the visible light image, and the fused image. The visual fusion index is normalized, and a fusion score of the fused image under the positive and negative semantic prompt texts is calculated based on the normalized visual fusion index. The positive and negative semantic prompt texts corresponding to the maximum value of the fusion score are taken as the optimal prompt pair. The optimal training module is used to train the preset fusion network model using the optimal prompt pair as the training set, and input the infrared image and the visible light image into the trained preset fusion network model to obtain the optimal fused image.

[0027] Thirdly, this application proposes an infrared-visible light image fusion system, including a memory and a processor; the memory is used to store instructions; the processor is used to operate according to the instructions to execute the infrared-visible light image fusion method proposed in this application.

[0028] This application proposes an infrared-visible light image fusion method, apparatus and system. The method realizes the collaborative design of fusion effect and task requirements, as well as a semantic guidance mechanism with dynamic semantic selection, and has the ability to generalize in complex scenarios. Attached Figure Description

[0029] Figure 1 This is a flowchart of an infrared-visible light image fusion method according to an embodiment of this application; Figure 2 This is a framework diagram of the semantic selection process in the infrared-visible image fusion process in the embodiments of this application; Figure 3 This is a diagram of the codec-based fusion inference framework in the embodiments of this application; Figure 4This is a comparison diagram of the simulation effects of semantic selection and non-semantic selection in the embodiments of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0031] Example 1 This embodiment provides a method for fusing infrared and visible light images, and the flowchart of the method is as follows. Figure 1 As shown, it includes the following steps: S1: Construct positive and negative semantic prompt text for downstream tasks; S2: Input the infrared image and the visible light image into a preset fusion network model to obtain a fused image. Calculate a visual fusion index based on semantic prompts using the infrared image, the visible light image, and the fused image. Then, normalize the visual fusion index. S3: Calculate the fusion score of the fused image under the positive and negative semantic prompt text based on the visual fusion index processed by normalization, and take the positive and negative semantic prompt text corresponding to the maximum value of the fusion score as the optimal prompt pair; S4: The optimal prompt is used as the training set for the preset fusion network model. The infrared image and the visible light image are input into the trained preset fusion network model to obtain the optimal fused image.

[0032] This method designs a refined semantic prompting system with multiple sets of scene adaptations, transforming the semantic requirements of downstream tasks into quantifiable constraint signals. This guides the model to accurately align with the semantic requirements of tasks while retaining its visual advantages, achieving a synergistic improvement in the visual quality of the fused graph and the performance of downstream tasks. It significantly enhances the robustness and cross-scene adaptability of the model in scenarios such as nighttime road monitoring and security in complex environments.

[0033] Example 2 This embodiment provides a method for fusing infrared and visible light images, including the following specific details: S1: Construct positive and negative semantic prompt text for downstream tasks; In one specific embodiment, the downstream tasks include object detection and semantic segmentation.

[0034] S2: Input the infrared image and the visible light image into a preset fusion network model to obtain a fused image. Calculate a visual fusion index based on semantic prompts using the infrared image, the visible light image, and the fused image. Then, normalize the visual fusion index. It should be noted that the preset fusion network model mentioned in step S2 includes the PromptFusion model. The PromptFusion model adopts a two-stage improved U-net architecture that includes a feature decomposition stage and a semantic fusion stage. The feature decomposition stage includes high-frequency and low-frequency encoders and decoders, and both the encoder and decoder adopt a three-layer stacked structure of convolutional layers, BatchNorm layers, and ReLU layers.

[0035] The semantic prompts in step S2 include the positive and negative semantic prompt texts. The visual fusion metrics include information entropy, standard deviation, spatial frequency, mutual information, visual information fidelity, and structural similarity index, and their calculation methods are as follows: Information entropy:

[0036] Where L represents the total number of gray levels in the fused image; Indicates grayscale value The probability of a pixel appearing in the fused image. ,in, The grayscale value is i The number of pixels; The total number of pixels in the fused image. M This represents the number of pixels on the longer side of the fused image. N This represents the number of pixels on the wide side of the fused image; Standard deviation:

[0037] For the fused image in coordinates The pixel grayscale value at that location; The average grayscale value of the fused image; Spatial frequency:

[0038] in, and These represent the horizontal and vertical spatial frequencies, respectively, used to measure the drastic changes in grayscale in the horizontal and vertical directions. The expressions are as follows:

[0039]

[0040] Mutual information:

[0041] in This represents the mutual information between the visible light image and the fused image; The mutual information between the infrared image and the fused image is expressed as follows:

[0042]

[0043] in This represents the joint probability density of the visible light image and the fused image. The joint probability density of the infrared image and the fused image. , , These are the edge probability densities of the visible light image, the infrared image, and the fused image, respectively. Visual information fidelity:

[0044] K is the number of sub-bands in the visible light image or the infrared image after wavelet transform. The variance of the information in the k-th sub-band of the visible light image or the infrared image is used to measure the effective visual information. The noise variance of the visible light image or the infrared image in the kth sub-band is used to measure distortion information.

[0045] Structural similarity index: Calculated by summing the structural similarity indices of the visible light image and the fused image, and the structural similarity indices of the infrared image and the fused image, and taking the average.

[0046] in This represents the structural similarity index between the visible light image and the fused image. The structural similarity index between the infrared image and the fused image is represented by the following expression:

[0047] Wherein, X represents the visible light image or the infrared image; Y represents the fused image; The local average grayscale value of the visible light image or the infrared image; The local average grayscale value of the fused image; The local variance of the visible light image or the infrared image; The local variance of the fused image; The local covariance of the visible light image and the fused image, or the infrared image and the fused image; It is a constant.

[0048] S3: Calculate the fusion score of the fused image under the positive and negative semantic prompt text based on the visual fusion index processed by normalization, and take the positive and negative semantic prompt text corresponding to the maximum value of the fusion score as the optimal prompt pair; It should be noted that the calculation method for the fusion score in step S3 is as follows: i The infrared image, visible light image, and fused image are processed by the first... j The formula for calculating the fusion score of positive and negative semantic cue texts is as follows:

[0049] in , , , , , The weights for normalized information entropy, standard deviation, spatial frequency, mutual information, visual information fidelity, and structural similarity index are respectively represented in the semantic selection framework diagram of the infrared-visible image fusion process, as shown below. Figure 2 As shown, SPS is the semantic selection module.

[0050] S4: The optimal prompt is used as the training set for the preset fusion network model. The infrared image and the visible light image are input into the trained preset fusion network model to obtain the optimal fused image.

[0051] In one specific embodiment, the codec-based fusion inference framework is as follows: Figure 3 As shown.

[0052] Example 3 This embodiment verifies the infrared-visible light image fusion method proposed in this application through simulation experiments. The specific details are as follows: This embodiment focuses on three core aspects: dataset configuration, model training, and testing and evaluation, ensuring the feasibility of the method and the comprehensiveness of performance verification.

[0053] In terms of dataset setup and preprocessing, to simulate the differences in data sources across sensors and environments in real-world scenarios, the training set incorporates images from two public datasets, RoadScene and M3FD, covering infrared-visible image pairs for scenarios such as nighttime roads, complex security, and outdoor remote sensing. The test set uses images from the TNO and MSRS datasets, including complex scenarios such as low light, rain and fog interference, and thermal noise superposition. During the preprocessing stage, all images were standardized to 640×512 pixels, infrared images were normalized to grayscale, and visible light images retained RGB three-channel normalization. Data augmentation strategies such as random flipping, ±15° rotation, and Gaussian noise with a variance of 0.01 were used to expand the diversity of training data and improve the model's robustness against interference.

[0054] The model training phase adopts the improved U-Net architecture of "feature decomposition-semantic fusion" in PromptFusion. The feature decomposition network includes high- and low-frequency encoders and decoders, both of which adopt a 3-layer structure of convolution + BatchNorm + ReLU. The feature fusion network embeds a coordinate attention module and achieves dual-modal feature adaptive fusion through concatenation + 1×1 convolution. The semantic prompt encoding relies on the pre-trained CLIP model (ViT-B / 32 version) and fine-tunes the top layer to adapt to the task. For training configuration, the Adam optimizer (β1=0.9, β2=0.999, weight decay 1e-5) was selected, with an initial learning rate of 0.001 and linear decay. The visual loss weight was 0.8, and the semantic cue loss weight was 1.2. Regularization and barrier function coefficients were dynamically adjusted. The batch size was set to 8, and the total number of iterations was 300. The first 100 iterations were used to pre-train the feature decomposition network, and the last 200 iterations were used for two-stage joint training. The network weights were initialized using Kaiming. All experiments were conducted on the PyTorch 1.18 framework on an NVIDIA RTX 4090 graphics card. Training was stopped and the optimal parameters were saved when the validation set metrics fluctuated less than 1e-4 for 20 consecutive iterations. The simulation results with and without semantic selection are shown in the figure below. Figure 4 As shown.

[0055] In the testing and evaluation phase, to comprehensively evaluate the overall performance of the proposed method, this embodiment employs six mainstream standard evaluation indicators: entropy (EN), standard deviation (SD), spatial frequency (SF), mutual information (MI), visual information fidelity (VIF), and structural similarity (SSIM). These indicators quantify the fusion effect from multiple dimensions, including information richness, detail clarity, modal complementarity, and structural consistency. Specifically, EN measures the information carrying capacity of the fused image; a higher value indicates richer bimodal information in the fused image. SD reflects the dispersion of image grayscale values, reflecting the contrast difference between the target and the background. SF focuses on the detail and texture clarity of the image; a higher value indicates more complete preservation of edges, textures, and other details in the fused image. MI evaluates the degree of complementary fusion of infrared and visible light modal information, intuitively reflecting the integration effect of bimodal advantages. VIF measures the visual information fidelity of the fused image; a value closer to 1 indicates more sufficient preservation of key information from the original image. SSIM quantifies the structural similarity between the fused image and the original image, effectively avoiding structural distortion, artifacts, and other problems during the fusion process.

[0056] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method of fusing infrared-visible light images, characterized in that, Includes the following steps: S1: Construct positive and negative semantic prompt text for downstream tasks; S2: Input the infrared image and the visible light image into a preset fusion network model to obtain a fused image. Calculate a visual fusion index based on semantic prompts using the infrared image, the visible light image, and the fused image. Then, normalize the visual fusion index. S3: Calculate the fusion score of the fused image under the positive and negative semantic prompt text based on the visual fusion index processed by normalization, and take the positive and negative semantic prompt text corresponding to the maximum value of the fusion score as the optimal prompt pair; S4: The optimal prompt is used as the training set for the preset fusion network model. The infrared image and the visible light image are input into the trained preset fusion network model to obtain the optimal fused image.

2. The method of claim 1, wherein, The preset fusion network model mentioned in step S2 includes the PromptFusion model, which adopts a two-stage improved U-net architecture that includes a feature decomposition stage and a semantic fusion stage.

3. The method of claim 2, wherein, The feature decomposition stage includes high-frequency and low-frequency encoders and decoders, both of which adopt a three-layer stacked structure of convolutional layer, BatchNorm layer and ReLU layer.

4. The method of claim 1, wherein, The semantic prompts in step S2 include the positive and negative semantic prompt texts.

5. The method of claim 1 or 4, wherein, The visual fusion metrics mentioned in step S2 include information entropy, standard deviation, spatial frequency, mutual information, visual information fidelity, and structural similarity index.

6. The method of claim 5, wherein, The calculation expressions for information entropy, standard deviation, spatial frequency, mutual information, visual information fidelity, and structural similarity index mentioned in step S2 are as follows: Information entropy: L represents the total number of gray levels of the fusion image; represents the probability that a pixel with a gray value of appears in the fusion image, wherein, is the number of pixels with a gray value of i ; is the total number of pixels of the fusion image, M represents the number of pixels of the long side of the fusion image, N represents the number of pixels of the wide side of the fusion image. Standard deviation: a pixel gray value of the fusion image at coordinates ; and an average gray value of the fusion image; Spatial frequency: in, and These represent the horizontal and vertical spatial frequencies, respectively, used to measure the drastic changes in grayscale in the horizontal and vertical directions. The expressions are as follows: Mutual information: in This represents the mutual information between the visible light image and the fused image; The mutual information between the infrared image and the fused image is expressed as follows: in This represents the joint probability density of the visible light image and the fused image. The joint probability density of the infrared image and the fused image. , , These are the edge probability densities of the visible light image, the infrared image, and the fused image, respectively. Visual information fidelity: K is the number of sub-bands in the visible light image or the infrared image after wavelet transform. The variance of the information in the k-th sub-band of the visible light image or the infrared image is used to measure the effective visual information. The noise variance of the visible light image or the infrared image in the kth sub-band is used to measure distortion information. Structural similarity index: Calculated by summing the structural similarity indices of the visible light image and the fused image, and the structural similarity indices of the infrared image and the fused image, and taking the average. in This represents the structural similarity index between the visible light image and the fused image. The structural similarity index between the infrared image and the fused image is represented by the following expression: Wherein, X represents the visible light image or the infrared image; Y represents the fused image; The local average grayscale value of the visible light image or the infrared image; The local average gray value of the fused image; The local variance of the visible light image or the infrared image; The local variance of the fused image; The local covariance of the visible light image and the fused image, or the infrared image and the fused image; It is a constant.

7. The infrared-visible image fusion method according to claim 6, characterized in that, The normalization process described in step S2 is as follows: No. i Infrared image, visible light image and fused image are processed J The set of visual fusion metrics obtained from positive and negative semantic cue texts is denoted as . , , , , , The normalized expression for information entropy is: Similarly, the standard deviation, spatial frequency, mutual information, visual information fidelity, and structural similarity index obtained by normalization are denoted as follows: .

8. The infrared-visible image fusion method according to claim 7, characterized in that, The calculation method for the fusion score in step S3 is as follows: No. i The infrared image, visible light image, and fused image are processed by the first... j The formula for calculating the fusion score of positive and negative semantic cue texts is as follows: in , , , , , These represent the calculated weights of the normalized information entropy, standard deviation, spatial frequency, mutual information, visual information fidelity, and structural similarity index, respectively.

9. An infrared-visible light image fusion device, characterized in that, The system includes a text construction module, an image fusion module, a semantic prompt selection module, and an optimal training module. The text construction module is used to construct positive and negative semantic prompt texts for downstream tasks. The image fusion module is used to input infrared and visible light images into a preset fusion network model to obtain a fused image. The semantic prompt selection module is used to calculate a visual fusion index based on semantic prompts from the infrared image, the visible light image, and the fused image. The visual fusion index is then normalized, and a fusion score of the fused image under the positive and negative semantic prompt texts is calculated based on the normalized visual fusion index. The positive and negative semantic prompt texts corresponding to the maximum value of the fusion score are taken as the optimal prompt pair. The optimal training module is used to train the preset fusion network model using the optimal prompt pair as the training set, and the infrared image and the visible light image are input into the trained preset fusion network model to obtain the optimal fused image.

10. An infrared-visible light image fusion system, characterized in that, It includes a memory and a processor; the memory is used to store instructions; the processor is used to operate according to the instructions to perform the method according to any one of claims 1-8.

Images