Limited collaborative target detection-oriented visible light and infrared image fusion method

Abstract

The invention discloses a visible light and infrared image fusion method for limited collaborative target detection, and belongs to the technical field of image processing. The method comprises the following steps: firstly, constructing a multi-modal unified target detection data set; respectively training VAE encoders corresponding to the RGB image and the IR image under the supervision of the GT image; joint reconstruction is carried out on the RGB image and the IR image based on a diffusion model and EM reasoning, and a preliminary reconstruction result is obtained; and finally, through a low-rank adapter mechanism, performing efficient fine tuning on the diffusion model by using the preliminary reconstruction result and the GT image. According to the multi-modal perceptual image fusion generation method based on the diffusion model, unified coding, multi-scale fusion and conditional diffusion type image generation of an IR image and an RGB image are realized through a mode of combining a structured encoder, a fusion module and a diffusion process control and decoder.

Description

Technical Field

[0001] This invention belongs to the field of image processing and computer vision technology, and in particular relates to a visible light and infrared image fusion method for limited cooperative target detection. Background Technology

[0002] Multimodal image fusion technology significantly improves the stability and robustness of target detection in complex environments by integrating complementary information from visible light (RGB) and infrared (IR) images. However, current technologies still face many challenges in practical applications, especially in extreme environments with limited drone collaboration. First, communication conditions are often extremely limited in environments such as disaster relief, with low bandwidth, high latency, and unstable links, leading to incomplete or even severe loss of data from various modalities. Second, RGB images frequently fail in environments with drastic changes in ambient light, especially at night or in strong backlighting scenarios, while infrared images, although possessing some resistance to interference, have limited resolution and detail representation capabilities. Furthermore, targets in real-world scenarios often exhibit complex dynamic characteristics such as rapid scale changes, posture rotation, and partial occlusion, and the background is often highly similar to the target, further increasing the difficulty of detection and tracking.

[0003] Faced with these multiple challenges, traditional image fusion methods that rely on high-quality, multimodal input are inadequate, severely limiting the perception and decision-making capabilities of UAVs in complex environments. Therefore, how to achieve effective multimodal target detection and tracking under non-ideal conditions such as limited communication, incomplete modalities, and asynchronous data has become a key technical problem that urgently needs to be solved.

[0004] The conventional approach in existing technologies is to extract features from infrared and visible light images separately, then perform simple stitching or weighted fusion before inputting them into an image generator, such as one based on GANs. These methods often have the following limitations: 1. Strong modality dependency: Existing technologies rely on multimodal inputs (such as Canny edges, depth, and high-quality synchronous input features from infrared and visible light) to generate high-quality fusion results. Performance drops significantly in scenarios where only a single modality is available (such as nighttime with only infrared images or communication interruptions leading to missing modal data). This invention, through a multimodal feature semantic alignment mechanism, achieves spatial consistency learning of fusion features from visible light (RGB) and infrared (IR) images during the training phase. During the inference phase, only a single modality input is needed to generate an enhanced fused image, significantly improving adaptability in limited collaborative scenarios. 2. Low efficiency of diffusion models: Traditional diffusion requires numerous sampling steps, making it difficult to meet real-time requirements. 3. Task fragmentation: Fusion and object detection are optimized independently, failing to leverage task correlation to improve overall performance.

[0005] Therefore, there is an urgent need for an efficient fusion and detection joint optimization method suitable for limited collaborative scenarios. This invention addresses this pressing practical need by proposing a multimodal image fusion detection and tracking technology framework for limited collaborative conditions. It systematically tackles key issues including collaborative data modeling, distributed fusion structure design, and robust collaborative algorithm research. By breaking through the traditional reliance on ideal conditions such as full modality, high bandwidth, and low latency, it enhances the autonomous perception capability and adaptability of UAV systems in complex environments, providing practical technical support for intelligent perception and possessing significant theoretical value and application prospects. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides a visible light and infrared image fusion method for limited cooperative target detection, belonging to the field of image processing technology. First, a multimodal unified target detection dataset is constructed. Then, under the supervision of ground truth (GT) images, VAE encoders corresponding to RGB and IR images are trained separately. Next, based on a diffusion model and EM inference, joint reconstruction of RGB and IR images is performed to obtain preliminary reconstruction results. Finally, a low-rank adapter mechanism is used to efficiently fine-tune the diffusion model using the preliminary reconstruction results and GT images. This invention's multimodal perception image fusion generation method based on a diffusion model achieves unified encoding, multi-scale fusion, and conditional diffusion-based image generation of IR and RGB images through a combination of structured encoders, fusion modules, diffusion process control, and decoders.

[0007] This invention provides a visible light and infrared image fusion method for limited cooperative target detection, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0008] Step 1: Construct a multimodal unified object detection dataset;

[0009] The images in the dataset include RGB images (visible light images), IR images (infrared images), and GT images (ground truth images); the GT images are pre-fused images used as supervised training images.

[0010] The images in the dataset cover complex environment samples;

[0011] The targets in the dataset include vehicles, static obstacles occlusion, and the corresponding target bounding box information annotation files.

[0012] Step 2: Under the supervision of the ground truth image, train the VAE encoders (Variational Auto-Encoder) corresponding to the RGB and IR images respectively to achieve multimodal spatial alignment;

[0013] Step 3: Joint reconstruction of RGB and IR images is performed based on diffusion model and EM (Expectation-Maximization) inference to obtain preliminary reconstruction results;

[0014] Step four: Using the low-rank adapter (LoRA) mechanism, the diffusion model is efficiently fine-tuned using the preliminary reconstruction results and GT images to enhance the cross-modal reconstruction effect.

[0015] In step one, the scenes in the images include daytime and nighttime switching, sunny and rainy days and foggy days, urban roads and wilderness terrain, as well as high-altitude wide field of view and near-ground targets at large angles;

[0016] The vehicles include cars, trucks, and buses;

[0017] The static obstacles include buildings and trees;

[0018] The target bounding box information annotation box file corresponding to the target includes YOLO (You Only Look Once) format or VOC (Visual Object Classes) format.

[0019] In step two, the method for training the VAE encoders corresponding to the RGB and IR images is as follows:

[0020] like Figure 3 As shown:

[0021] Step 2.1: Input the three-modal images into the corresponding three VAE encoders respectively;

[0022] The three-modal images are RGB images, IR images, and GT images; RGB images IR image GT images Where C is the number of channels, H and W are the image height and image depth, respectively; R represents the set of real numbers;

[0023] Step 2.2, the VAE encoder performs normalization processing on the three-modal images respectively:

[0024] Map the pixel values ​​of the trimodal image from [0,1] to [-1,1]; the normalization formula is: ,in, represents the normalized image; x represents the image before normalization;

[0025] Step 2.3, the VAE encoder encodes the three-modal images respectively:

[0026] After the three-modal images are encoded by the corresponding VAE encoders, the latent distributions of the RGB images are obtained respectively. Potential distribution of IR images and the potential distribution of GT images :

[0027] (2.1);

[0028] Among them, z rgb z ir and z gt These represent the potential representations of RGB, IR, and GT images, respectively. , and These are the normalized RGB image, IR image, and GT image, respectively;

[0029] Step 2.4: Sample the latent distribution using the reparameter sampling function and perform standard scaling to obtain z. rgb z ir and z gt :

[0030] (2.2);

[0031] The scaling factor s = 0.18215 is used to maintain the consistency of the potential representation scale. The scaling factor is set based on the experience of the diffusion model; Sample() is the reparameter sampling function.

[0032] Step 2.5, Multinomial Loss Function Constraints;

[0033] The multiple loss functions include KL divergence. Mixed Latent Representations Alignment Loss mix And pixel-level reconstruction loss L Pixel ;

[0034] Furthermore, the KL divergence L KL The calculation method is as follows:

[0035] (2.3);

[0036] in, Represents the latent distribution Relative potential distribution KL divergence; Represents the latent distribution Relative potential distribution KL divergence;

[0037] Furthermore, the hybrid latent representation alignment loss L mix z is constructed using a weighted mixing method. rgb and z ir Mixed latent characterization z mix :

[0038] (2.4);

[0039] in, and These are the weighting coefficients for the IR and RGB images, respectively.

[0040] RGB and IR images are input into the YOLOv11 model for detection; the detection results are normalized to [0,1] to achieve causal analysis; the YOLOv11 model automatically selects weighting coefficients based on the relative contributions of RGB and IR; the discernibility of targets in different modal images is measured by the detection score, thereby achieving adaptive fusion;

[0041] By using the object detection scores of RGB and IR images on the same candidate box b of the YOLOv11 detector as proxies for causal inference, the contributions of RGB and IR modalities to the final task are estimated; rgb and s ir , respectively, are the object detection scores on the candidate bounding box b in the RGB and IR images. Object detection is performed by the pre-trained YOLOv11 detector D. rgb and D ir Give;

[0042] (2.4.1);

[0043] To prevent numerical anomalies caused by target detection failures, a robust strategy is introduced:

[0044] If s rgb and s ir If any score in the above is zero, then the weighting coefficient is uniformly set to 0. ,Right now:

[0045] like ,but .4.2);

[0046] Otherwise, the fusion weights are calculated based on the normalized detector scores:

[0047] (2.4.3);

[0048] in, This indicates the introduction of a constant to avoid the denominator being zero;

[0049] The causal analysis strategy introduces the idea of ​​causal intervention: that is, by observing the causal effect of each modality on detection performance, a more reasonable fusion weight is given, thereby improving the robustness and accuracy of multimodal perception;

[0050] z mix For z rgb and z ir Hybrid latent representations; Hybrid latent representation alignment loss L mix The formula is:

[0051] (2.5);

[0052] Among them, MSE(z) rgb ,z mix ) represents z rgb With z mix Mean squared error loss; MSE(z) ir ,z mix ) represents z ir With z mix Mean square error;

[0053] Furthermore, the pixel-level reconstruction loss L Pixel The latent representations corresponding to the three modes are input into the corresponding VAE decoders for reconstruction, resulting in decoded and reconstructed images. ; Including RGB reconstructed images IR reconstructed images and GT reconstructed image :

[0054] (2.6);

[0055] in, This represents the VAE decoder parameters corresponding to the RGB image. This represents the VAE decoder parameters corresponding to the IR image. This represents the VAE decoder parameters corresponding to the GT image;

[0056] The pixel-level reconstruction error L is calculated based on the downstream task, choosing either L1 norm loss, Mean Square Error (MSE), or Binary Cross Entropy Loss (BCE Loss). Pixel :

[0057] (2.7);

[0058] Represents the original image before input to the VAE decoder, either RGB or IR; This indicates the loss function selected based on the downstream task; Loss Pixel express and The pixel-level error loss function;

[0059] The downstream tasks include object detection or image semantic segmentation;

[0060] Step 2.6, Cross-modal representation consistency optimization:

[0061] Supervision is provided by using Info Noise Contrastive Estimation (InfoNCE) and Cosine Similarity Loss functions to align the VAE decoders for RGB and IR respectively, thereby reducing the Euclidean distance between the latent representations of RGB and IR.

[0062] Step 2.6.1: Construct the contrast loss L between the latent representations of the three-modal images using InfoNCE Loss. NCE :

[0063] (2.8);

[0064] Step 2.6.2, using cross-entropy loss L cosine (Cosine Similarity Loss) will transform the latent representation z rgb z ir and z gt After normalization, the cosine similarity L between the three modal images is calculated. cos :

[0065]

[0066] Step 2.6.3 Cross-reconstruction loss L cross Calculation of (Cross Reconstruction Loss);

[0067] First, reconstruction is performed using a swap encoder: the RGB latent representation is input into the VAE decoder corresponding to the IR, and the IR latent representation is input into the VAE decoder corresponding to the RGB, to obtain the VAE decoding result of the swap model:

[0068] (2.10);

[0069] in, This represents the IR image output after the RGB image is input into the IR decoder; This represents the RGB image output after the IR image is input into the RGB decoder;

[0070] Then, the pixel MSE is calculated by comparing the image output from the VAE decoder with the image input to the decoder, and L is obtained. cross :

[0071] (2.11);

[0072] Step 2.6.4: Fuse the latent representation outputs to obtain the final optimized VAE. :

[0073] .

[0074] The loss in formulas (2.1)-(2.11) is optimized in a coordinated manner so that the potential representations of RGB and IR are aligned to the unified space of GT. During training, an end-to-end joint training strategy is adopted to optimize the parameters of the VAE corresponding to the RGB and IR modes by gradient descent. The optimized VAE enhances the cross-modal unified representation capability. At the same time, the aligned cross-modal potential representations are closer in spatial distance than the unaligned potential representations, as shown in Figure 4. Therefore, the steps of the GDEM-based optimization strategy in step 3 are reduced.

[0075] In step three, the diffusion model adopts the Denoising Diffusion Implicit Models (DDIM), which includes a noise prediction network (UNet), a scheduler that controls the diffusion or reverse process, and a reverse sampling workflow.

[0076] In step three, the joint reconstruction process of the RGB and IR images based on the diffusion model and EM inference is as follows:

[0077] Step 3.1: Encode the RGB image, IR image, and GT image using the trained VAE encoder to obtain the corresponding latent representations x. t ;

[0078] First, give clean sample x t Noise is added to the mixture until it matches the isotropic Gaussian distribution.

[0079] Subsequently, the UNet network receives the latent representation x after noise has been added. t By combining the time step t with the conditional encoding c (RGB / IR embedding), the noise residual is predicted. Inversely deduce the initial value of the posterior latent representation. :

[0080] ;

[0081] in, It is the cumulative noise figure of the noise dispatcher. , For each time step Noise control parameters;

[0082] Step 3.2: Predict the initial value of the posterior latent representation using UNet. A joint reconstruction method with the gradient descent expectation-maximization algorithm (GDEM);

[0083] Step 2: In the known Based on this, gradient optimization is used for further fine-tuning. Gradient optimization is influenced by L. posterior （ Supervision constraints (for posterior fine-tuning of the loss function):

[0084] In the known Based on this, gradient optimization is used to further refine the initial latent representation f0, and gradient optimization is influenced by the posterior fine-tuning loss function L. posterior Supervision and constraints:

[0085] ;

[0086] Among them, L diff For diffusion likelihood loss; L pix For pixel-level fusion prior loss; L lat For potential spatial distribution constraints; L prior For hybrid latent representation prior loss; This is the regularization term for the total variation; The total variation weighting control factor is set to 0.5;

[0087] The diffusion likelihood loss Diffusion likelihood loss is used to maintain consistency with the diffusion process. :

[0088] ;

[0089] The pixel-level fusion prior loss L pix The latent representation is decoded into the pixel space for constraint, and the prior loss L is fused at the pixel level. pix :

[0090] ;

[0091] Among them, I rgbRepresents an RGB image, I ir Represents an IR image. and These are I obtained using the causal analysis module. rgb and I ir The weight value; L fusion Pixel-level L1 loss is used for calculation; Decode represents the VAE decoder corresponding to different modalities (RGB or IR);

[0092] The potential spatial distribution constraint L lat Wasserstein-2 distance is used to measure the distance between the point estimate and the RGB or IR distribution, with latent spatial distribution constraint L. lat :

[0093] ;

[0094] Where W2 represents the Wasserstein-2 distance; These represent the mean and variance of the three-modal images, respectively. and These represent the mean values ​​of the RGB and IR images, respectively. These represent the mean square error of the RGB and IR images, respectively. This indicates that it conforms to the mean. and variance Normal distribution;

[0095] The hybrid latent representation prior loss L prior Constructing a convex combination of RGB and IR images as the supervision target, and using a hybrid latent representation prior loss L prior :

[0096] ;

[0097] in, For mixed weights, set to 0.5;

[0098] The Total Variation regularization term Total Variation regularization term used to suppress artifacts and gradient explosion. :

[0099] ;

[0100] Based on the posterior fine-tuning loss function L posterior For the initial potential representation Perform iterative updates to obtain the updated latent representation. :

[0101] ;

[0102] in, The learning rate for gradient descent, For the latent representation estimation in the k-th iteration, after completing M steps of latent representation fine-tuning, the optimized latent representation is... The reverse diffusion process is then re-injected, and the final decoding is performed using a standardized DDIM deterministic sampling strategy to restore the clean reconstruction result. The update formula for the deterministic reverse diffusion process of DDIM is as follows:

[0103] ;

[0104] Where, x t Represents the noisy latent characterization of time step t. This represents the cumulative noise figure in the noise scheduler of the diffusion model. This represents the potential fusion characterization after M-step optimization;

[0105] To accelerate sampling, deterministic inverse coefficients are introduced into DDIM. = 0, to avoid random noise interference, so that the entire diffusion reconstruction process behaves as follows:

[0106] ;

[0107] That is, it has no randomness, converges quickly, and maintains consistency; E represents the expectation.

[0108] Furthermore, in step four, the process of efficiently fine-tuning the diffusion model using the preliminary reconstruction results and the ground truth image through the low-rank adapter mechanism is as follows:

[0109] Insert a low-rank adapter into the linear transformation structure inside the noise prediction network;

[0110] The low-rank adapter includes a dimensionality reduction module and a dimensionality increase module, which are smaller in size than the original weight parameters. It only trains the low-rank part, enabling the diffusion model to have lightweight learnable capabilities when performing cross-modal fusion without updating the large-scale original weights.

[0111] During fine-tuning, the low-rank adapter directly uses the posterior fine-tuning loss function. Optimize;

[0112] During fine-tuning, the preliminary reconstructed latent representation obtained in step three is used as input, and the latent representation corresponding to the GT image is used as a supervision signal. By minimizing the posterior fine-tuning loss, the low-rank adapter learns the deviation correction between the RGB image and the IR image in the GT latent space. Furthermore, the noise prediction results, latent distribution, pixel-level structure, and mixed latent priors are constrained to maintain consistency with the diffusion process in step three.

[0113] By fine-tuning the loss in step three, the diffusion model completes lightweight learning through a low-rank adapter without changing the original parameters, making the cross-modal fusion result approximate the GT image in terms of texture restoration, edge consistency, and illumination coordination. The quality of the final fused image is significantly improved, providing a more stable and higher-quality input for subsequent object detection tasks.

[0114] An electronic device includes: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the above-described visible light and infrared image fusion method for finite cooperative target detection.

[0115] A computer-readable storage medium storing program code that can be invoked by a processor to execute the above-described visible light and infrared image fusion method for finite cooperative target detection.

[0116] The technical effects of this invention are as follows:

[0117] By introducing a dynamic weight adaptive mechanism and a multi-scale feature fusion module, the problem of blurred target outlines (e.g., halo diffusion around vehicle lights) and loss of texture information (e.g., faded details of road markings) often occurs in scenarios with complex nighttime lighting (e.g., strong direct light, interference from multiple light sources) and significant differences between infrared and visible light modes.

[0118] By suppressing the weight allocation of overexposed pixels in strong light areas, the texture of objects around the light source is preserved.

[0119] After optimization by multiple loss functions, the feature distributions of the three modalities achieved good semantic alignment in the latent space, laying the foundation for subsequent cross-modal fusion and single-modal inference. Through the combined effect of KL divergence loss, hybrid latent representation alignment loss, contrastive learning loss and cosine similarity loss, semantic consistency learning of cross-modal features was effectively achieved.

[0120] In extreme scenarios where a single modality is missing, knowledge transfer is performed through the aligned latent feature space, enabling the prediction of the hue and texture of the current infrared scene even after single-modal inference, while maintaining the original infrared attributes. This fully verifies the beneficial effect of cross-modal feature semantic consistency on downstream tasks. The present invention can effectively restore the necessary information of the entire scene and preserve and fuse the transferred information of each modality in various scenarios, providing strong support for downstream tasks. Attached Figure Description

[0121] Figure 1 This is an overall flowchart of the multimodal image fusion method of the present invention;

[0122] Figure 2 This is a diagram illustrating the overall architecture of the multimodal image fusion generation system of the present invention.

[0123] Figure 3 A diagram illustrating feature space alignment for a VAE encoder;

[0124] Figure 4 is a comparison of the distribution of latent representations before and after feature space alignment of the VAE encoder.

[0125] In Figure 4, Figure 4(a) shows the feature distribution when the VAE encoder is not aligned; Figure 4(b) shows the feature distribution after the VAE encoder is aligned.

[0126] Figure 5 is a comparison of RGB, IR, and Gen images of an outdoor parking lot at night;

[0127] In Figure 5, Figure 5(a) is an RGB image; Figure 5(b) is an IR image; and Figure 5(c) is a Gen image.

[0128] Figure 6 is a comparison of RGB, IR, and Gen images of well-lit urban roads at night.

[0129] In Figure 6, Figure 6(a) is an RGB image; Figure 6(b) is an IR image; and Figure 6(c) is a Gen image.

[0130] Figure 7 is a comparison of RGB, IR, and Gen images of a parking lot in an urban area at night;

[0131] In Figure 7, Figure 7(a) is an RGB image; Figure 7(b) is an IR image; and Figure 7(c) is a Gen image.

[0132] Figure 8 is a comparison of RGB, IR, and Gen images of a parking lot in a residential area at night;

[0133] In Figure 8, Figure 8(a) is an RGB image; Figure 8(b) is an IR image; and Figure 8(c) is a Gen image. Detailed Implementation

[0134] The technical approach and system architecture of this invention. Figure 1 This paper presents the overall technical roadmap for a multimodal image fusion method for finite cooperative object detection, including data preprocessing, feature extraction, fusion strategies, and the application of the fused image in downstream object detection and segmentation tasks. Key steps are outlined. Figure 2 The system architecture of a multimodal sensing image fusion generation system based on a diffusion model is demonstrated. It adopts a structured encoding-fusion-diffusion-decoding framework to achieve unified encoding and conditional diffusion generation of infrared and visible light images.

[0135] like Figure 3 As shown, the GDEM process begins with a clean image at time step 0. The forward process of the diffusion model is used to control the degree of noise addition based on the weights obtained from the causal analysis module. Then, after injecting RGB information, the backward diffusion process guided by EM supervision begins to obtain the fused image.

[0136] Figure 4 shows a visual comparison of the latent values ​​before and after feature space alignment in the VAE encoder. The diagram illustrates the feature space alignment part of the VAE encoder. For each modality image, there is a corresponding VAE. The latent values ​​of each modality (RGB, IR, and GT) are obtained through the VAE's encoder. The aforementioned loss function is used to achieve the effect of bringing the latent values ​​closer (reducing losses such as spatial distance and cross-entropy to align the latent values).

[0137] A comparison of the feature space alignment performance of the VAE encoder. Figure (a) shows the distribution of RGB Latent (blue), IR Latent (red), and GT Latent (green) in the latent feature space without alignment training, obtained from the ground-value image through the VAE encoder. The distribution of the three modalities in the latent feature space reveals significant differences in the shape of the feature distributions, indicating a lack of semantic consistency. Figure (b) shows the alignment performance after joint optimization using multiple loss functions. The legend is consistent with Figure (a), demonstrating good semantic alignment of the feature distributions of the three modalities in the latent space, laying the foundation for subsequent cross-modal fusion and single-modal inference. Through the combined effect of KL divergence loss, hybrid latent representation alignment loss, contrastive learning loss, and cosine similarity loss, effective semantic consistency learning of cross-modal features is achieved.

[0138] As shown in Figures 5 to 8, the fused image generation effect is displayed. Figure (a) is the original visible light input image (RGB Image), Figure (b) is the original infrared input image (IR Image), where the visible light is selected for a scene with uneven nighttime lighting, and Figure (c) is the diffusion model fusion result image (Gen Image).

[0139] Further observation of the details in the fused images shown in Figures 5 to 8 reveals that existing modalities often exhibit problems such as blurred target outlines (e.g., halo diffusion around vehicle lights) and loss of texture information (e.g., faded details of road markings) in complex nighttime lighting conditions (e.g., strong direct light, interference from multiple light sources) and scenes with significant differences between infrared and visible light modal edges. However, by introducing a dynamic weight adaptive mechanism and a multi-scale feature fusion module, this method effectively addresses these pain points: in strong light areas, the model preserves the texture of objects around the light source by suppressing the weight allocation of overexposed pixels; at modal edge transitions (e.g., the boundary between building edges and the sky), the model achieves similar results.

[0140] Furthermore, in extreme scenarios where a single modality is missing (as shown in Figure 8, which can be simulated as complete blackout in visible light, considered as a single-modal infrared input), this method performs knowledge transfer through the aligned latent feature space. This allows the hue and texture of the current infrared scene to be predicted even after single-modal inference, while maintaining the original infrared attributes. This fully verifies the beneficial effect of cross-modal feature semantic consistency on downstream tasks. Therefore, this model can effectively reconstruct the necessary information of the entire scene and preserve and fuse the transferred information from various modalities in multiple scenarios, providing strong support for downstream tasks.

Claims

1. A visible light and infrared image fusion method for limited cooperative target detection, characterized in that, Includes the following steps: Step 1: Construct a multimodal unified object detection dataset; The images in the dataset include RGB images, IR images, and GT images; the GT images are pre-fused images used as supervised training images. The images in the dataset cover complex environment samples; The targets in the dataset include vehicles, static obstacles occlusion, and the corresponding target bounding box information annotation files. Step 2: Under the supervision of the ground truth image, train the VAE encoders corresponding to the RGB and IR images respectively to achieve multimodal spatial alignment; Step 3: Based on the diffusion model and EM inference, the RGB and IR images are jointly reconstructed to obtain preliminary reconstruction results; Step four: Using a low-rank adapter mechanism, the diffusion model is efficiently fine-tuned using the preliminary reconstruction results and GT images to enhance the cross-modal reconstruction effect.

2. The visible light and infrared image fusion method for limited cooperative target detection according to claim 1, characterized in that, In step one, the scenes in the images include daytime and nighttime switching, sunny and rainy days and foggy days, urban roads and wilderness terrain, as well as high-altitude wide field of view and near-ground targets at large angles; The vehicles include cars, trucks, and buses; The static obstacles include buildings and trees; The target bounding box information annotation box file corresponding to the target includes YOLO format or VOC format.

3. The visible light and infrared image fusion method for limited cooperative target detection according to claim 1, characterized in that, In step two, the method for training the VAE encoders corresponding to the RGB and IR images is as follows: Step 2.1: Input the three-modal images into the corresponding three VAE encoders respectively; The three-modal images are RGB images, IR images, and GT images; RGB images IR image GT images Where C is the number of channels, H and W are the image height and image width, respectively; R represents the set of real numbers; Step 2.2, the VAE encoder performs normalization processing on the three-modal images respectively: Map the pixel values ​​of the trimodal image from [0,1] to [-1,1]; the normalization formula is: ,in, represents the normalized image; x represents the image before normalization; Step 2.3, the VAE encoder encodes the three-modal images respectively: After the three-modal images are encoded by the corresponding VAE encoders, the latent distributions of the RGB images are obtained respectively. Potential distribution of IR images and the potential distribution of GT images : （2.1）； Among them, z rgb z ir and z gt These represent the potential representations of RGB, IR, and GT images, respectively. , and These are the normalized RGB image, IR image, and GT image, respectively; Step 2.4: Sample the latent distribution using the reparameter sampling function and perform standard scaling to obtain z. rgb z ir and z gt : （2.2）； Where s is the scaling factor; Sample() is the reparameter sampling function; Step 2.5, Multinomial Loss Function Constraints; The multiple loss functions include KL divergence. Hybrid latent representation alignment loss L mix and pixel-level reconstruction loss L Pixel ; The KL divergence L KL The calculation method is as follows: （2.3）； in, Represents the latent distribution Relative potential distribution KL divergence; Represents the latent distribution Relative potential distribution KL divergence; The hybrid latent representation alignment loss L mix z is constructed using a weighted mixing method. rgb and z ir Mixed latent characterization z mix : （2.4）； in, and These are the weighting coefficients for the IR and RGB images, respectively. By using the object detection scores of RGB and IR images on the same candidate box b of the YOLOv11 detector as proxies for causal inference, the contributions of RGB and IR modalities to the final task are estimated; rgb and s ir The values ​​are the object detection scores for the RGB and IR images on the candidate bounding box b, respectively. Object detection is performed by the pre-trained YOLOv11 detector D. rgb and D ir Give; （2.4.1）； To prevent numerical anomalies caused by target detection failures, a robust strategy is introduced: If s rgb and s ir If any score in the above is zero, then the weighting coefficient is uniformly set to 0. ,Right now: （2.4.2）； Otherwise, the fusion weights are calculated based on the normalized detector scores: （2.4.3）； in, This indicates the introduction of a constant to avoid the denominator being zero; z mix For z rgb and z ir Hybrid latent representations; Hybrid latent representation alignment loss L mix The formula is: （2.5）； Among them, MSE(z) rgb ,z mix ) represents z rgb With z mix Mean squared error loss; MSE(z) ir ,z mix ) represents z ir With z mix Mean square error; The pixel-level reconstruction loss L Pixel The latent representations corresponding to the three modes are input into the corresponding VAE decoders for reconstruction, resulting in decoded and reconstructed images. ; Including RGB reconstructed images IR reconstructed images and GT reconstructed image : （2.6）； in, This represents the VAE decoder parameters corresponding to the RGB image. This represents the VAE decoder parameters corresponding to the IR image. This represents the VAE decoder parameters corresponding to the GT image; The pixel-level reconstruction error L is calculated based on the downstream task, choosing either L1 norm loss, mean squared error loss, or cross-entropy loss. Pixel : （2.7）； Represents the original image before input to the VAE decoder, either RGB or IR; This indicates the loss function selected based on the downstream task; Loss Pixel express and The pixel-level error loss function; Step 2.6, Cross-modal representation consistency optimization: The VAE decoder is supervised using contrastive learning loss and cosine similarity loss functions to align RGB and IR respectively.

4. The visible light and infrared image fusion method for limited cooperative target detection according to claim 3, characterized in that, In step 2.6, the cross-modal representation consistency optimization process is as follows: Step 2.6.1: Construct the contrast loss L between the latent representations of the three-modal images using InfoNCE Loss. NCE : （2.8）； Each contrastive learning loss term, InfoNCE, uses the standard contrastive learning formula, with a temperature coefficient. ; Step 2.6.2, use cross-entropy loss to transform the latent representation z rgb z ir and z gt After normalization, the cosine similarity L between the three modal images is calculated. cos : （2.9）； Step 2.6.3 Cross-reconstruction loss L cross Calculation; First, reconstruction is performed using a swap encoder: the RGB latent representation is input into the VAE decoder corresponding to the IR, and the IR latent representation is input into the VAE decoder corresponding to the RGB, to obtain the VAE decoding result of the swap model: （2.10）； in, This represents the IR image output after the RGB image is input into the IR decoder; This represents the RGB image output after the IR image is input into the RGB decoder; Then, the pixel MSE is calculated by comparing the image output from the VAE decoder with the image input to the decoder, and L is obtained. cross : （2.11）； Step 2.6.4: Fuse the latent representation outputs to obtain the final optimized VAE. : 。 5. The visible light and infrared image fusion method for limited cooperative target detection according to claim 1, characterized in that, In step three, the diffusion model adopts a denoised diffusion probability model, which includes a noise prediction network, a scheduler that controls the diffusion or reverse process, and a reverse sampling workflow.

6. The visible light and infrared image fusion method for limited cooperative target detection according to claim 1, characterized in that, In step three, the joint reconstruction process of the RGB and IR images based on the diffusion model and EM inference is as follows: Step 3.1: Encode the RGB image, IR image, and GT image using the trained VAE encoder to obtain the corresponding latent representations x. t ; First, give clean sample x t Noise is added to the mixture until it approximates an isotropic Gaussian distribution; Subsequently, the UNet network receives the latent representation x after noise has been added. t By combining time step t and conditional coding c, the noise residual is predicted. Inversely deduce the initial value of the posterior latent representation. : ； in, It is the cumulative noise figure of the noise dispatcher. , For each time step Noise control parameters; Step 3.2: Predict the initial value of the posterior latent representation using UNet. A joint reconstruction method with the gradient descent expectation-maximization algorithm (GDEM); Step 2: In the known Based on this, gradient optimization is used for further fine-tuning. Gradient optimization is influenced by L. posterior（ Supervision constraints (for posterior fine-tuning of the loss function): In the known Based on this, gradient optimization is used to further refine the initial latent representation f0, and gradient optimization is influenced by the posterior fine-tuning loss function L. posterior Supervision and constraints: ； Among them, L diff For diffusion likelihood loss; L pix For pixel-level fusion prior loss; L lat For potential spatial distribution constraints; L prior For hybrid latent representation prior loss; This is the regularization term for the total variation; This is the total variation weighting control factor; The diffusion likelihood loss Diffusion likelihood loss is used to maintain consistency with the diffusion process. : ； The pixel-level fusion prior loss L pix The latent representation is decoded into the pixel space for constraint, and the prior loss L is fused at the pixel level. pix : ； Among them, I rgb Represents an RGB image, I ir Represents an IR image. and These are I obtained using the causal analysis module. rgb and I ir The weight value; L fusion Pixel-level L1 loss is used for calculation; Decode represents the VAE decoder corresponding to different modalities; The potential spatial distribution constraint L lat Wasserstein-2 distance is used to measure the distance between the point estimate and the RGB or IR distribution, with latent spatial distribution constraint L. lat : ； Where W2 represents the Wasserstein-2 distance; These represent the mean and variance of the three-modal images, respectively. and These represent the mean values ​​of the RGB and IR images, respectively. These represent the mean square error of the RGB and IR images, respectively. This indicates that it conforms to the mean. and variance normal distribution; The hybrid latent representation prior loss L prior Constructing a convex combination of RGB and IR images as the supervision target, and using a hybrid latent representation prior loss L prior : ； in, Mixed weights; The Total Variation regularization term Total Variation regularization term used to suppress artifacts and gradient explosion. : ； Based on the posterior fine-tuning loss function L posterior For the initial potential representation Perform iterative updates to obtain the updated latent representation. : ； in, The learning rate for gradient descent, For the latent representation estimation in the k-th iteration, after completing M steps of latent representation fine-tuning, the optimized latent representation is... The reverse diffusion process is re-injected, and the final decoding is performed using a standardized DDIM deterministic sampling strategy to restore the clean reconstruction result. The update formula for the deterministic reverse diffusion process of DDIM is as follows: ； Where, x t Represents the noisy latent characterization of time step t. This represents the cumulative noise figure in the noise scheduler of the diffusion model. This represents the potential fusion characterization after M-step optimization; To accelerate sampling, deterministic inverse coefficients are introduced into DDIM. = 0, to avoid random noise interference, so that the entire diffusion reconstruction process behaves as follows: ； That is, it has no randomness, converges quickly, and maintains consistency; E represents the expectation.

7. The visible light and infrared image fusion method for limited cooperative target detection according to claim 1, characterized in that, In step four, the process of efficiently fine-tuning the diffusion model using the preliminary reconstruction results and the ground truth image through the low-rank adapter mechanism is as follows: Insert a low-rank adapter into the linear transformation structure inside the noise prediction network; The low-rank adapter includes a dimensionality reduction module and a dimensionality increase module; During fine-tuning, the low-rank adapter directly uses the posterior fine-tuning loss function. Optimize; During fine-tuning, the preliminary reconstructed latent representation obtained in step three is used as input, and the latent representation corresponding to the GT image is used as a supervision signal. By minimizing the posterior fine-tuning loss, the low-rank adapter learns the deviation correction between the RGB image and the IR image in the GT latent space. Furthermore, the noise prediction results, latent distribution, pixel-level structure, and mixed latent priors are constrained to maintain consistency with the diffusion process in step three.

8. An electronic device, characterized in that, include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images