LVM guided infrared and visible light image fusion method based on object perception and context contrast learning

Abstract

The invention discloses an LVM guided infrared and visible light image fusion method based on object perception and context contrast learning, and belongs to the technical field of image fusion processing. According to the method, the feature interaction fusion network is constructed, the feature interaction fusion block is designed, and global space information is reserved while an object specific to a modal is highlighted. And an SAM guide mask generator is provided, and a complete detection frame is obtained by using a text-driven target detection model and is used as a frame prompt in the SAM. For comparative learning in a multi-modal task with content information, a context space is constructed to calculate the distance between samples. In order to provide more cohesive samples for a context space, a feature interaction fusion network is designed, key features are strategically and preferentially considered, meanwhile, complex inter-modal relationships are captured, and context-related information is highlighted through a feature interaction fusion block. According to the method, the scene information can be reserved while the integrity of the highlighted object in the fused image is consolidated.

Description

Technical Field

[0001] The present invention belongs to the technical field of image fusion processing, and specifically proposes a method for fusion of infrared and visible light images guided by a large-vision-model based on object perception and context contrast learning, called OCCO; and designs a novel feature interaction fusion network to solve the information conflict problem in the fused image caused by modality differences, and further improve the ability to perceive salient targets. Background Art

[0002] The multimodal image fusion task aims to fuse images of different modalities captured by different sensors in the same scene and make full use of complementary features to generate a fused image containing effective information.

[0003] In the fusion of infrared and visible light images, visible light images have clear texture details and show relatively complete scene information. However, in extreme cases such as darkness or object occlusion, visible light image information is severely lost. On the other hand, infrared sensors use thermal radiation to perceive imaging and are less affected by the environment. By applying complementary characteristics and a good fusion strategy, the fused image can contain comprehensive scene information while highlighting thermal targets such as human bodies and vehicles. In theory, this will allow advanced visual tasks such as tracking, target detection, and semantic segmentation to achieve better performance.

[0004] Infrared and visible light image fusion can be divided into three types according to the fusion orientation: (1) vision-driven methods; (2) downstream-oriented methods; and (3) task-driven methods. Over time, a large number of non-deep learning and deep learning methods have been created and applied to image fusion, resulting in improved image quality. Infrared and visible light image fusion aims to preserve the source modality information as much as possible, however, downstream tasks require clear boundaries. This task difference limits the further development of image fusion.

[0005] Therefore, task-driven methods are essential for image fusion. Vision-driven methods focus on visual perception, which retains or eliminates information based on carefully designed algorithms, and may reduce the focus on important objects due to low pixel intensity. Figure 1 In (a), in order to obtain the best visual effect, the source information is retained as much as possible without highlighting the key scene elements, which will lead to the problem of false positive, false negative and low confidence in the target detection task. In order to eliminate this defect, some scholars have proposed to cascade the fusion network and the segmentation network to improve the performance of downstream tasks, such as Figure 1As shown in (b) in the figure. Although the detection results are improved, the substantial differences between the two tasks may destroy the fusion results, resulting in fragmented information retention. In addition, researchers have also tried to effectively interact the features between the two tasks to achieve multi-task collaborative development. However, in order to converge the segmentation loss, Figure 1 The fusion result in (c) shows forced information diffusion to enhance edge texture.

[0006] To this end, we choose to only use the large visual model to provide prior semantic information to guide training, without introducing segmentation networks and segmentation losses, such as Figure 1 As shown in (d) in . This avoids the gradient interference caused by display constraints while retaining the semantic perception ability to the greatest extent. Contrastive learning can optimize the fusion results based on the extracted semantic information and enhance the semantic content without interfering with the fusion network. We propose an LVM-guided object-aware contextual contrastive learning for infrared and visible light image fusion, called OCCO. This method can solve the problem of incomplete semantic information on salient targets due to modality differences. With the advancement of LVM, the guidance information that can be captured from the source image is richer. As the most advanced segmentation model, Segment Anything Model has accurate segmentation and generalization capabilities that far exceed other segmentation models, which enables us to carefully analyze which information in each modality is more beneficial to the fused image. Therefore, SAM will serve as a high-level visual task consultant to guide the entire training process.

[0007] Contrastive learning is a powerful training strategy in both supervised and unsupervised learning. It is particularly suitable for multimodal image fusion tasks that lack labels, and many methods have successfully introduced contrastive learning into multimodal image fusion tasks. Among them, CoCoNet can only be trained on a few sample datasets such as TNO because it requires manual annotation of salient targets, and the visible images of these datasets lack scene diversity and texture details. Therefore, in this invention, we propose a new SAM-guided mask generator, which uses the text-driven object detection model Ground-DINO to obtain a complete detection box and use it as a prompt box in SAM to quickly extract a large number of accurately annotated semantic samples.

[0008] The difference between the two modalities can be intuitively reflected in the mask: for the same position information, if the visible light image contains rich information and the infrared image lacks information, then the information-rich part will only be shown in the visible light mask, which forms a pair of positive samples (rich visible light information) and negative samples (information-lost infrared images). We expect the anchor sample (fused image) to contain the information-rich modality and stay away from the information-sparse modality.

[0009] In addition, we introduce a new latent feature space, the context space, to calculate the distance between samples. The context space is a feature space designed for non-aligned data, which compares the similarity between feature regions based on semantic content while ignoring the spatial position. In order to provide more cohesive samples to the context space, we design a feature interaction fusion network that prioritizes key features while capturing complex inter-modal relationships and highlights content-related information through the feature interaction fusion module. In this way, our proposed OCCO can both consolidate the integrity of salient objects in the fused image and retain rich scene information. Summary of the invention

[0010] To overcome the shortcomings of current techniques, this paper proposes a novel LVM-guided contrastive learning training strategy that can improve the performance of downstream visual tasks without interfering with the fusion results. In addition, a feature interaction fusion network based on context space is proposed to improve the quality of feature representation and its sensitivity to spatial context.

[0011] The technical solution of the present invention:

[0012] An infrared and visible light image fusion method based on LVM guided by object perception and context contrastive learning includes the following steps:

[0013] Step S1: Constructing Feature Interaction Fusion Network (FIFN)

[0014] like Figure 2 As shown, the feature interaction fusion network includes four convolution blocks for feature extraction, four feature interaction fusion blocks for feature fusion, and three up-convolution blocks;

[0015] Each convolution block uses conventional convolution operations to extract the information of each modality. It includes two convolution units and a maximum pooling operation. Each convolution unit includes a 3×3 convolution, a batch normalization, and a ReLU activation function. Each upconvolution block includes two convolution units and an upsampling operation.

[0016] After the visible light image vi and the infrared image ir are convolved through four layers with the same structure but different parameters, four single-modal features can be extracted: represents the unimodal features extracted from the visible light image after passing through the i-th layer conversion block, represents the unimodal features extracted from the infrared image after passing through the conversion block of the i-th layer, where i∈(1, ···, 4) represents the number of layers, B represents the batch size, C represents the number of channels, H represents the feature height, and W represents the feature width.

[0017] To ensure that the fused features effectively capture semantic and scene information, we design a Feature Interaction Fusion Block (FIFB) to preserve the global spatial information while highlighting modality-specific objects.

[0018] The structure of FIFB is as follows Figure 3 As shown, the architecture consists of three parts: spatial enhancement module, cross-channel attention module and cross-modal attention module.

[0019] (1) The cross-channel attention module can enhance the semantic information between modalities.

[0020] 4 unimodal features First, we perform average pooling and maximum pooling respectively. Average pooling will retain the basic low-frequency information from the global, while maximum pooling will focus more on high-frequency details. We cascade the pooling vectors obtained by average pooling and maximum pooling, and sequentially pass a linear mapping MLP and a sigmoid function to obtain the channel weight CW∈R B×2×1×C . These weights are then split into and This process can be formalized as follows:

[0021]

[0022] Where δ is the sigmoid function, Cat(·,·) represents the cascade operation, and MA represents two pooling operations, including maximum pooling and average pooling.

[0023] Then, the intermediate features of the two modalities are enhanced by the channel attention weights to obtain Where * represents channel dimension multiplication.

[0024] (2) The cross-modal attention module enables the model to learn richer modal feature representations and improves the ability to handle complex scenes through multimodal features in complementary regions.

[0025] Will and It is flattened into a linear sequence in the channel dimension, denoted as and

[0026] Then project the sequence into and Attention feature map Depend on The attention map is used to enhance the relevant parts of the visible image in the infrared image. The enhanced features are expressed as Similarly, we obtain

[0027] (3) Spatial attention can enhance the important information of the modality itself.

[0028] The spatial attention map is obtained by spatial multiplication of the MLP and the input features along the channel axis. The spatial enhancement process is formulated as

[0029]

[0030] where ⊙ represents the spatial dimension multiplication, C x×x is a convolutional layer with a kernel size of x×x, and R(·) is the relu activation function.

[0031] At the same time, and Flatten into a sequence, respectively with Finally, the two feature vectors are combined into a fusion vector represented as

[0032]

[0033] Where Ω is batch normalization.

[0034] For channel restoration and fusion feature refinement, the output fusion feature is

[0035]

[0036] Calculated. Where DC 3×3 Represents a depthwise convolution with a kernel size of 3×3. Depthwise convolution captures features from each channel and processes them independently, enabling the model to learn deeper features and improve its overall feature extraction capability.

[0037] In addition, the residual connections help stabilize the proposed model during the training phase, preventing it from focusing only on the most salient information. The four outputs extracted by FIFB will be used to reconstruct the fused image.

[0038] The four fused features obtained have decreasing size differences. In order to make the sizes of two adjacent fused features the same, it is necessary to complete the scale recovery through the up-convolution block. For the fused features obtained in the fourth layer, only the upsampling operation is used to complete the scale recovery. Finally, the channel recovery is completed through a layer of convolution, and a fused image with the same size as the original image is output.

[0039] Step S2: Training the Feature Interaction Fusion Network (FIFN)

[0040] (1) During the training process, a SAM-guided mask generator is used to extract the mask information of salient targets in the modalities of the visible light image vi and the infrared image ir for contextual contrastive learning.

[0041] SAM can accurately segment objects triggered by various cues such as points or boxes. To further improve its accuracy, we adopt a text-driven object detection model, Ground DINO, to generate detection box cues for SAM. The text cue “scene contains pedestrians or cars” ensures that relevant aspects are captured in both modes, producing two masks: Where GD stands for Ground DINO.

[0042] like Figure 4 As shown in Figure 1, Ground DINO detects distant people in infrared images that are not included in the labels provided by the dataset. With the box cues, SAM can accurately segment the individuals within the boxes, providing a finer segmentation than the labels. These masks convey the maximum semantic data that SAM extracts from the source image.

[0043] like Figure 5 As shown in Figure 1, the original visible light image vi and infrared image ir are input into the SAM-guided mask generator to extract the mask information of the salient targets in the modalities of the visible light image vi and the infrared image ir, and the two mask information are divided into modality-specific information and shared information. The specific process is as follows: the shared information can be obtained by intersecting the two masks: where · represents the Hadamard product. The modality-unique data is the original mask obtained by removing the shared information: Finally, the background region can be obtained by removing these three semantic parts: Here 1 H×W represents an all-one matrix of size H×W. Therefore, we obtain three semantic masks and background masks based on these two modalities

[0044] (2) For contextual contrastive learning, positive and negative samples are generated by calculating three semantic masks.

[0045] Obviously, when using As an information filter, the content extracted from the visible image is most beneficial to the segmentation network, while the information of the infrared image is redundant. The samples obtained on the visible image are designated as positive samples and are denoted as The samples extracted from the infrared image are taken as negative samples and are denoted as The anchor sample at this time is Where f represents the fused image obtained by the fusion network.

[0046] Contrastive learning will perform filtering operations three times based on three semantic information filters. We can simply understand it as extracting samples with its own modality as a mask, then itself is a positive sample and the other modality is a negative sample. For the background part, it is necessary to retain the texture information in the visible light image to make the fused image more natural, that is, set the visible light image as a positive sample and the infrared image as a negative sample.

[0047] A large number of negative samples are more conducive to anchor samples to learn the potential features between positive and negative samples. Therefore, we divide the samples evenly to produce b groups in each batch. Each batch consists of one anchor sample, one positive sample, and b negative samples. This grouping strategy means that the first group will generate anchors, positive samples, and negative samples at the same time, while the remaining b-1 groups only generate negative samples. The formula of the contrast loss function is as follows:

[0048] L contrast =L uniqe +L share +L bg

[0049]

[0050]

[0051]

[0052] Where D(,) and D bg (,) represents the distance between two samples, ω1 and ω2 are fixed weights, set to 0.5 and 0.5. Here we can understand that the numerator is a positive sample, such as using the visible light unique information mask from the visible light image vi The extracted sample is a positive sample represented by The denominator is a negative sample, which is masked from infrared images using unique information of visible light. The sample extracted is a negative sample The other samples are the same. The fused image f is used as an anchor sample, which is close to the positive sample and far away from the negative sample.

[0053] Furthermore, for the latent feature space, CoCoNet selected the commonly used VGG19 model, extracted features from 5 layers, and calculated the Euclidean distance between samples. However, samples restricted only by the l2 norm lack high-frequency information, resulting in overly smooth textures, which is undesirable in image fusion tasks.

[0054] We also use pre-trained VGG19 for feature extraction, but the context distance is only calculated on the deep features. The context loss uses the gamma matrix to convert the input features into feature points and calculate the cosine similarity of the feature points of the two modalities. And normalize it to convert from similarity to distance. In addition, the absolute distance also needs to improve the fusion accuracy. So the distance D between samples can be expressed as:

[0055]

[0056] where vgg i (·) represents the i-th layer feature extracted by vgg. Finally, the background information does not contain special semantic information, and the Euclidean distance is used to measure the similarity in the shallow feature space (i=1):

[0057]

[0058] Here d is the l2 norm.

[0059] Beneficial effects of the present invention:

[0060] 1. To collect shallow features and deep semantic features from different modalities, we construct a Feature Interaction Fusion Network (FIFN). To ensure that the fused features effectively capture semantic and scene information, we design a Feature Interaction Fusion Block (FIFB) to preserve global spatial information while highlighting modality-specific objects.

[0061] 2. We propose a novel SAM-guided mask generator that uses the text-driven object detection model GroundDINO to obtain complete detection boxes and use them as box prompts in SAM. This process is conducive to extracting a large number of accurately annotated samples.

[0062] 3. For contrastive learning in multimodal tasks with content information, we construct a new latent feature space: context space to calculate the distance between samples. Context space is a feature space designed for non-aligned data, which semantically compares the similarity between feature regions while ignoring their spatial positions. In order to provide more cohesive samples to the context space, we design a feature interaction fusion network. The network strategically prioritizes key features while capturing complex inter-modal relationships, highlighting context-related information through feature interaction fusion blocks. In this way, the proposed OCCO can preserve scene information while consolidating the integrity of the highlighted objects in the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1This is a comparison between our method and existing fusion methods: (a) vision-driven, (b) semantic-driven, (c) feature-level semantic-driven, and (d) target-aware.

[0064] Figure 2 This is the OCCO framework diagram.

[0065] Figure 3 It is the FIFB module diagram.

[0066] Figure 4 It is the mask acquisition flow chart.

[0067] Figure 5 This is a diagram of mask information disassembly.

[0068] Figure 6 : is a comparison of the fusion results obtained on FMB by the existing fusion method and the method OCCO proposed in the present invention, wherein Figure (a) is a visible light image, Figure (b) is a fusion image based on DenseFuse, Figure (c) is a fusion image based on SeAFusion, Figure (d) is a fusion image based on LRRNet, Figure (e) is an infrared image, Figure (f) is a fusion image based on CDDFuse, Figure (g) is a fusion image based on CoCoNet, Figure (h) is a fusion image based on SemLA, Figure (i) is a fusion image based on M2Fusion, and Figure (k) is a fusion image based on the fusion method OCCO proposed in the present invention.

[0069] Figure 7 3 is a comparison of the fusion results obtained by the existing fusion method and the method OCCO proposed in the present invention on MSRS-Day, wherein Figure (a) is a visible light image, Figure (b) is a fusion image based on DenseFuse, Figure (c) is a fusion image based on SeAFusion, Figure (d) is a fusion image based on LRRNet, Figure (e) is an infrared image, Figure (f) is a fusion image based on CDDFuse, Figure (g) is a fusion image based on CoCoNet, Figure (h) is a fusion image based on SemLA, Figure (i) is a fusion image based on M2Fusion, and Figure (k) is a fusion image based on the fusion method OCCO proposed in the present invention.

[0070] Figure 8: is a comparison of the fusion results obtained by the existing fusion method and the method OCCO proposed in the present invention on MSRS-Night, wherein Figure (a) is a visible light image, Figure (b) is a fusion image based on DenseFuse, Figure (c) is a fusion image based on SeAFusion, Figure (d) is a fusion image based on LRRNet, Figure (e) is an infrared image, Figure (f) is a fusion image based on CDDFuse, Figure (g) is a fusion image based on CoCoNet, Figure (h) is a fusion image based on SemLA, Figure (i) is a fusion image based on M2Fusion, and Figure (k) is a fusion image based on the fusion method OCCO proposed in the present invention.

[0071] Fig. 9 : is a comparison of the fusion results obtained by the existing fusion method and the method OCCO proposed in the present invention on M3FD, wherein Figure (a) is a visible light image, Figure (b) is a fusion image based on DenseFuse, Figure (c) is a fusion image based on SeAFusion, Figure (d) is a fusion image based on LRRNet, Figure (e) is an infrared image, Figure (f) is a fusion image based on CDDFuse, Figure (g) is a fusion image based on CoCoNet, Figure (h) is a fusion image based on SemLA, Figure (i) is a fusion image based on M2Fusion, and Figure (k) is a fusion image based on the fusion method OCCO proposed in the present invention.

[0072] Fig.10 : is a comparison of the fusion results obtained by the existing fusion method and the method OCCO proposed in the present invention on TNO, wherein Figure (a) is a visible light image, Figure (b) is a fusion image based on DenseFuse, Figure (c) is a fusion image based on SeAFusion, Figure (d) is a fusion image based on LRRNet, Figure (e) is an infrared image, Figure (f) is a fusion image based on CDDFuse, Figure (g) is a fusion image based on CoCoNet, Figure (h) is a fusion image based on SemLA, Figure (i) is a fusion image based on M2Fusion, and Figure (k) is a fusion image based on the fusion method OCCO proposed in the present invention.

[0073] Fig.11 The comparison of the detection results obtained by the existing fusion method and the method proposed in this invention, OCCO, on MSRS, where (a) is the label, visible light image, and (b) is based on DenseFus eFigure (c) is the detection result based on LRRNet, Figure (d) is the detection result based on CoCoNet, Figure (e) is the detection result based on visible light images, Figure (f) is the detection result based on CDDFuse, Figure (g) is the detection result based on SeAFusion, Figure (h) is the detection result based on SegMif, Figure (i) is the detection result based on infrared images, Figure (i) is the detection result based on SemLA, Figure (k) is the detection result based on M2Fusion, and Figure (1) is the detection result based on the fusion method OCCO proposed in the present invention.

[0074] Fig.12 The comparison of the detection results obtained by the existing fusion method and the method OCCO proposed in this invention on M3FD, where Figure (a) is the label, visible light image, and Figure (b) is based on DenseFus e Figure (c) is the detection result based on LRRNet, Figure (d) is the detection result based on CoCoNet, Figure (e) is the detection result based on visible light images, Figure (f) is the detection result based on CDDFuse, Figure (g) is the detection result based on SeAFusion, Figure (h) is the detection result based on SegMif, Figure (i) is the detection result based on infrared images, Figure (i) is the detection result based on SemLA, Figure (k) is the detection result based on M2Fusion, and Figure (1) is the detection result based on the fusion method OCCO proposed in the present invention. DETAILED DESCRIPTION

[0075] The present invention introduces a new LVM guided image fusion method, which can ensure the quality of the fused image while improving the performance of high-level visual tasks. First, a large visual model SAM is used to generate semantic masks from two different modalities, and the mask is decomposed into modality-unique information and modality-shared information according to its information components. Various semantic masks are used to extract rich modality samples, and the fused image is driven to retain semantic information through contrastive learning. Samples with more semantic content are considered to be positive samples, while samples with sparse semantic content are considered to be negative samples, with the fused image as an anchor sample. In addition, a new latent feature space (context space) is proposed, which aggregates features on the basis of each feature point, significantly enhancing the coherence and significance of features compared to the Euclidean space. The following is an illustration of the present invention in combination with a specific embodiment. This embodiment is carried out according to the method proposed by the present invention, and the specific implementation steps are as follows:

[0076] Step S1: Model training:

[0077] The model is trained using pixel-level loss functions and contextual loss to jointly constrain training:

[0078] L pixel =L ssim +ω3L int +ω4L texture

[0079] L ssim =2-SSIM(f,vi)-SSIM(f,ir)

[0080]

[0081] L total =L pixel +ω5L contrast

[0082] Here L ssim is the structural similarity loss, L int is the brightness loss, L texture is the texture loss, L contrast is the contextual contrast loss introduced above. SSIM is the structural similarity calculation, I ir (i, j) is the pixel intensity of the infrared image at pixel (i, j), V represents the gradient calculation. ω3, ω4 and ω5 are hyperparameters set to 10, 1, 1.

[0083] Simulation experiment

[0084] In the training process, we randomly crop two 256x256 images from the FMB dataset input pair as input, forming a total of 2440 training samples. Epoch and batchsize are set to 30 and 12 respectively, and the learning rate is set to 1e-4. We use eight state-of-the-art methods, which are DenseFuse, SeAFusion, LRRNet, CDDFuse, CoCoNet, SemLA, SegMif and M2Fusion. The fused images are generated using public code and pre-trained weights of each algorithm. Post-processing techniques are used to convert grayscale images produced by DenseFuse, LRRNet and CDDFuse into color images for fair comparison. And four public multimodal datasets are used to evaluate the fusion performance, including: FMB, MSRS, M3FD and TNO, with 280 pairs, 361 pairs, 300 pairs and 21 pairs of infrared and visible light images respectively. The pixel-level image quality consists of four indicators: entropy (EN), spatial frequency (SF), average gradient (AG) and correlation coefficient (CC).

[0085] 1. Comparative experiment

[0086] (1) Comparison of fusion performance on the FMB dataset

[0087] Figure 6 In the figure, the pedestrian (red frame) in the infrared image is partially blocked by branches, resulting in an incomplete shape. DenseFuse and LRRNet lose a lot of brightness information in the overall image. The other six methods all perceive high-intensity radiation on the human body, but are disturbed by scene information, resulting in scattered leg information. CoCoNet, SemLA, SegMif, and CDDFuse all show very blurred leg information. SeAFusion and M2Fusion retain good overall information of pedestrians. However, M2Fusion loses the head information of pedestrians. Finally, compared with SeAFusion, our fusion result is clear and the leg information is not blurred at all. Due to the presence of certain haze interference in the visible scene, the road signs marked with green frames in the visible information have certain noise, and no method except SegMif optimizes this interference. In addition, compared with SegMif, our boundary information is more intuitive and the road signs are clear and easy to read. Our method outperforms other methods in terms of salient targets and scene information. As shown in Table 1, the proposed method obtains all the best values, indicating that the fused image obtained on FMB has better image quality than other methods. The best value, second best value, and third best value are indicated in bold, underlined, and double underlined, respectively.

[0088] Table 1 Quantitative comparison of various methods on the FMB dataset.

[0089]

[0090]

[0091] (2) Comparison of fusion performance on the M3FD dataset

[0092] like Figure 7As shown in the figure, a person with a backpack (highlighted in the red box) shows two boundaries. This phenomenon is attributed to the interaction between the individual's black clothing and the infrared data, causing potential misinterpretation of the subject's outline. In this case, methods such as SeAFusion, M2Fusion, CDDFuse, and our proposed method show superior ability in maintaining boundary consistency. In contrast, several other techniques have obvious seam problems, which reduce the clarity of the image. On the contrary, in the group of individuals participating in the experiment, as shown in the green box, methods such as DenseFuse, LRRNet, and SegMif tend to merge the subject with the background environment, resulting in blurred human outlines. Although SeAFusion and M2Fusion perform well in all aspects, they still have limitations in preserving background information. Notably, M2Fusion completely fails to capture the horizon sky where the ocean meets the sun, an omission that raises concerns about the fidelity and realism of the resulting images. This loss of contextual boundaries is not only unreasonable, but also reduces the effectiveness of the method in accurately rendering complex scenes. Table 2 is a quantitative comparison of the fusion results obtained by the existing fusion method and the method proposed in the present invention on M3FD, where the best value, the second best value and the third best value are represented by bold, underline and double underline, respectively.

[0093] Table 2 Quantitative comparison of various methods on the M3FD dataset.

[0094]

[0095] As shown in Table 2, the proposed method (OCCO) also obtains all the best values ​​on the five indicators. These results show that compared with other existing fusion methods, the fusion method proposed in this paper has more robust fusion performance.

[0096] (3) Comparison of fusion performance on the MSRS dataset.

[0097] During the day, visible light images have rich details and scene information, and infrared images highlight important targets. However, due to the characteristics of the equipment, the same target in different modalities may have different positions. In the fused image, the conflict of modal information often leads to ghosting effects at a single edge, which is not conducive to the subsequent tasks. Figure 8As shown in the figure, the close-up individuals highlighted in the red box have modal misalignment. Only our method and CoCoNet enhance the target infrared information, which is helpful for human shape recognition. However, the training strategy of CoCoNet is not perfect. It does not fully retain the boundary information, resulting in highly blurred fused images, while our method can retain complete individual information. For targets with low infrared brightness, other methods also perform poorly. Even SemLA, which is famous for registration tasks, cannot perceptually align the modalities. The distant crowd identified by the green box is almost invisible in the visible light image and only vaguely visible in the infrared image. None of the other methods can directly discover this group. Only our method highlights these individuals in the fusion result, while the background still retains visible information, which forms a sharp contrast with the crowd.

[0098] At night, the information in visible light images is significantly weakened, and infrared information needs to be more prominent. Fig. 9 In the infrared image, there is a person in the green box, Fig. 9 Similarly, only CoCoNet and our method highlight the brightness of individuals, but CoCoNet loses the shape of people. M2Fusion only retains visible light information and loses all infrared information. Other methods also tend to highlight visible light information but not infrared information. Our method not only highlights people while retaining their shapes, but also sees the parts of their hands. Regarding scene information, except CoCoNet, all other methods tend to depict dark night scenes in visible images. The mark in the green box is difficult to identify clearly in other algorithms, but we can clearly observe the existence of this mark on each pillar.

[0099] Table 3 is a quantitative comparison of the fusion results obtained by the existing fusion method and the method proposed in the present invention on M3FD, where the best value, the second best value and the third best value are represented by bold, underline and double underline respectively.

[0100] Table 3 Quantitative comparison of various methods on the MSRS dataset.

[0101]

[0102]

[0103] These metric values ​​also prove that our proposed fusion network can achieve better fusion performance in complex scenes (M3FD). On MSRS, our method still achieves the four best metric values ​​(MI, SD, MI, and VIF). These observations show that our fusion method achieves higher interference resistance compared to the current state-of-the-art fusion methods.

[0104] (4) Fusion performance table on TNO dataset

[0105] Fig.10 The data in the figure is quite blurry in the infrared information. Notably, our method stands out by effectively suppressing the jagged noise that appears at the edges of the individuals highlighted by the red boxes, as well as mitigating the black pseudo-shadows emanating from these edges. Among the other methods, CoCoNet can reduce the appearance of black pseudo-shadows, although it cannot completely eliminate the destructive jagged noise. In contrast, M2Fusion exhibits a rather coarse method of extracting infrared information, capturing only the most prominent highlights while completely losing key edge details. When we turn our attention to the details in the visible light image, the chairs in the store are marked with green boxes, showing sharp outlines. The outputs of methods such as DenseFuse, LRRNet, CDDFuse, SegMif, and M2Fusion only capture the reflections on the armrests, failing to fully present a representation of the chair. In contrast, CoCoNet, SeAFusion, and SemLA manage to convey a blurry impression of the chair's shape. Notably, only our method provides a clear depiction of the entire chair, with all four corners clearly visible. Our resulting image shows the least blur, reflecting the superior clarity of our method. Table 4 is a quantitative comparison of the fusion results obtained by the existing fusion method and the method proposed in the present invention on TNO. Similarly, the best value, the second best value and the third best value are represented by bold, underline and double underline respectively.

[0106] Table 4 Quantitative comparison of various methods on the TNO dataset

[0107]

[0108] These metric values ​​also prove that our proposed fusion network can achieve better fusion performance in complex scenes (M3FD). On M3FD, our method still achieves the four best metric values ​​(MI, SD, MI, and VIF). These observations show that our fusion method achieves higher interference resistance compared to the current state-of-the-art fusion methods.

[0109] Through comparative analysis of the above four datasets and eight state-of-the-art algorithms in different scenarios, it can be seen that OCCO can perceive salient objects and enhance their presence while maximizing the optimization of scene information.

[0110] 2. Detection performance analysis

[0111] In addition, we verify the improvement of object detection on downstream visual task fusion results on two multimodal datasets, MSRS and M3FD. We train 40 epochs on these two datasets using the same pre-trained weights to obtain detection results on YOLOV5.

[0112] Although the labels from the MSRS dataset do not annotate distant pedestrians, such as Fig.11 As shown in the figure, after training, all models except DenseFuse can detect these pedestrians. The most significant difference is in the detection of vehicles, where only our method, SeAFusion, SegMif, and M2Fusion annotate two vehicles. Obviously, our confidence is the highest, surpassing the original image and all other methods.

[0113] The MSRS dataset also includes bicycle classification, but due to the low label quality, all methods only reach about 40% in terms of mAP, so they are not included in the statistics. Since OCCO is specifically learned for people and cars, we won the first place in the detection results of people and cars in Table 5.

[0114] Table 5 Comparison of detection performance of various methods on the MSRS dataset (bold is the first, double underline is the second)

[0115]

[0116] Table 6 Comparison of detection performance of various methods on the M3FD dataset (bold is the first, double underline is the second)

[0117]

[0118]

[0119] The M3FD dataset is not conducive to extracting portraits due to the excessive noise of infrared information. As can be seen from Table 6, OCCO performs well in the vehicle category, second only to the infrared mode itself, which is lacking in the infrared mode. However, due to the lack of attention to other scene information, the detection performance of "lights" is poor. In general, we have achieved the best performance. Fig.12 As shown in Figure 2, CDDFuse, CoCoNet, and SeAFusion mistakenly identify the human-shaped background plate as a person. In addition to the rider, our accuracy in detecting "people" is over 79%, which is significantly higher than other methods. We also perform well in identifying motorcycles.

Claims

1. A method for infrared and visible light image fusion guided by LVM based on object perception and context contrastive learning, comprising the following steps: Step S1: Construct feature interaction fusion network The feature interaction fusion network includes four convolution blocks for feature extraction, four feature interaction fusion blocks FIFB for feature fusion, and three up-convolution blocks; Each convolution block uses conventional convolution operations to extract the information of each modality. It includes two convolution units and a maximum pooling operation. Each convolution unit includes a 3×3 convolution, a batch normalization, and a ReLU activation function. Each upconvolution block includes two convolution units and an upsampling operation. After the visible light image vi and the infrared image ir are convolved through four layers with the same structure but different parameters, four single-modal features can be extracted: represents the unimodal features extracted from the visible light image after passing through the i-th layer conversion block, represents the unimodal features extracted from the infrared image after passing through the conversion block of the i-th layer, where i∈1,···,4, represents the number of layers, B represents the batch size, C represents the number of channels, H represents the feature height, and W represents the feature width; FIFB consists of three parts: spatial enhancement module, cross-channel attention module and cross-modal attention module; (1) The cross-channel attention module can enhance the semantic information between modalities; 4 unimodal features First, the average pooling and maximum pooling are performed respectively, and the pooling vectors obtained by the average pooling and the maximum pooling are cascaded, and then a linear mapping MLP and a sigmoid function are used in turn to obtain the channel weight CW∈R B×2×1×C ; These weights are then split into and This process can be formalized as follows: Where δ is the sigmoid function, Cat(·,·) represents the cascade operation, and MA represents two pooling operations, including maximum pooling and average pooling; Then, the intermediate features of the two modalities are enhanced by the channel attention weights to obtain Where * represents channel dimension multiplication; (2) The cross-modal attention module enables the model to learn richer modal feature representations and improves the ability to handle complex scenes through multimodal features in complementary regions; Will and It is flattened into a linear sequence in the channel dimension, denoted as and Then project the sequence into and Attention feature map Depend on The attention map is used to enhance the relevant parts of the visible image in the infrared image. The enhanced features are expressed as Similarly, we obtain (3) Spatial attention can enhance the important information of the modality itself; The spatial attention map is obtained by spatial multiplication of the MLP and the input features along the channel axis; the spatial enhancement process is expressed as where ⊙ represents the multiplication of spatial dimensions, C x×x is a convolutional layer with a kernel size of x×x, and R(·) is the relu activation function; At the same time, and Flatten into a sequence, respectively with Cascade; Finally, the two feature vectors are combined into a fusion vector represented as Where Ω is batch regularization; For channel restoration and fusion feature refinement, the output fusion feature is Calculated; DC 3×3 represents a depthwise convolution with a kernel size of 3×3; The four fused features obtained have decreasing size differences. In order to make the sizes of two adjacent fused features the same, it is necessary to complete the scale recovery through the up-convolution block. For the fused features obtained in the fourth layer, only the up-sampling operation is used to complete the scale recovery. Finally, the channel recovery is completed through a layer of convolution, and the fused image with the same size as the original image is output; Step S2: Training the feature interaction fusion network (1) During the training process, the SAM-guided mask generator is used to extract the mask information of salient targets in the visible light image vi and infrared image ir modalities and use it for contextual contrastive learning; SAM can accurately segment objects triggered by various cues; to further improve its accuracy, a text-driven object detection model, Ground DINO, is used to generate detection box cues for SAM; the text cue "scene contains pedestrians or cars" ensures that relevant aspects are captured in both modes, resulting in two masks: Where GD stands for Ground DINO; Ground DINO detects distant people in infrared images, which are not included in the labels provided by the dataset; With the box hint, SAM can accurately segment the individuals within the box, providing a finer segmentation than labels; These masks convey the maximum semantic data that SAM extracts from the source images; The original visible light image vi and infrared image ir are input into the SAM-guided mask generator to extract the mask information of the salient targets in the modalities of the visible light image vi and the infrared image ir, and the two mask information are divided into modality-specific information and shared information. The specific process is as follows: the shared information can be obtained by intersecting the two masks: where · represents the Hadamard product; the modality-specific data is the original mask obtained by removing the shared information: Finally, the background area can be obtained by removing these three semantic parts: Here 1 H×W represents an all-one matrix of size H×W; thus, three semantic masks are obtained and background masks based on these two modalities (2) For contextual contrastive learning, positive and negative samples are generated by calculating three semantic masks; When using As an information filter, the content extracted from the visible image is most beneficial to the segmentation network, while the information of the infrared image is redundant; therefore, the The samples obtained on the visible image are designated as positive samples and are denoted as The samples extracted from the infrared image are taken as negative samples and are denoted as The anchor sample at this time is Where f represents the fused image obtained through the fusion network; Contrastive learning will perform filtering operations three times based on three semantic information filters. Simply put, it can be understood as extracting samples with its own modality as a mask, so that it is a positive sample and the other modality is a negative sample. For the background part, it is necessary to retain the texture information in the visible light image to make the fused image more natural, that is, set the visible light image as a positive sample and the infrared image as a negative sample. A large number of negative samples are more conducive to anchor samples to learn the potential features between positive and negative samples; therefore, the samples are evenly divided to generate b groups in each batch; each batch consists of one anchor sample, one positive sample, and b negative samples; this grouping strategy means that the first group will generate anchors, positive samples, and negative samples at the same time, while the remaining b-1 groups only generate negative samples; the formula of the contrast loss function is as follows: L contrast =L uniqe +L s h are +L bg Where D(,) and D bg (,) represents the distance between two samples, v1 and v2 are fixed weights; the fused image f is the anchor sample, which is close to the positive sample and far away from the negative sample; Furthermore, the pre-trained VGG19 is used for feature extraction, but the context distance is only calculated on the deep features; the context loss uses the gamma matrix to convert the input features into feature points, calculates the cosine similarity of the two modal feature points, and normalizes them to convert from similarity to distance; in addition, the absolute distance also needs to improve the fusion accuracy; so the distance D between samples is expressed as: where vgg i (·) represents the i-th layer feature extracted by vgg; finally, the background information does not contain special semantic information, and the Euclidean distance is used to measure the similarity in the shallow feature space: Here d is the l2 norm; The model is trained using pixel-level loss functions and contextual loss to jointly constrain training: L pixel =L ssim +ω3L int +ω4L texture L ssim =2-SSIM(f,vi)-SSIM(f,ir) L total =L pixel +ω5L contrast Here L ssim is the structural similarity loss, L int is the brightness loss, L texture is the texture loss, L contrast is the context contrast loss; SSIM is the structural similarity calculation, I ir (i, j) is the pixel intensity of the infrared image at pixel point (i, j), represents the gradient calculation; ω3, ω4 and ω5 are hyperparameters.

Images