Non-aligned visible light-thermal infrared video target detection method based on multi-modal basic model guidance

Abstract

The invention discloses a non-aligned visible light-thermal infrared video target detection method based on multi-modal basic model guidance, and the method comprises the steps: carrying out the feature extraction of an original visible light image and an original thermal infrared image in the same group through a multi-modal basic model encoder, and obtaining the respective modal representation; the two-stage progressive aligner is used for completing coarse-to-fine explicit alignment of the thermal infrared image relative to the visible light image on the pixel level and the image level of two modals, so that a fine-aligned thermal infrared image is obtained. And then carrying out basic model guided alignment consistency constraint on the fine alignment thermal infrared image and the original visible light image, finally realizing multi-modal fusion under receptive field difference by using a thermal infrared to visible light fusion module, and outputting a final detection frame. According to the method, modal difference is broken through large-scale pre-training prior knowledge, dependence on registration data is avoided, visible light-thermal infrared image alignment is achieved more efficiently and stably, and the method is excellent in performance in downstream detection tasks.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image detection technology, specifically relating to an unaligned visible light-thermal infrared video target detection method guided by a multimodal basic model. Background Technology

[0002] Visible-to-thermal infrared (RGBT) video target detection aims to fuse complementary information from visible light (RGB) and thermal infrared (Thermal) modes to achieve robust target detection in challenging environments such as low lighting, cluttered backgrounds, or severe weather. However, real-world RGBT image pairs often suffer from spatial misalignment due to sensor parallax, focal length differences, and other factors. Existing methods for addressing this misalignment problem are mainly categorized into four types: methods based on manually aligning RGBT image pairs, methods based on homography matrix annotation, and methods based on self-supervised learning and unsupervised learning.

[0003] The first type of method relies on manually aligning data, eliminating parallax through extensive manual annotation or cropping. This is not only time-consuming and labor-intensive, but also difficult to handle complex nonlinear distortions. Jia et al., Llvip: A visible-infrared paired dataset for low-light vision. IEEE / CVF International Conference on ComputerVision, 2021., established a standard manual alignment data processing workflow.

[0004] The second type of method attempts to achieve cross-modal alignment by estimating affine transformation parameters. (Deng et al., Cross-modality and cross-resolution homography estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024). This method learns and estimates the homography matrix, extending it to downstream tasks such as object detection. However, these methods typically require accurate homography matrix annotations for supervised learning, which is costly, and the homography matrix is ​​difficult to characterize the complex nonlinear misalignment caused by disparity.

[0005] The third and fourth types of methods achieve alignment by projecting features from two modalities into a shared latent space and utilizing self-supervised or unsupervised learning. (Wang et al., Improving misaligned multi-modality imagefusion with one-stage progressive dense registration. IEEE Transactions on Circuits and Systems for Video Technology, 2024). However, the alignment learned by these methods is often based on synthetic artificial constraints rather than real-world geometric or semantic correspondences, resulting in poor model generalization ability and difficulty in handling complex and diverse misalignment scenarios (such as resolution differences, viewpoint changes, etc.). They can only learn the implicit latent space and ignore the inherent differences between modalities.

[0006] For example, Chinese patent application CN116912298A discloses a progressive UAV visible light, thermal, and infrared image registration method. It uses a supervised network to learn registration capabilities after registration, but it still does not escape the limitation of consuming manual annotation. Chinese patent application CN121095296A discloses an infrared image registration method and an unsupervised learning image registration model training method. This scheme artificially constructs non-aligned image pairs, which does not escape the scope of manual annotation. Moreover, the model belongs to the third and fourth categories, learns implicit alignment strategies, and has poor generalization ability.

[0007] In summary, existing methods generally rely on manual data alignment or additional geometric truth supervision, which has problems such as high annotation costs, poor adaptability to complex nonlinear misalignments, and insufficient generalization ability in real-world scenarios. Summary of the Invention

[0008] Purpose of the invention: The purpose of this invention is to address the shortcomings of existing technologies and provide a non-aligned visible light-thermal infrared video target detection method based on a multimodal fundamental model. By breaking modal differences through large-scale pre-trained prior knowledge, it avoids dependence on registration data, thereby achieving visible light-thermal infrared image alignment more efficiently and stably, and performs excellently in downstream detection tasks.

[0009] Technical Solution: This invention provides a method for detecting unaligned visible light-thermal infrared video targets based on a multimodal base model. The method constructs and trains a target detection network, inputs the same set of unaligned visible light images and thermal infrared images into the target detection network for feature fusion, and outputs the target detection result. The target detection network includes a multimodal base model encoder, a two-stage progressive aligner, an imageBind-guided alignment consistency constraint, a thermal infrared-to-visible light fusion module (T2RF), and a temporal detection head, which sequentially executes the following steps:

[0010] Step 1: Use a multimodal fundamental model encoder to extract features from the same set of original visible light images and original thermal infrared images to obtain infrared mode representations in the same embedding space. and visible light mode representation The multimodal base model encoder can use ImageBind or the Bind series, such as LanguageBind, PointBind, OmniBind, etc., or other unified embedding space models trained based on multimodal contrastive learning, and set the modality-specific encoder parameters of the base model to freeze to stably provide cross-modal alignment priors.

[0011] Step 2: Represent the infrared modes and visible light mode representation The input is fed into a two-stage progressive aligner, which performs explicit coarse-to-fine alignment of the thermal infrared image relative to the visible light image at the pixel / image level; the two-stage progressive aligner includes a linear transformation module and a warp transformation module.

[0012] The Linear Transformation module predicts the global position (center coordinates (x,y)) and scale information (H,W) of the thermal infrared image in the visible light coordinate system based on the two-modal embedding difference output by the base model. It generates linear transformation parameters and resamples the thermal infrared image once to achieve global coarse alignment. (The center coordinates (x,y) and scale (H,W) are ultimately used for affine sampling mesh in the linear transformation stage.)

[0013] The Warp Transformation module, based on the global coarse alignment result, estimates cross-modal pixel-level differences through multi-level dense matching, predicts the nonlinear deformation field, and resamples to compensate for misalignments such as disparity and local deformation that are difficult for linear models to cover, thus obtaining a finely aligned thermal infrared image. ;

[0014] Step 3: Align the fine thermal infrared image Alignment consistency constraints between the base model and the original visible light image This is used to ensure stable convergence of the alignment process and avoid semantic drift or meaningless deformation in the fine alignment stage. Specifically, it means that the aligned thermal infrared image and the original visible light image are fed back into the basic model encoder, and the consistency constraint loss of the two is calculated in their unified embedding space, so that the alignment results are continuously close at the "semantic to geometric consistency" level.

[0015] Step 4: After alignment, use the thermal infrared to visible light fusion module to perform multimodal fusion, but avoid global misfusion caused by the limitation of the thermal infrared mode receptive field. That is, first calculate the cross-modal local similarity to enhance the effective area, and then use the same source attention mechanism. The thermal infrared features are used as both query and key, and the visible light features are used as values, so that the attention is more biased towards the areas with clearer foreground boundaries in the thermal infrared mode, thereby achieving higher fidelity region fusion.

[0016] Step 5, Timing Detection Header: Used to temporally complement the information of the current frame and the reference frame, and output the final detection box (it can also be used for non-video, i.e., image detection, in which case the timing information will not be used).

[0017] This invention leverages the cross-modal consistency and emergent alignment capabilities formed by a multimodal base model within a unified embedding space. It transforms the alignment process from an implicit constraint of zero-learning to an explicit alignment guided by prior knowledge of the base model, achieving stable optimization through a coarse-to-fine progressive alignment strategy. Specifically, inspired by the multimodal base model, this invention proposes a two-stage progressive aligner and an alignment consistency constraint loss. By mimicking the manual alignment process, the two-stage progressive aligner effectively addresses the error problem caused by the misalignment of multimodal images. Furthermore, after alignment, under resolution differences, an effective region fusion strategy fully utilizes the advantages of thermal infrared, ultimately improving the performance of visible light-thermal infrared video target detection.

[0018] Furthermore, the two-stage progressive aligner receives infrared mode representations. and visible light mode representation Then (where L is the fixed embedding length), difference magnification is performed to better predict the alignment parameters. The detailed processing procedure is as follows:

[0019] Step 2.1, Linear Transformation Stage,

[0020] Step 2.1.1: Embed the two modalities in the channel dimension and concatenate them to obtain a joint representation. To simultaneously capture which semantic channels better reflect misalignment and which embedding positions better reflect misalignment, two types of amplification coefficients, channel weight and spatial weight, are constructed through the difference amplification module. Channel weight is used to weight the embedded channel dimension, and spatial weight is used to weight the length dimension.

[0021] In the channel weight branch, first consider the joint representation. Perform average pooling and max pooling in L dimensions to obtain the channel description, i.e., let and Let L represent the pooling operator along the L-dimensional plane, then its output shapes are respectively The vectors are concatenated to obtain the channel description. :

[0022] ;

[0023] Then, the channel description Input multilayer perceptron Compress to And via the Sigmoid function The weight vector is normalized to the interval (0,1), let This means that a vector of length 2C is split into two segments, the first segment being C and the second segment being C, where C is a fixed number of channels. This yields the channel amplification factors corresponding to the visible light channels. Channel amplification factor with thermal infrared :

[0024] ;

[0025] The channel amplification factor weights characterize which channel dimensions of the two modes should be emphasized in a unified embedding space to reflect the misalignment differences;

[0026] In the spatial weight branch, convolutional networks are used to weight the joint representation. Perform C-dimensional modeling, that is, let This represents a 1D convolution operator along the C-dimensional plane. The activation function is used; a weight map is generated through the structure "Conv–ReLU–Conv–Sigmoid", and its output shape is as follows. Then through Spatial magnification factor of visible light obtained by separation Spatial magnification factor of thermal infrared ;

[0027] ;

[0028] The spatial magnification factor weights characterize which positions in the L-dimensional plane contain stronger dislocation cues and should be magnified in the two modes;

[0029] Step 2.1.2: After obtaining the channel weights and spatial weights, the difference amplification module enhances the embedding by injecting counterpart modal information. With the original embedding ( When multiplying, scale each L column; when and During multiplication, each channel row is scaled.

[0030] For thermal infrared mode enhancement, spatial weights and channel weights of visible light are used to represent the visible light modes, respectively. The results are amplified, averaged, and then injected into the infrared mode representation. Enhanced thermal infrared embedding ;

[0031] ;

[0032] When enhancing visible light modes, spatial weights and channel weights of thermal infrared are used to represent infrared modes respectively. The results are amplified, averaged, and then injected into the visible light mode representation. Enhanced visible light embedding :

[0033] ;

[0034] Where ⊙ represents element-wise multiplication;

[0035] The aforementioned cross-modal injection enhancement has two effects: first, it explicitly emphasizes the channels and positions sensitive to misalignment by using the weights of the other modality (e.g., visible light), thereby more strongly introducing differences that can indicate geometric misalignment into another modality; second, it preserves the stable semantic structure of the original embedding through residual form, avoiding semantic drift caused by over-amplification.

[0036] Step 2.1.3: To provide the amplified misalignment cues to the linear alignment parameter regressor, construct an enhanced embedding differential representation, that is, predict the center coordinates and scale parameters of the thermal infrared in the visible light coordinate system through differential estimation:

[0037] ;

[0038] in To subtract element by element, The regression head prediction network consists of several convolutions and pooling operations, outputting the center coordinates (x, y) and scale (H, W); an affine sampling grid is constructed from the predicted (x, y, H, W). : ;

[0039] in This represents an operator that constructs a sampling grid from parameters;

[0040] grid Coarse alignment results obtained by applying the thermal infrared image ;

[0041] Step 2.2, Twisting Transformation Stage:

[0042] Based on linear alignment, nonlinear deformation is estimated through multi-level matching blocks. Let the visible light characteristics and thermal infrared characteristics of the j-th matching layer be respectively... And the thermal infrared features are based on the previous deformation field Obtained by performing nonlinear transformation and local reconstruction The first layer does not have the deformation field of the previous layer, and then the cross-modal correlation is calculated. The expression is:

[0043] ;

[0044] Cross-modal correlation Visible light characteristics and the deformation field of the next layer After splicing, the deformation field of the current layer is restored. And progressive refinement is achieved using residual connections:

[0045] ;

[0046] in For splicing, It is a convolutional layer;

[0047] income Used to achieve fine local correction after image sampling, solving non-linear alignment caused by camera differences;

[0048] Finally, the overall deformation field is obtained. ,right Resampling yields finely aligned thermal infrared images .

[0049] Furthermore, the detailed method for using the alignment consistency constraint guided by the basic model in step 3 is as follows:

[0050] Fine alignment of thermal infrared images The original visible light image and the original visible light image are fed into the basic model encoder to obtain the embedded image. and ;

[0051] Choose the global representation class token from both, and denote the selection operator as . Calculate the similarity matrix :

[0052]

[0053] Take similarity matrix The alignment consistency loss is obtained by taking the average of the main diagonal elements. :

[0054]

[0055] in, This refers to taking the elements from the main diagonal. This refers to calculating the mean;

[0056] This loss causes the two modalities to converge within the unified embedding space of the base model, thereby stabilizing the two-stage alignment optimization and suppressing semantic drift.

[0057] Furthermore, the detailed method for multimodal fusion by the thermal infrared to visible light fusion module in step 4 is as follows:

[0058] Suppose that the local features of the two modes at a certain scale after alignment are, respectively, expanded by position as follows: ,in h and w are the feature scales, representing height and width, respectively; the expansion is done row-wise, with a total size of [size missing]. ;

[0059] Then calculate the cross-modal cosine similarity. : ;

[0060] Then use cross-modal cosine similarity The features of both sides are mutually enhanced (⊙ represents element-wise multiplication), and the nonlinear expression is preserved by ReLU:

[0061] ;

[0062] ;

[0063] Subsequently, using thermal infrared light as the query key source for homologous attention, the expansion operator was analyzed. (Flattening the space) Defines attention affinity :

[0064] ;

[0065] in For activation function,

[0066] Finally, affinity is used to perform matrix multiplication on the features to enhance the visible light features, while residual connections are used to ensure the integrity of the original visible light features, resulting in the final fused features. :

[0067] ;

[0068] in Represents matrix multiplication. The unfolded features are restored to a spatial structure. For activation function, The weighting factor is a learnable factor that learns the degree to which infrared features affect visible light.

[0069] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0070] 1. This invention guides visible light-thermal infrared image alignment based on a multimodal foundational model: It utilizes a unified embedding space trained through multimodal contrastive learning to ensure transferable priors in semantic and spatial consistency across different modalities. This prior serves as an external knowledge source for alignment learning, transforming alignment from an implicit constraint learned from scratch into an explicit optimization guided by the foundational model. The foundational model is not limited to ImageBind, but also includes the Bind series and other unified embedding space models derived from multimodal contrastive learning.

[0071] 2. The two-stage progressive aligner and consistency loss of this invention: Both the two-stage progressive aligner and consistency loss are built based on a multimodal fundamental model. A progressive strategy of achieving stable convergence through linear coarse alignment for global localization and scale correction, and nonlinear fine alignment for fine-grained correction, is employed. Furthermore, an alignment consistency loss is introduced into the unified embedding space to continuously impose semantic and geometric consistency constraints on the alignment results, thereby suppressing drift and meaningless deformation in the fine alignment stage and improving generalization ability and interpretability.

[0072] 3. This invention performs thermal infrared to visible light fusion: After completing explicit alignment, regional high-fidelity fusion is achieved through cross-modal similarity enhancement and homologous attention mechanism, so that the advantages of thermal infrared in terms of foreground boundary and low-light visibility can be fully utilized, while avoiding global misfusion caused by the limitation of thermal infrared receptive field and resolution difference, thereby further improving the downstream target detection performance. Attached Figure Description

[0073] Figure 1 This is a schematic diagram of the overall network framework of the present invention;

[0074] Figure 2 This is a diagram illustrating the alignment effect in the embodiment.

[0075] Figure 3 This is a diagram illustrating the results of inconsistent alignment in the embodiments;

[0076] Figure 4 This is a diagram illustrating the thermal infrared to visible light fusion in the embodiment;

[0077] Figure 5 The above is a comparison diagram of detection results in the example. Detailed Implementation

[0078] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0079] like Figure 1 As shown, this invention provides a method for detecting unaligned visible light-thermal infrared video targets based on a multimodal fundamental model. The method constructs and trains a target detection network, inputs the same set of unaligned visible light images and thermal infrared images into the target detection network for feature fusion, and outputs the target detection result. The target detection network includes a multimodal fundamental model encoder, a two-stage progressive aligner, a fundamental model-guided alignment consistency constraint, a thermal infrared-to-visible light fusion module, and a temporal detection head, which sequentially executes the following steps:

[0080] Step 1: Use a multimodal fundamental model encoder to extract features from the same set of original visible light images and original thermal infrared images to obtain infrared mode representations in the same embedding space. and visible light mode representation ;

[0081] Step 2: Represent the infrared modes and visible light mode representation The input is fed into a two-stage progressive aligner to perform coarse-to-fine explicit alignment of the thermal infrared image relative to the visible light image at both the pixel and image levels. The two-stage progressive aligner includes a linear transformation module and a warp transformation module. The linear transformation module predicts the global position and scale information of the thermal infrared image in the visible light coordinate system based on the modal embedding difference between the two modes described in step 1, generates linear transformation parameters, and resamples the thermal infrared image once to achieve global coarse alignment.

[0082] The distortion transform module, based on global coarse alignment, estimates cross-modal pixel-level differences through multi-level dense matching, predicts the nonlinear deformation field, and resamples to obtain fine-aligned thermal infrared images. ;

[0083] Step 3: Align the fine thermal infrared image Alignment consistency constraints are applied to the original visible light image and the base model-guided image. Specifically, the finely aligned thermal infrared image and the original visible light image are fed back into the multimodal base model encoder, and the alignment consistency constraint loss is calculated within their unified embedding space. ;

[0084] Step 4: Using the thermal infrared to visible light fusion module, first calculate the cross-modal local similarity, and then use the same source attention mechanism to use thermal infrared features as both query and key, and visible light features as value, so that the attention is more biased towards the region with clearer foreground boundaries in the thermal infrared mode, thereby achieving multimodal fusion.

[0085] Step 5: The timing detection head performs timing complementation between the information of the current frame and the reference frame, and outputs the final detection box.

[0086] The multimodal base model encoder of the present invention can use ImageBind, or can be extended to the Bind series, or other models based on multimodal contrastive learning training that map multimodalities to a shared embedding space, so as to achieve transferable and reusable alignment guidance capabilities.

[0087] The two-stage progressive aligner in this embodiment receives infrared modal representation. and visible light mode representation Then, difference amplification is performed to predict alignment parameters. The detailed processing procedure is as follows:

[0088] Step 2.1, Linear Transformation Stage,

[0089] Step 2.1.1: Embed the two modalities in the channel dimension and concatenate them to obtain a joint representation. Two types of amplification coefficients, channel weight and spatial weight, are constructed through the difference amplification module. The channel weight is used to weight the embedded channel dimension, and the spatial weight is used to weight the length dimension.

[0090] In the channel weight branch, first consider the joint representation. Perform average pooling and max pooling in L dimensions to obtain the channel description, i.e., let and Let L represent the pooling operator along the L-dimensional plane, then its output shapes are respectively The vectors are concatenated to obtain the channel description. :

[0091] ;

[0092] Then, the channel description Input multilayer perceptron Compress to And via the Sigmoid function The weight vector is normalized to the interval (0,1), let This means that a vector of length 2C is split into two segments, the first segment being C and the second segment being C, where C is a fixed number of channels. This yields the channel amplification factors corresponding to the visible light channels. Channel amplification factor with thermal infrared :

[0093] ;

[0094] In the spatial weight branch, convolutional networks are used to weight the joint representation. Perform C-dimensional modeling, that is, let This represents a 1D convolution operator along the C-dimensional plane. The activation function is used; a weight map is generated through a Conv–ReLU–Conv–Sigmoid structure, and its output shape is as follows. Then through Spatial magnification factor of visible light obtained by separation Spatial magnification factor of thermal infrared ;

[0095] ;

[0096] Step 2.1.2: After obtaining the channel weights and spatial weights, the difference amplification module enhances the embedding by injecting the other modality information. When the spatial amplification coefficients of the two modalities are multiplied by the modality representation obtained in Step 1, each L column is scaled. When the channel coefficients of the two modalities are multiplied by the modality representation obtained in Step 1, each channel row is scaled.

[0097] For thermal infrared mode enhancement, spatial weights and channel weights of visible light are used to represent the visible light modes, respectively. The results are amplified, averaged, and then injected into the infrared mode representation. Enhanced thermal infrared embedding ;

[0098] ;

[0099] When enhancing visible light modes, spatial weights and channel weights of thermal infrared are used to represent infrared modes respectively. The results are amplified, averaged, and then injected into the visible light mode representation. Enhanced visible light embedding :

[0100] ;

[0101] Where ⊙ represents element-wise multiplication;

[0102] Step 2.1.3: Predict the center coordinates and scale parameters of the thermal infrared radiation in the visible light coordinate system using difference analysis:

[0103] ;

[0104] in To subtract element by element, The regression head prediction network consists of several convolutions and pooling operations, outputting the center coordinates (x, y) and scale (H, W); an affine sampling grid is constructed from the predicted (x, y, H, W). : ;

[0105] in This represents an operator that constructs a sampling grid from parameters;

[0106] grid Coarse alignment results obtained by applying the thermal infrared image ;

[0107] Step 2.2, the distortion transformation stage: Based on linear alignment, nonlinear deformation is estimated through multi-level matching blocks. Let the visible light characteristics and thermal infrared characteristics of the j-th matching layer be respectively... And the thermal infrared features are based on the previous deformation field Obtained by performing nonlinear transformation and local reconstruction Then calculate the cross-modal correlation. The expression is as follows:

[0108] ;

[0109] Cross-modal correlation Visible light characteristics and the deformation field of the next layer After splicing, the deformation field of the current layer is restored. And progressive refinement is achieved using residual connections:

[0110] ;

[0111] in For splicing, It is a convolutional layer;

[0112] Finally, the overall deformation field is obtained. ,right Resampling yields finely aligned thermal infrared images .

[0113] The detailed method for step 3 of this embodiment, which uses the alignment consistency constraint guided by the basic model, is as follows:

[0114] Fine alignment of thermal infrared images The original visible light image and the original visible light image are fed into the basic model encoder to obtain the embedded image. and ;

[0115] Choose the global representation class token from both, and denote the selection operator as . Calculate the similarity matrix :

[0116] ;

[0117] Take similarity matrix The alignment consistency loss is obtained by taking the average of the main diagonal elements. :

[0118] ;

[0119] in, This refers to taking the elements from the main diagonal. It refers to calculating the mean.

[0120] The detailed method for multimodal fusion by the thermal infrared to visible light fusion module in step 4 of this embodiment is as follows:

[0121] The local features of the two aligned modalities at a certain scale are unfolded according to their positions as follows: ,in h and w are the scales of the local features, representing height and width, respectively; the expansion is done row-wise, with the total size being [size missing]. ;

[0122] Then calculate the cross-modal cosine similarity. : ;

[0123] Then use cross-modal cosine similarity The features of both sides are mutually enhanced (⊙ represents element-wise multiplication), and the nonlinear expression is preserved by ReLU:

[0124] ;

[0125] ;

[0126] Subsequently, using thermal infrared light as the query key source for homologous attention, the expansion operator was analyzed. (Flattening the space) Defines attention affinity :

[0127] ;

[0128] in For activation function,

[0129] Finally, affinity is used to perform matrix multiplication on the features to enhance the visible light features, while residual connections are used to ensure the integrity of the original visible light features, resulting in the final fused features. :

[0130] ;

[0131] in, Represents matrix multiplication. The unfolded features are restored to a spatial structure. For activation function, The weighting factor is a learnable factor that learns the degree to which infrared features affect visible light.

[0132] Table 1 Comparison of Method Results

[0133] Table 2 Comparison of Method Results

[0134] Table 3 Comparison of Method Results

[0135] This embodiment evaluates the algorithm of this application by comparing its performance with state-of-the-art RGBT object detection methods on the UVT-VOD2024, VT-VOD50, LLVIP, and CVC-14 datasets to verify the effectiveness of the proposed method (Table 1). The most commonly used evaluation metrics for each dataset are selected for quantitative performance evaluation.

[0136] Commonly used average precision metrics were applied to datasets such as UVT-VOD2024, VT-VOD50, and LLVIP. and take And the overall AP (average IoU from 0.50 to 0.95 with a step size of 0.05), Table 2. Log-mean false negative rate was used for the CVC-14 dataset. (within the range of FPPI values) arrive (Calculated above, lower values ​​indicate better performance, as shown in Table 3).

[0137] This embodiment performs target detection on a set of unaligned visible light and thermal infrared images of a specific scene. The alignment effect of applying the technical solution of this invention is as follows: Figure 2 As shown, the effect of two-modal fusion is as follows: Figure 4 As shown, the existing technology does not achieve the effect of alignment consistency. Figure 3 As shown, the effects of the present invention compared with existing methods are as follows: Figure 5 As shown.

[0138] In summary, this invention demonstrates significant advantages on multiple basic datasets, and it reduces the reliance on manual alignment and additional geometric annotations.

Claims

1. A non-aligned visible light-thermal infrared video target detection method guided by a multimodal fundamental model, characterized in that, A target detection network is constructed and trained. The same set of unaligned visible light images and thermal infrared images are input into the target detection network for feature fusion, and the target detection result is output. The target detection network includes a multimodal base model encoder, a two-stage progressive aligner, a base model-guided alignment consistency constraint, a thermal infrared-to-visible light fusion module, and a temporal detection head. The following steps are executed sequentially: Step 1: Use a multimodal fundamental model encoder to extract features from the same set of original visible light images and original thermal infrared images to obtain infrared mode representations in the same embedding space. and visible light mode representation ; Step 2: Represent the infrared modes and visible light mode representation The input is fed into a two-stage progressive aligner, which performs explicit coarse-to-fine alignment of the thermal infrared image relative to the visible light image at both the pixel and image levels. The two-stage progressive aligner includes a linear transformation module and a warp transformation module. The linear transformation module predicts the global position and scale information of the thermal infrared image in the visible light coordinate system based on the mode embedding difference between the two modes described in step 1, generates linear transformation parameters, and performs a resampling of the thermal infrared image to achieve global coarse alignment. The distortion transform module, based on global coarse alignment, estimates cross-modal pixel-level differences through multi-level dense matching, predicts the nonlinear deformation field, and resamples to obtain fine-aligned thermal infrared images. ; Step 3: Align the fine thermal infrared image Alignment consistency constraints are applied to the original visible light image and the base model-guided image. Specifically, the finely aligned thermal infrared image and the original visible light image are fed back into the multimodal base model encoder, and the alignment consistency constraint loss is calculated within their unified embedding space. ; Step 4: Using the thermal infrared to visible light fusion module, first calculate the cross-modal local similarity, and then use the same source attention mechanism to use thermal infrared features as both query and key, and visible light features as value, so that the attention is more biased towards the region with clearer foreground boundaries in the thermal infrared mode, thereby achieving multimodal fusion. Step 5: The timing detection head performs timing complementation between the information of the current frame and the reference frame, and outputs the final detection box.

2. The unaligned visible-thermal infrared video target detection method based on a multimodal fundamental model as described in claim 1, characterized in that: The two-stage progressive aligner receives infrared mode representation and visible light mode representation Then, difference amplification is performed to predict alignment parameters. The detailed processing procedure is as follows: Step 2.1, Linear Transformation Stage, Step 2.1.1: Embed the two modalities in the channel dimension and concatenate them to obtain a joint representation. Two types of amplification coefficients, channel weight and spatial weight, are constructed through the difference amplification module. The channel weight is used to weight the embedded channel dimension, and the spatial weight is used to weight the length dimension. In the channel weight branch, first consider the joint representation. Perform average pooling and max pooling in L dimensions to obtain the channel description, i.e., let and Let L represent the pooling operator along the L-dimensional plane, then its output shapes are respectively The vectors are concatenated to obtain the channel description. : ； Then, the channel description Input multilayer perceptron Compress to And via the Sigmoid function The weight vector is normalized to the interval (0,1), let This means that a vector of length 2C is split into two segments, the first segment being C and the second segment being C, where C is a fixed number of channels. This yields the channel amplification factors corresponding to the visible light channels. Channel amplification factor with thermal infrared : ； In the spatial weight branch, convolutional networks are used to weight the joint representation. Perform C-dimensional modeling, that is, let This represents a 1D convolution operator along the C-dimensional plane. The activation function is used; a weight map is generated through a Conv–ReLU–Conv–Sigmoid structure, and its output shape is as follows. Then through Spatial magnification factor of visible light obtained by separation Spatial magnification factor of thermal infrared ; ； Step 2.1.2: After obtaining the channel weights and spatial weights, the difference amplification module enhances the embedding by injecting the other modality information. When the spatial amplification coefficients of the two modalities are multiplied by the modality representation obtained in Step 1, each L column is scaled. When the channel coefficients of the two modalities are multiplied by the modality representation obtained in Step 1, each channel row is scaled. For thermal infrared mode enhancement, spatial weights and channel weights of visible light are used to represent the visible light modes, respectively. The results are amplified, averaged, and then injected into the infrared mode representation. Enhanced thermal infrared embedding ; ； When enhancing visible light modes, spatial weights and channel weights of thermal infrared are used to represent infrared modes respectively. The results are amplified, averaged, and then injected into the visible light mode representation. Enhanced visible light embedding : ； Where ⊙ represents element-wise multiplication; Step 2.1.3: Predict the center coordinates and scale parameters of the thermal infrared radiation in the visible light coordinate system using difference analysis: ； in To subtract element by element, The regression head prediction network consists of several convolutions and pooling operations, outputting the center coordinates (x, y) and scale (H, W); an affine sampling grid is constructed from the predicted (x, y, H, W). : ; in This represents an operator that constructs a sampling grid from parameters; grid Coarse alignment results obtained by applying the thermal infrared image ; Step 2.2, the distortion transformation stage: Based on linear alignment, nonlinear deformation is estimated through multi-level matching blocks. Let the visible light characteristics and thermal infrared characteristics of the j-th matching layer be respectively... And the thermal infrared features are based on the previous deformation field Obtained by performing nonlinear transformation and local reconstruction Then calculate the cross-modal correlation. The expression is as follows: ； Cross-modal correlation Visible light characteristics and the deformation field of the next layer After splicing, the deformation field of the current layer is restored. And progressive refinement is achieved using residual connections: ； in For splicing, It is a convolutional layer; Finally, the overall deformation field is obtained. ,right Resampling yields finely aligned thermal infrared images .

3. The unaligned visible-thermal infrared video target detection method based on a multimodal fundamental model guided by claim 1, characterized in that, The detailed method for the alignment consistency constraint guided by the basic model in step 3 is as follows: Fine alignment of thermal infrared images The original visible light image and the original visible light image are fed into the basic model encoder to obtain the embedded image. and ; Choose the global representation class token from both, and denote the selection operator as . Calculate the similarity matrix : ； Take similarity matrix The alignment consistency loss is obtained by taking the average of the main diagonal elements. : ； in, This refers to taking the elements from the main diagonal. It refers to calculating the mean.

4. The non-aligned visible light-thermal infrared video target detection method based on a multimodal fundamental model guided by claim 1, characterized in that, The detailed method for multimodal fusion by the thermal infrared to visible light fusion module in step 4 is as follows: The local features of the two aligned modalities at a certain scale are unfolded according to their positions as follows: ,in h and w are the scales of the local features, representing height and width, respectively; the expansion is done row-wise, with the total size being [size missing]. ; Then calculate the cross-modal cosine similarity. : ; Then use cross-modal cosine similarity The features of both sides are mutually enhanced (⊙ represents element-wise multiplication), and the nonlinear expression is preserved by ReLU: ； ； Subsequently, using thermal infrared light as the query key source for homologous attention, the expansion operator was analyzed. (Flattening the space) Defines attention affinity : ； in For activation function, Finally, affinity is used to perform matrix multiplication on the features to enhance the visible light features, while residual connections are used to ensure the integrity of the original visible light features, resulting in the final fused features. : ； in, Represents matrix multiplication. The unfolded features are restored to a spatial structure. For activation function, The weighting factor is a learnable factor that learns the degree to which infrared features affect visible light.

Images