RGB-Infrred target detection method based on cross-modal alignment adaptation

Abstract

The invention discloses an RGB-Infrred target detection method based on cross-modal alignment adaptation, and the method comprises the steps: carrying out the preprocessing of an RGB image and an infrared image which are paired, extracting the multi-scale features of the two images, and flattening the multi-scale features into RGB tokens and infrared tokens; a position embedding vector is initialized for each infrared token, and the RGB tokens, the infrared tokens and the corresponding position embedding vectors are input into a DINO Transform encoder together; an alignment adapter is introduced into a DINO Transformer encoder, and the RGB features are aligned to the infrared features; the method comprises the following steps of: introducing a self-adaptive cross-modal adapter into a DINO Transform encoder, and carrying out interaction on two modal characteristics; the outputs of the two modes of encoders are added to obtain a multi-mode feature encoding result, and the multi-mode feature encoding result is input into a DINO Transformer decoder; a group of learning queries and anchor frames are initialized, and a multi-modal feature coding result is learned in a DINO Transform decoder through a multi-head cross attention mechanism; a self-adapter framework is introduced, and the target characterization quality is improved; according to the method, the limitation of an existing method in the aspect of pre-training model adaptation is effectively solved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to an RGB-Infrared target detection method based on cross-modal alignment adaptation. Background Technology

[0002] Object detection is a fundamental image analysis technique in the fields of computer vision and image processing. Currently, most object detection models are only designed for RGB images. However, RGB images are difficult to clearly display targets in complex environments with low light, such as fog or darkness, which limits the application of RGB-based detection models in real-world scenarios. Compared to RGB object detection, RGB-Infrared-based object detection can utilize two complementary modal data, RGB and infrared images, effectively meeting the needs of real-world scenarios. In research on RGB and infrared fusion, relevant patents and literature have proposed innovative methods for face anti-spoofing detection and small target detection. For example, CN117437677A proposes a multimodal face anti-spoofing detection method based on cross-modal fusion. This method extracts multi-scale features from RGB, IR, and depth images and performs cross-modal fusion to generate target feature maps, thereby improving the accuracy of face authenticity detection. CN117894082A introduces a multimodal face liveness detection method based on an attention mechanism. This method removes noise from face images through preprocessing of RGB, IR, and depth images, and combines convolutional neural networks with an attention mechanism to achieve multi-level feature fusion, improving the model's discrimination ability and accuracy. CN116863245A discloses a lightweight RGB-IR fusion small target detection method. This method aligns IR and RGB images through homography transformation and forms a four-channel tensor, which is then input into a lightweight network for target detection, suitable for embedded devices with limited computing power. All of these methods train the fusion model from scratch to meet task requirements.

[0003] In recent years, pre-trained base models have received widespread attention. These models learn on large datasets using a self-supervised approach, demonstrating strong generalization capabilities. With the rise of such large models, many researchers have begun to explore how to efficiently adapt these models to RGB-Infrared object detection tasks, such as Prompt-based and adapter-based methods. Unlike the above-mentioned de novo model training methods, the adaptation method in this invention trains a small number of added modules on top of the original base model. Specifically, the Prompt-based method can achieve efficient adaptation for multimodal tasks by introducing a small number of learnable tokens at the input level. However, this method mainly relies on the RGB modality and fails to fully utilize the characteristics of other modalities. In contrast, the adapter-based method, by inserting lightweight learnable modules into the pre-trained model, can better utilize the characteristics of different modalities.

[0004] While existing multimodal adaptation methods have shown great potential in object detection tasks, they still face challenges in RGB-Infrared object detection. First, RGB-T images are often misaligned, increasing the difficulty of associating targets across different modalities. Second, the relative detection capabilities of RGB and infrared modalities are significantly affected by the environment, and feature fusion without differentiation introduces substantial noise. Based on these technical problems, this invention provides an RGB-Infrared object detection method based on cross-modal alignment adaptation. Summary of the Invention

[0005] To address the aforementioned technical problems in existing technologies, this invention provides an RGB-Infrared target detection method based on cross-modal alignment adaptation.

[0006] The present invention adopts the following technical solution: This invention provides an RGB-Infrared target detection method based on cross-modal alignment adaptation, comprising: Step 1, pair the RGB images and infrared images Preprocessing is performed by using a ResNet-50 network to extract multi-scale features from the two images, flattening the multi-scale features into RGB tokens and infrared tokens; a position embedding vector is initialized for each infrared token, and the RGB tokens, infrared tokens and their corresponding position embedding vectors are input into the DINO Transformer encoder. Step 2: Introduce an alignment adapter in the DINO Transformer encoder to align RGB features to infrared features; restore RGB tokens and infrared tokens to RGB feature maps and infrared feature maps, respectively, and perform uniform windowing and offset windowing on the two feature maps; within each group of windows, align RGB features to infrared features through affine transformation, merge the alignment results under the two window divisions and add them on average to obtain the final aligned RGB feature map; flatten the final aligned RGB feature map and infrared feature map into aligned tokens, and input them to the next level module; Step 3: An adaptive cross-modal adapter is introduced into the DINO Transformer encoder to interact with the features of the two modalities; adaptive fusion weights are calculated based on the aligned RGB tokens and infrared tokens, and the fusion weights are combined with the corresponding modal tokens and summed using the Hadamard product operation to obtain the preliminary fusion features; a two-layer bottleneck architecture is used to enhance the representation capability of the preliminary fusion features to obtain the final fusion features; modality-independent spatial attention modules are used to learn the specific features of the two modalities respectively, and the specific features are summed with the final fusion features to obtain the encoder outputs of the two modalities; the encoder outputs of the two modalities are added together to obtain the multimodal feature encoding result and input into the DINO Transformer decoder; Step 4: Initialize a set of learning queries and anchor boxes, and learn the multimodal feature encoding results in the DINO Transformer decoder through a multi-head cross-attention mechanism; introduce a self-adaptor architecture to parameterize and adjust the learned features to improve the quality of target representation; the DINO Transformer decoder outputs optimized learnable queries and anchor boxes for target detection.

[0007] Furthermore, in step 1, the paired RGB images and infrared images Preprocessing includes: For the input RGB image and infrared images Perform normalization to scale the image pixel values ​​to the [0,1] range and enlarge the image size to 1.5 times the original image size; Using a ResNet-50 pre-trained on the ImageNet-1k dataset as the backbone network, the original dimensions of the two images are extracted. Figure 1 Multi-scale feature maps of 4, 1 / 8, 1 / 16, and 1 / 32 are obtained. The multi-scale feature maps are flattened to obtain RGB tokens and infrared tokens respectively. A position embedding vector is randomly initialized for each infrared token. The RGB tokens and infrared tokens are concatenated with their respective position embedding vectors and then input into the DINO Transformer encoder.

[0008] Furthermore, in step 2, the RGB features are aligned to the infrared features through an affine transformation, as shown in equation (1), including: (1), in, The coordinates of the aligned RGB features. To align the coordinates of the RGB features before, It is a 2×3 affine transformation matrix. The parameters are learned through the network of equation (2): (2), in, For the k-th window in the modal feature map, For splicing layers, It is a four-layer convolutional network. For the activation function layer, It is a fully connected layer.

[0009] Furthermore, step 2, merging the alignment results under the two window divisions, includes: The RGB features after uniform window partitioning and the RGB features after offset window partitioning are processed by affine transformation to obtain uniform window alignment features. Alignment features with offset window As shown in equation (3): (3), in, Indicates passage Features Perform affine transformation. This represents the k-th RGB window that is uniformly divided. The k-th RGB window is divided by offset; Align features with uniform windows Alignment features with offset window The averages are added together to obtain the final aligned RGB feature maps. As shown in equation (4): (4).

[0010] Furthermore, step 3, calculating the adaptive fusion weights, includes: In the alignment tokens settings, the alignment tokens corresponding to the RGB modality are set as follows: H RGB The alignment tokens corresponding to the infrared mode are H T ; Using a lightweight multilayer perceptron, respectively... H RGB and H T The feature transformation is performed, and then the two intermediate weight values ​​are obtained by passing the sigmoid activation function. The fusion weights of the RGB modes are obtained by normalizing the intermediate values ​​of the two weights using a normalization function. W RGB Fusion weights with infrared modesW T As shown in equation (5): (5), in, H RGB and H T For RGB and infrared feature tokens, It is a lightweight multilayer perceptron. It is the sigmoid activation function. This is a normalization function that returns the normalized result of the two input values.

[0011] Furthermore, the preliminary fusion features obtained in step 3 include: The fusion weight W of the RGB modes RGB With H RGB Perform the Hadamard product operation to obtain the RGB weighted features; The fusion weight W of the infrared modes T With H T Perform the Hadamard product operation to obtain the infrared weighted features; The RGB weighted features and the infrared weighted features are summed to obtain the preliminary fusion feature Hmid, as shown in equation (6): (6), in, This indicates that the normalized weights and the original features will be multiplied by Hadamard.

[0012] Furthermore, in step 3, a two-layer bottleneck architecture is used to enhance the representational capability of the preliminary fused features, resulting in the final fused features, including: Perform a dimensionality reduction projection operation on the initial fused feature Hmid to obtain the dimensionality-reduced feature; perform a linear transformation on the dimensionality-reduced feature to obtain the linearly transformed feature; perform a dimensionality increase projection operation on the linearly transformed feature to obtain the final fused feature H. fuse As shown in equation (7): (7), in, Indicates up-dimensional projection. Represents a linear transformation. This represents a dimension reduction projection.

[0013] Furthermore, through RGB modal-independent spatial attention modules The initial fusion feature Hmid is processed to obtain RGB-specific features; the RGB-specific features are then combined with the final fusion feature H. fuse Add them together to get the encoder output for the RGB mode. As shown in equation (8): (8), Through infrared modal-independent spatial attention module The initial fusion feature Hmid is processed to obtain the infrared-specific feature; the infrared-specific feature is then combined with the final fusion feature H. fuse Adding them together yields the encoder output for the infrared mode. As shown in equation (9): (9).

[0014] Furthermore, the parameterization adjustment of the self-adaptor architecture in step 4 includes: In the DINO Transformer decoder, the multimodal feature encoding results serve as the Key and Value of the attention mechanism, while the initialized learnable query and anchor boxes serve as the Query. The initial decoded features are obtained through a multi-head cross-attention mechanism. ; The initial decoding features Input self-adaptor, self-adaptor sequentially to Perform dimensionality reduction projection, linear transformation, and dimensionality increase projection to obtain the optimized decoding features. As shown in equation (10): (10) Optimized decoding features The corresponding learnable queries and anchor boxes are used to output the target's category, location, width, and height information.

[0015] Furthermore, in step 1, when the image size is enlarged to 1.5 times that of the original image, the maximum image size is limited to 1200×1200.

[0016] Compared with the prior art, the superior effects of the present invention are as follows: 1. The RGB-Infrared object detection method based on cross-modal alignment adaptation described in this invention improves the adaptability of RGB-Infrared object detection by introducing an adapter mechanism and is specifically designed for the characteristics of RGB-Infrared object detection tasks, thereby effectively solving the limitations of existing methods in terms of pre-trained model adaptation. It proposes an alignment adapter and an adaptive cross-modal adapter to improve the problems of poor modal space alignment and environmental dynamic change interference in RGB-Infrared object detection, respectively, and significantly improve the feature fusion quality and the generalization ability of the model. 2. The RGB-Infrared target detection method based on cross-modal alignment adaptation described in this invention performs excellently in RGB-Infrared target detection, and can achieve accurate and robust RGB-Infrared target detection. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the RGB-Infrared target detection method based on cross-modal alignment adaptation in an embodiment of the present invention. Detailed Implementation

[0018] To better understand the above-mentioned objectives, features and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. Example

[0019] This invention proposes an innovative target detection framework called "cross-modal alignment and adaptation RGB-Infrared target detection framework". It solves problems such as misalignment of images from different modalities and large amount of noise caused by simple fusion by using inter-modal and intra-modal adapters, and makes full use of multimodal features for target detection tasks. like Figure 1 As shown, the RGB-Infrared target detection method based on cross-modal alignment adaptation includes: Step 1, pair the RGB images and infrared images Preprocessing is performed by using a ResNet-50 network to extract multi-scale features from the two images, flattening the multi-scale features into RGB tokens and infrared tokens; a position embedding vector is initialized for each infrared token, and the RGB tokens, infrared tokens and their corresponding position embedding vectors are input into the DINO Transformer encoder. Step 2: Introduce an alignment adapter in the DINO Transformer encoder to align RGB features to infrared features; restore RGB tokens and infrared tokens to RGB feature maps and infrared feature maps, respectively, and perform uniform windowing and offset windowing on the two feature maps; within each group of windows, align RGB features to infrared features through affine transformation, merge the alignment results under the two window divisions and add them on average to obtain the final aligned RGB feature map; flatten the final aligned RGB feature map and infrared feature map into aligned tokens, and input them to the next level module; Step 3: An adaptive cross-modal adapter is introduced into the DINO Transformer encoder to interact with the features of the two modalities; adaptive fusion weights are calculated based on the aligned RGB tokens and infrared tokens, and the fusion weights are combined with the corresponding modal tokens and summed using the Hadamard product operation to obtain the preliminary fusion features; a two-layer bottleneck architecture is used to enhance the representational capability of the preliminary fusion features to obtain the final fusion features; the specific features of the two modalities are learned separately through modal-independent spatial attention modules, and the specific features are summed with the final fusion features to obtain the encoder outputs of the two modalities; the encoder outputs of the two modalities are added together to obtain the multimodal feature encoding result and input into the DINO Transformer decoder; Step 4: Initialize a set of learnable queries and anchor boxes. In the DINO Transformer decoder, multimodal feature encoding results are learned through a multi-head cross-attention mechanism. A self-adaptor architecture is introduced to parameterize and adjust the learned features to improve the quality of target representation. The DINO Transformer decoder outputs optimized learnable queries and anchor boxes for target detection.

[0020] Furthermore, in step 1, the paired RGB images and infrared images Preprocessing includes: For the input RGB image and infrared images Perform normalization to scale the image pixel values ​​to the [0,1] range and enlarge the image size to 1.5 times the original image size; Using a ResNet-50 pre-trained on the ImageNet-1k dataset as the backbone network, the original dimensions of the two images are extracted. Figure 1 Multi-scale feature maps of 4, 1 / 8, 1 / 16, and 1 / 32 are obtained. The multi-scale feature maps are flattened to obtain RGB tokens and infrared tokens respectively. A position embedding vector is randomly initialized for each infrared token. The RGB tokens and infrared tokens are concatenated with their respective position embedding vectors and then input into the DINO Transformer encoder.

[0021] Furthermore, in step 2, the RGB features are aligned to the infrared features through an affine transformation, as shown in equation (1), including: (1), in, The coordinates of the aligned RGB features. To align the coordinates of the RGB features before, It is a 2×3 affine transformation matrix. The parameters are learned through the network of equation (2): (2), in, For the k-th window in the modal feature map, For splicing layers, It is a four-layer convolutional network. For the activation function layer, It is a fully connected layer.

[0022] Furthermore, step 2, merging the alignment results under the two window divisions, includes: The RGB features after uniform window partitioning and the RGB features after offset window partitioning are processed by affine transformation to obtain uniform window alignment features. Alignment features with offset window As shown in equation (3): (3), in, Indicates passage Features Perform affine transformation. This represents the k-th RGB window that is uniformly divided. The k-th RGB window is divided by offset; Align features with uniform windows Alignment features with offset window The averages are added together to obtain the final aligned RGB feature maps. As shown in equation (4): (4).

[0023] Furthermore, step 3, calculating the adaptive fusion weights, includes: In the alignment tokens settings, the alignment tokens corresponding to the RGB modality are set as follows: H RGB The alignment tokens corresponding to the infrared modes are H T ; Using a lightweight multilayer perceptron, respectively... H RGB and H T The feature transformation is performed, and then the two intermediate weight values ​​are obtained by passing the sigmoid activation function. The fusion weights of the RGB modes are obtained by normalizing the intermediate values ​​of the two weights using a normalization function. W RGB Fusion weights with infrared modesW T As shown in equation (5): (5), in, H RGB and H T For RGB and infrared feature tokens, It is a lightweight multilayer perceptron. It is the sigmoid activation function. This is a normalization function that returns the normalized result of the two input values.

[0024] Furthermore, the preliminary fusion features obtained in step 3 include: The fusion weight W of the RGB modes RGB With H RGB Perform the Hadamard product operation to obtain the RGB weighted features; The fusion weight W of the infrared modes T With H T Perform the Hadamard product operation to obtain the infrared weighted features; The RGB weighted features and the infrared weighted features are summed to obtain the preliminary fusion feature Hmid, as shown in equation (6): (6), in, This indicates that the normalized weights and the original features will be multiplied by Hadamard.

[0025] Furthermore, in step 3, a two-layer bottleneck architecture is used to enhance the representational capability of the preliminary fused features, resulting in the final fused features, including: Perform a dimensionality reduction projection operation on the initial fused feature Hmid to obtain the dimensionality-reduced feature; perform a linear transformation on the dimensionality-reduced feature to obtain the linearly transformed feature; perform a dimensionality increase projection operation on the linearly transformed feature to obtain the final fused feature H. fuse As shown in equation (7): (7), in, Indicates up-dimensional projection. Represents a linear transformation. This represents a dimension reduction projection.

[0026] Furthermore, through RGB modal-independent spatial attention modules The initial fusion feature Hmid is processed to obtain RGB-specific features; the RGB-specific features are then combined with the final fusion feature H. fuse Add them together to get the encoder output for the RGB mode. As shown in equation (8): (8), Through infrared modal-independent spatial attention module The initial fusion feature Hmid is processed to obtain the infrared-specific feature; the infrared-specific feature is then combined with the final fusion feature H. fuse Adding them together yields the encoder output for the infrared mode. As shown in equation (9): (9).

[0027] Furthermore, the parameterization adjustment of the self-adaptor architecture in step 4 includes: In the DINO Transformer decoder, the multimodal feature encoding results serve as the Key and Value of the attention mechanism, while the initialized learnable query and anchor boxes serve as the Query. The initial decoded features are obtained through a multi-head cross-attention mechanism. ; The initial decoding features Input self-adaptor, self-adaptor sequentially to Perform dimensionality reduction projection, linear transformation, and dimensionality increase projection to obtain the optimized decoding features. As shown in equation (10): (10) Optimized decoding features The corresponding learnable queries and anchor boxes are used to output the target's category, location, width, and height information.

[0028] Furthermore, in step 1, when the image size is enlarged to 1.5 times that of the original image, the maximum image size is limited to 1200×1200.

[0029] To further illustrate, the RGB-Infrared target detection method based on cross-modal alignment adaptation described in this invention includes two stages: model training and image target detection and recognition. Phase 1, Model Training: Model input preprocessing, where the model input consists of pairs of RGB-Infrared images and target location category labels; Image preprocessing involves enlarging the image by 1.5 times, with a maximum size of 1200*1200; image normalization is then performed, which scales the pixel value range of the image to the standard [0,1] range, thereby ensuring the stability of the image values, facilitating information extraction, enhancing the generalization ability of the deep model, and accelerating model convergence. Image normalization does not affect the information storage and visual features of the image. Image feature calculation: The preprocessed image is used to extract multi-scale image features using ResNet-50 pre-trained on ImageNet-1k provided by DINO. The features of each scale are flattened to obtain tokens, and a location embedding vector is randomly initialized for each token. The two-modal tokens and the corresponding location embedding vectors are input into the DINO Transformer encoder. The tokens are transformed into feature maps, and then they are divided into uniform and offset windows. Based on the divided windows, affine transformations are performed within both window divisions to achieve local alignment of RGB features to infrared features, as shown in equation (1) and equation (2) above. Using the affine transformation network described above, uniform window alignment features are obtained. Alignment features with offset window As shown in equation (3) above; The results of the two window affine transformations are combined into a feature map, and the average sum is used to obtain the aligned RGB modal feature map, as shown in equation (4) above. The feature maps of both modalities are restored to the shape of tokens; first, the fusion weights are dynamically generated based on the features of each modality, as shown in equation (5) above; The preliminary fusion characteristics were calculated, as shown in equation (6); Modality-independent spatial attention modules are used to learn modality-specific features, and a two-layer bottleneck architecture is used to improve the representational ability of fused features, generating modality features with known cross-modality semantics, as shown in Equations (11) and (12): (11), (12), The two modal tokens are added to the corresponding position embedding codes to obtain the fused encoding result, which is then input into the Transformer decoder. The fused feature is decoded by randomly initializing a set of learning queries and anchor boxes, and multi-head cross-attention is used to learn the fused features. The target representation ability is optimized through a self-adaptor.

[0030] The second phase is model testing. The input preprocessing involves enlarging the image size by 1.5 times, with a maximum size of 1200*1200; image normalization is performed, which scales the pixel value range of the image to the standard [0,1] range; the preprocessed image is then processed through a backbone network to extract multi-scale features, which are then flattened into tokens; the location embedding vector is initialized using a random initialization strategy during the initialization phase, and the multi-modal tokens and location embedding vectors are input into the DINO Transformer encoder; In the alignment adapter, the tokens are first restored to feature maps, and the feature maps are divided into uniform windows and offset windows. Within the divided windows, affine transformations are performed to align the RGB features to the infrared features at the window level. Then, the average sum of the two transformation results is obtained to obtain the aligned RGB features. Finally, the two modal feature maps are flattened into tokens. The fusion weights are calculated based on the modal features, and a preliminary fusion result is obtained based on the fusion weights. Modality-independent spatial attention modules are used to learn modality-specific features, and a two-layer bottleneck architecture is used to improve the representational ability of the fusion features. The two are summed to obtain the modality features with known cross-modal semantics. The two modality features are summed to obtain the final fusion feature. A set of learning queries and anchor boxes is initialized, and multi-head cross-attention is used in the DINO Transformer decoder to learn fused features. Then, its target representation ability is optimized through self-adaptor to obtain the target detection results. Table 1

[0031] To verify the superior technical effect of the method described in the embodiments of this application, the method is applied in... Tests were conducted on publicly available datasets. The RGB method represents a target detection method adapted to NIDO using RGB modalities; the T method represents a target detection method adapted to DINO using infrared images; RGB-T represents a method adapted to DINO using both RGB and infrared images but without using the alignment adapter and cross-modal adaptive adapter proposed in this application; Ours represents the method described in the embodiments of this application; Day, Overcast, Night, Challenge, and All represent daytime, cloudy, nighttime, complex scenes, and comprehensive situations, respectively; as shown in Table 1, On the dataset, the method described in this application achieved mAP50 of 82.5%, 90.0%, 93.1%, and 88.8% in daytime, cloudy, nighttime, and complex scenarios, respectively, and 86.9% in the comprehensive scenario (All).

[0032] This invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed. The scope of protection of this invention is defined by the appended claims.

Claims

1. An RGB-Infrared target detection method based on cross-modal alignment adaptation, characterized in that, include: Step 1, pair the RGB images and infrared images Preprocessing is performed by using a ResNet-50 network to extract multi-scale features from the two images, flattening the multi-scale features into RGB tokens and infrared tokens; a position embedding vector is initialized for each infrared token, and the RGB tokens, infrared tokens and their corresponding position embedding vectors are input into the DINO Transformer encoder. Step 2: Introduce an alignment adapter in the DINO Transformer encoder to align RGB features to infrared features; restore RGB tokens and infrared tokens to RGB feature maps and infrared feature maps, respectively, and perform uniform window division and offset window division on the two feature maps; within each group of windows, align RGB features to infrared features through affine transformation, merge the alignment results under the two window divisions and add them on average to obtain the final aligned RGB feature map; The finally aligned RGB feature map and infrared feature map are flattened into aligned tokens and input into the next level module; Step 3: Introduce an adaptive cross-modal adapter into the DINO Transformer encoder to interact with the features of the two modalities; Adaptive fusion weights are calculated based on the aligned RGB tokens and infrared tokens. The fusion weights are combined with the corresponding modal tokens and summed using Hadamard product operations to obtain preliminary fusion features. A two-layer bottleneck architecture is used to enhance the representational capabilities of the preliminary fusion features to obtain the final fusion features. Modality-independent spatial attention modules are used to learn the specific features of the two modalities respectively. The specific features are summed with the final fusion features to obtain the encoder outputs of the two modalities. The encoder outputs of the two modalities are added together to obtain the multimodal feature encoding result and input into the DINO Transformer decoder. Step 4: Initialize a set of learning queries and anchor boxes, and learn the multimodal feature encoding results in the DINO Transformer decoder through a multi-head cross-attention mechanism; A self-adaptor architecture is introduced to parameterize the learned features and improve the quality of the target representation. The DINO Transformer decoder outputs optimized learnable queries and anchor boxes for object detection.

2. The RGB-Infrared target detection method based on cross-modal alignment adaptation according to claim 1, characterized in that, In step 1, the paired RGB images and infrared images Preprocessing includes: For the input RGB image and infrared images Perform normalization to scale the image pixel values ​​to the [0,1] range and enlarge the image size to 1.5 times the original image size; A ResNet-50 pre-trained on the ImageNet-1k dataset is used as the backbone network to extract multi-scale feature maps of two images with sizes of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image. The multi-scale feature maps are flattened to obtain RGB tokens and infrared tokens respectively. A position embedding vector is randomly initialized for each infrared token. The RGB tokens and infrared tokens are concatenated with their respective position embedding vectors and then input into the DINO Transformer encoder.

3. The RGB-Infrared target detection method based on cross-modal alignment adaptation according to claim 1, characterized in that, In step 2, the RGB features are aligned to the infrared features through an affine transformation, as shown in equation (1), including: （1）， in, The coordinates of the aligned RGB features. To align the coordinates of the RGB features before, It is a 2×3 affine transformation matrix. The parameters are learned through the network of equation (2): （2）， in, For the k-th window in the modal feature map, For splicing layers, It is a four-layer convolutional network. For the activation function layer, It is a fully connected layer.

4. The RGB-Infrared target detection method based on cross-modal alignment adaptation according to claim 3, characterized in that, Step 2, which merges the alignment results from the two window divisions, includes: The RGB features after uniform window partitioning and the RGB features after offset window partitioning are processed by affine transformation to obtain uniform window alignment features. Alignment features with offset window As shown in equation (3): （3）， in, Indicates passage Features Perform affine transformation. This represents the k-th RGB window that is uniformly divided. The k-th RGB window is divided by offset; Align features with uniform windows Alignment features with offset window The averages are added together to obtain the final aligned RGB feature maps. As shown in equation (4): （4）。 5. The RGB-Infrared target detection method based on cross-modal alignment adaptation according to claim 1, characterized in that, Step 3, calculating the adaptive fusion weights, includes: In the alignment tokens settings, the alignment tokens corresponding to the RGB modality are set as follows: H RGB The alignment tokens corresponding to the infrared mode are H T ; Using a lightweight multilayer perceptron, respectively... H RGB and H T The feature transformation is performed, and then the two intermediate weight values ​​are obtained by passing the sigmoid activation function. The fusion weights of the RGB modes are obtained by normalizing the intermediate values ​​of the two weights using a normalization function. W RGB Fusion weights with infrared modes W T As shown in equation (5): （5）， in, H RGB and H T For RGB and infrared feature tokens, It is a lightweight multilayer perceptron. It is the sigmoid activation function. This is a normalization function that returns the normalized result of the two input values.

6. The RGB-Infrared target detection method based on cross-modal alignment adaptation according to claim 5, characterized in that, The preliminary fusion features obtained in step 3 include: The fusion weight W of the RGB modes RGB With H RGB Perform the Hadamard product operation to obtain the RGB weighted features; The fusion weight W of the infrared modes T With H T Perform the Hadamard product operation to obtain the infrared weighted features; The RGB weighted features and the infrared weighted features are summed to obtain the preliminary fusion feature Hmid, as shown in equation (6): （6）， in, This indicates that the normalized weights and the original features will be multiplied by Hadamard.

7. The RGB-Infrared target detection method based on cross-modal alignment adaptation according to claim 6, characterized in that, The two-layer bottleneck architecture in step 3 includes: Perform a dimensionality reduction projection operation on the initial fused feature Hmid to obtain the dimensionality-reduced feature; perform a linear transformation on the dimensionality-reduced feature to obtain the linearly transformed feature; perform a dimensionality increase projection operation on the linearly transformed feature to obtain the final fused feature H. fuse As shown in equation (7): （7）， in, Indicates up-dimensional projection. Represents a linear transformation. This represents a dimension reduction projection.

8. The RGB-Infrared target detection method based on cross-modal alignment adaptation according to claim 7, characterized in that, Through RGB modal independent spatial attention module The initial fusion feature Hmid is processed to obtain RGB-specific features; Combine RGB specific features with the final fusion feature H fuse Add them together to get the encoder output for the RGB mode. As shown in equation (8): （8）， Through infrared modal-independent spatial attention module The initial fusion feature Hmid is processed to obtain infrared-specific features; Infrared specific features are combined with the final fusion feature H fuse Adding them together yields the encoder output for the infrared mode. As shown in equation (9): （9）。 9. The RGB-Infrared target detection method based on cross-modal alignment adaptation according to claim 1, characterized in that, The parameterization adjustment of the self-adaptor architecture in step 4 includes: In the DINO Transformer decoder, the multimodal feature encoding results serve as the Key and Value of the attention mechanism, while the initialized learnable query and anchor boxes serve as the Query. The initial decoded features are obtained through a multi-head cross-attention mechanism. ; The initial decoding features Input self-adaptor, self-adaptor sequentially to Perform dimensionality reduction projection, linear transformation, and dimensionality increase projection to obtain the optimized decoding features. As shown in equation (10): （10）， Optimized decoding features The corresponding learnable queries and anchor boxes are used to output the target's category, location, width, and height information.

10. The RGB-Infrared target detection method based on cross-modal alignment adaptation according to claim 2, characterized in that, In step 1, when the image size is enlarged to 1.5 times that of the original image, the maximum image size is limited to 1200×1200.

Images