A multi-modal image feature matching method combining global affine transformation and local deformable convolution

Abstract

This invention discloses a multimodal image feature matching method combining global affine transformation and local deformable convolution, comprising the following steps: constructing a multimodal target detection network architecture to obtain original visible light feature maps and original infrared feature maps; using a global affine transformation module to globally regulate large-scale rigid misalignment between images to obtain a coarsely aligned visible light feature map; extracting modality-invariant features to provide matching references for downstream networks, and performing deformable convolution on the two modality-invariant features; combining feature modulation masks to obtain the final finely aligned visible light feature map; performing feature fusion to output the final detection result. This invention, through global affine transformation, a priori corrects rigid geometric distortion between images, compressing spatial errors to the effective capture range of the fine alignment module, thereby ensuring the convergence and robustness of subsequent fine alignment and resolving the contradiction between large-scale spatial mismatch and limited local receptive field.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a multimodal image feature matching method that combines global affine transformation and local deformable convolution. Background Technology

[0002] In the field of computer vision technology, multimodal target detection technology is a research hotspot, with high research value in multiple fields such as video surveillance, intelligent transportation, and drone inspection. In particular, visible light-infrared multimodal image fusion detection technology, which combines infrared and visible light images, can utilize the rich texture details of visible light images and the thermal radiation penetration capability of infrared images, exhibiting strong robustness in complex scenarios such as changing lighting and inclement weather.

[0003] However, in practical visible-infrared multimodal image fusion detection applications, especially in scenarios involving UAVs equipped with dual-light pods, significant spatial misalignment often occurs between paired visible and infrared images due to differences in the physical positions, focal lengths, and imaging times of the visible and infrared sensors, as well as camera motion and complex target movements. To achieve efficient multimodal feature fusion, high-precision matching of the extracted features must be performed within the network.

[0004] Currently, in the field of multimodal feature alignment, existing problems can be divided into two main categories. The first is the global misalignment problem caused by differences in imaging time and radiometric distortion between multimodal sensors. Radiometric distortion typically occurs during sensor imaging; the spectral emissivity of ground objects differs from their true spectral emissivity. These differences cause the image to present different features for the same object, resulting in scale and positional differences for the same object in two modal images due to radiometric distortion. Furthermore, due to different imaging times of different sensors, clock deviations between the two sensors can cause pixel misalignment in image pairs, leading to inconsistent positions and angles of the same object in different modalities. The second problem is the inevitable local inconsistency in the multimodal spatial distribution caused by complex target motion, resulting in feature mismatch at corresponding locations. Since detection tasks are highly sensitive to spatial location, this type of problem leads to a significant performance degradation. Solutions to this problem mainly include two types of corresponding methods: one is alignment methods based on global rigid transformations, such as predicting the global homography matrix or affine transformation matrix, such as translation, rotation, and scaling, to uniformly distort the entire feature map. The drawback of this type of approach is that real-world scenes often involve depth variations, and targets may undergo localized non-rigid deformations. Global transformations cannot address the fine-grained mismatches caused by complex target motion. Another type of alignment method is based on local flexible transformations, such as using optical flow or deformable convolutions to predict pixel-level offsets for alignment. The drawback of this approach is that the search range, or receptive field, of deformable convolutions is limited. When there are large displacements such as translations or significant rotations between visible light and infrared images, the true features often exceed the effective capture range of the convolution kernel, causing the network to blindly search and even learn incorrect background noise, thus resulting in alignment failure.

[0005] Furthermore, existing local matching methods typically predict offsets by directly calculating the differences between two original features by identifying the distribution at the image or feature level. However, in visible-infrared multimodal imaging tasks, these differences stem not only from spatial offsets but also from modal discrepancies, which arise from the differences in imaging principles and wavelengths between visible-light and infrared sensors. Due to the fundamental heterogeneity of their imaging mechanisms—for example, visible light reflects surface reflectivity and color while infrared reflects the thermal radiation intensity of an object—the pixel distribution and contrast of the same physical target often differ drastically between the two modes. If the network directly calculates the spatial offset based on the original features containing this modality-specific information, it introduces a large amount of noise interference, potentially leading to difficulties in correctly matching cross-modal representations spatially, resulting in misleading offset estimates.

[0006] A method for fusing visible light and infrared images based on feature matching from the perspective of an unmanned aerial vehicle (UAV) is disclosed in Chinese patent document CN120823463B. This method involves image processing technology, including an image preprocessing module, a target detection module, a feature extraction module, a feature-level fusion module, and a feature reconstruction module. The above method abandons the traditional pixel-by-pixel processing approach and instead starts from the feature level. By extracting and aligning the salient features of the same target in the visible light image and the infrared image, it achieves efficient and robust image fusion. However, the original feature fusion method based on feature matching from the perspective of an UAV is affected by modal differences, resulting in low matching accuracy, and it lacks a two-level (coarse and fine) hierarchical matching approach.

[0007] A method for detecting aerial targets based on the fusion of infrared and visible light images from different perspectives is disclosed in Chinese patent document CN120526267A. This method includes: performing an i-th fusion operation based on the output feature map of the (i-1)-th fusion operation to obtain a pair of output feature maps of the i-th fusion operation and an i-th fused feature map F1; the fusion operation includes an addition operation or at least one first operation; the first operation includes, in sequence: feature matching, feature fusion, modal normalization processing and addition operation, the output feature map of the 0th fusion operation is the feature obtained by feature extraction from the infrared image to be tested and the visible light image to be tested, the infrared image to be tested and the visible light image to be tested have the same shooting scene but different shooting angles; and performing visible light modal detection and infrared modal detection on all fused feature maps F1-FN to obtain the target detection result. The invention described above can achieve target detection based on infrared and visible light images from different perspectives, thereby improving detection efficiency and expanding applicable scenarios. However, the aerial target detection method based on the fusion of infrared and visible light images from different perspectives suffers from insufficient global feature matching and local pixel alignment accuracy.

[0008] A style transfer-based method for detecting unaligned visible and infrared modal fusion targets is disclosed in Chinese patent document CN115578246B. This style transfer-based method employs a learnable affine transformation network to perform affine transformations on the infrared modalities, achieving feature alignment between modalities and effectively improving the modal fusion effect. The invention uses a learnable affine transform module (LATM) for feature alignment and a style transfer fusion module for inter-modal feature fusion. This invention can effectively handle the task of detecting unaligned visible and infrared images. The unaligned visible and infrared modal fusion target detection method employs a learnable affine transformation network to perform affine transformations on the infrared modalities, achieving feature alignment between modalities and effectively improving the modal fusion effect. The invention employs a learnable affine transform module (LATM) for feature alignment and a style transfer fusion module for intermodal feature fusion. This invention can effectively address the target detection task of unaligned visible light and infrared image fusion. However, this style transfer-based unaligned visible light and infrared modal fusion target detection method does not eliminate cross-modal interference at the feature level, and the matching accuracy has an upper limit.

[0009] To address the shortcomings of the existing technologies, providing a multimodal image feature matching method that combines global affine transformation and local deformable convolution is a problem worthy of further research. Summary of the Invention

[0010] The purpose of this invention is to overcome the problems of existing registration methods neglecting the contradiction between global and local features and introducing noise due to modal differences, resulting in poor registration performance. This invention provides a multimodal image feature matching method that combines global affine transformation and local deformable convolution, which integrates two types of alignment strategies from coarse to fine, achieving stable operation and accurate fusion in multimodal target detection.

[0011] The objective of this invention is achieved through the following technical solution: A multimodal image feature matching method combining global affine transformation and local deformable convolution includes the following steps: Step 1: Construct a multimodal target detection network architecture. Input the paired raw visible light images and raw infrared images into the dual-stream backbone network for feature extraction to obtain the raw visible light feature map and the raw infrared feature map. Step 2: Use the global affine transformation module to globally control the large-scale rigid misalignment between images to obtain a coarsely aligned visible light feature map; Step 3: Construct a mode-invariant feature encoder. Input the coarsely aligned visible light feature map and the original infrared feature map into the mode-invariant feature encoder respectively. Extract mode-invariant features through the constraints of two loss functions to provide a matching reference for the downstream network without interference from mode-specific information. Step 4: Perform deformable convolution on the two modality-invariant features, combine it with the feature modulation mask, and use the deformable convolution kernel output by the two modality-invariant features to adaptively adjust and fine-tune the coarsely aligned visible light feature map at the pixel level to obtain the final finely aligned visible light feature map. Step 5: After completing all spatial matching work, perform feature fusion, input the fused features into the downstream target detection head, and output the final detection result.

[0012] Preferably, the multimodal target detection network architecture in step one includes a dual-stream backbone network, a coarse alignment module, a fine alignment module, and a detection head; The specific process of building a multimodal object detection network architecture is as follows: Pairs of visible light images to be processed... With infrared thermal imaging images The data are fed into a dual-stream backbone network, where convolutional neural networks are used for forward propagation and feature extraction to obtain the original visible light feature maps. and original infrared feature map At this point, due to camera parallax and target motion, the two feature maps are spatially misaligned.

[0013] Preferably, the global affine transformation module alignment in step two specifically includes the following steps: S1, using the original infrared feature map As a spatial reference, the original visible light feature map within the proposed box area will be used. Compared with the original infrared feature map Perform pixel-by-pixel subtraction to calculate the feature difference map; S2. Input the difference map into a multilayer fully connected network for regression to predict the global positional deviation, size deviation, and angular deviation, and combine them into an affine parameter matrix. S3. Resample the original visible light feature map based on the affine parameter matrix, and output a coarsely aligned visible light feature map through global translation and rotation. We use feature difference map regression of affine parameters to directly calculate the global offset in the high-dimensional feature space.

[0014] Preferably, the modality-invariant feature encoder in step three internally uses a 5-layer densely connected block, where the output of each layer is concatenated with the outputs of all preceding layers before being input to the next layer; subsequently, it enters a fusion block consisting of 3 consecutive convolutional layers, ultimately outputting visible light invariant feature maps. Infrared Invariant Feature Map The five densely connected blocks reuse shallow features layer by layer, fully preserving cross-modal shared geometric information such as image edges and contours, and will not lose subtle target boundaries due to multiple convolutions.

[0015] Preferably, the modality-invariant feature encoder introduces the following two losses during the training phase: Structural similarity loss , , This loss forces the visible light invariant characteristic Infrared invariant features It exhibits high consistency in macroscopic spatial topology and contrast distribution, effectively filtering out modality-specific information such as color and temperature. Dual-path gradient fidelity loss , , in For a two-dimensional discrete spatial gradient operator, the loss constraint is that regardless of whether the features are visible light invariant or infrared invariant, the sharp physical abrupt edges of their respective original input images must be preserved. This mechanism avoids the feature mean smoothing phenomenon, provides high-precision physical boundary anchors for subsequent local alignment, and avoids the feature collapse shortcut where the network reduces all features to 0. The gradient calculation and loss backpropagation mentioned above are performed during the model training phase. During deployment, a forward lightweight feature encoder is executed to reduce the time overhead of this step, avoid the disappearance of small target contours, and ensure sufficient anchor points for fine local alignment.

[0016] Preferably, the specific operation process of performing deformable convolution on the two modality-invariant features is as follows: the deformable convolution kernel used subsequently has K samples, the output feature modulation mask is processed by the Sigmoid activation function, and its value range is limited to (0,1). This mask represents the confidence weight of the network for the features extracted from these K sampling points. The confidence mask weighted sampling feature is introduced to distinguish between the target effective pixels and background noise pixels, suppress the erroneous offset learning in the background region, and reduce the probability of false matching.

[0017] Preferably, the specific process for calculating the finely aligned visible light feature map in step four is as follows: For any center pixel on the target feature map Establish a regular sampling grid R for the standard convolution kernel (the grid of the 3×3 convolution is R={(-1,-1),(-1,0),…,(1,1)}), and denote the k-th standard sampling coordinate offset in the grid as . ; Deformable convolution kernels traverse coarse-aligned visible light invariant feature maps At that time, its actual sampling position is the coordinate: To accurately obtain the feature value at this location ; Multiply the extracted feature values ​​by the corresponding feature modulation weights. Then, combined with the inherent learned weights of the deformable convolution kernel. We perform weighted summation to obtain the finely aligned target output feature values. Its discrete mathematical calculation formula is as follows: , After traversing the entire image and performing the above calculations, the network finally outputs a finely aligned visible light feature map. It integrates the inherent weights of convolution, sampling offset, and confidence mask, and the sampling results take into account semantic features, spatial coordinates, and pixel credibility. The local alignment accuracy is much higher than that of traditional fixed grid convolution.

[0018] Preferably, the feature fusion and calculation process in step five specifically includes: Perform element-by-element addition or channel stitching on the precisely aligned visible light feature map and the original infrared feature map; It is then fed into the subsequent classification and regression detection head to output the final target category and accurate bounding box localization results. Two compatible fusion strategies are provided, which can be flexibly selected according to the scenario. Channel splicing retains all information of both channels, while element-wise addition compresses channels and reduces the amount of computation.

[0019] Positive and beneficial effects: 1. This multimodal image feature matching method, which combines global affine transformation and local deformable convolution, overcomes the limitations of a single alignment method. It adopts a cascaded strategy from coarse to fine. Without a coarse alignment module, relying solely on a fine alignment module based on deformable convolution, limited by the local receptive field of the convolution kernel, cannot capture large translational and rotational deviations between images, easily getting trapped in local optima and ignoring global deformation. Furthermore, when multimodal images experience large displacements of tens of pixels due to severe device jitter, a single deformable convolution will completely fail due to its limited receptive field. By using global affine transformation, rigid geometric distortions between images are corrected a priori, compressing spatial errors to the effective capture range of the fine alignment module. This ensures the convergence and robustness of subsequent fine alignment, resolving the contradiction between large-scale spatial mismatch and limited local receptive field.

[0020] 2. This multimodal image feature matching method, combining global affine transformation and local deformable convolution, introduces a modality-invariant feature extraction mechanism in the fine alignment module. Visible light and infrared images have different physical imaging mechanisms and contain modality-specific information; for example, visible light images contain rich color textures, while infrared images contain rich thermal radiation information. Directly calculating local offsets can be affected by noise from this modality-specific information. By using structural similarity priors and dual-path gradient fidelity loss as joint constraints, modality-invariant features that remove modality-specific information are extracted. This method abandons the approach of directly comparing the original features of visible light and infrared images, avoiding interference from inherent modality differences in network position determination. By extracting pure geometric contours for position comparison, the network can focus on solving spatial misalignments, significantly improving accuracy.

[0021] 3. This multimodal image feature matching method, combining global affine transformation and local deformable convolution, has a clear design logic. It balances detection accuracy with computational efficiency. The coarse alignment module is implemented using lightweight fully connected layers, resulting in low computational cost. The fine alignment module only performs local fine-tuning on features that have already undergone coarse alignment, avoiding the computational overhead of blindly increasing the number of deformable convolutional layers to expand the receptive field. This method significantly improves the accuracy and resistance to non-alignment interference in visible-infrared multimodal detection under complex perspectives such as those from UAVs. Attached Figure Description

[0022] Figure 1 This is a block diagram illustrating the overall coarse-to-fine feature matching and target detection architecture of the present invention.

[0023] Figure 2 This is a structural block diagram of the global affine transformation part of the present invention.

[0024] Figure 3 This is a structural block diagram of the locally deformable convolution portion of the present invention. Detailed Implementation

[0025] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention. Example 1

[0026] like Figures 1 to 3 As shown, a multimodal image feature matching method combining global affine transformation and local deformable convolution includes the following steps: Step 1: Construct a multimodal target detection network architecture. Input the paired raw visible light images and raw infrared images into the dual-stream backbone network for feature extraction to obtain the raw visible light feature map and the raw infrared feature map. Step 2: Use the global affine transformation module to globally control the large-scale rigid misalignment between images to obtain a coarsely aligned visible light feature map; Step 3: Construct a mode-invariant feature encoder. Input the coarsely aligned visible light feature map and the original infrared feature map into the mode-invariant feature encoder respectively. Extract mode-invariant features through the constraints of two loss functions to provide a matching reference for the downstream network without interference from mode-specific information. Step 4: Perform deformable convolution on the two modality-invariant features, combine it with the feature modulation mask, and use the deformable convolution kernel output by the two modality-invariant features to adaptively adjust and fine-tune the coarsely aligned visible light feature map at the pixel level to obtain the final finely aligned visible light feature map. Step 5: After completing all spatial matching work, perform feature fusion, input the fused features into the downstream target detection head, and output the final detection result.

[0027] like Figures 1 to 3 As shown, the multimodal target detection network architecture in step one includes a dual-stream backbone network, a coarse alignment module, a fine alignment module, and a detection head; The specific process of building a multimodal object detection network architecture is as follows: Pairs of visible light images to be processed... With infrared thermal imaging images The data are fed into a dual-stream backbone network, where convolutional neural networks are used for forward propagation and feature extraction to obtain the original visible light feature maps. and original infrared feature map At this point, due to camera parallax and motion jitter, the two feature maps are spatially misaligned. This differs from the traditional staged approach of aligning offline images before detection. It also limits the dual-stream independent feature extraction branches, and uses dedicated extraction networks for visible light and infrared modes to adapt to the differences in their imaging mechanisms. This results in higher quality basic feature extraction and provides reliable input for subsequent alignment.

[0028] like Figure 2 As shown, the global affine transformation module alignment in step two specifically includes the following steps: S1, using the original infrared feature map As a spatial reference, the original visible light feature map within the proposed box area will be used. Compared with the original infrared feature map Perform pixel-by-pixel subtraction to calculate the feature difference map; S2. Input the difference map into a multilayer fully connected network for regression to predict the global positional deviation, size deviation, and angular deviation, and combine them into an affine parameter matrix. S3. Resample the original visible light feature map based on the affine parameter matrix, and output a coarsely aligned visible light feature map through global translation and rotation. It uses feature difference map regression of affine parameters to directly calculate global offset in high-dimensional feature space. Compared with pixel-level difference, it has stronger resistance to illumination and thermal noise. At the same time, it regresses three types of global distortion parameters: translation, scaling and angle, which fully covers all rigid misalignments caused by dual-light device jitter, focal length difference and clock deviation. Example 2

[0029] like Figures 1 to 3 As shown, the modality-invariant feature encoder in step three internally uses a 5-layer densely connected block, where the output of each layer is concatenated with the outputs of all preceding layers before being input to the next layer; subsequently, it enters a fusion block consisting of 3 consecutive convolutional layers, ultimately outputting visible light invariant feature maps. Infrared Invariant Feature Map The five densely connected blocks reuse shallow features layer by layer, fully preserving cross-modal shared geometric information such as image edges and contours, and will not lose subtle target boundaries due to multiple convolutions.

[0030] like Figure 3 As shown, the modality-invariant feature encoder introduces the following two losses during the training phase: Structural similarity loss , , This loss forces the visible light invariant characteristic Infrared invariant features It exhibits high consistency in macroscopic spatial topology and contrast distribution, effectively filtering out modality-specific information such as color and temperature. Dual-path gradient fidelity loss , , in For a two-dimensional discrete spatial gradient operator, the loss constraint is that regardless of whether the features are visible light invariant or infrared invariant, the sharp physical abrupt edges of their respective original input images must be preserved. This mechanism avoids the feature mean smoothing phenomenon, provides high-precision physical boundary anchors for subsequent local alignment, and avoids the feature collapse shortcut where the network reduces all features to 0. The gradient calculation and loss backpropagation mentioned above are performed during the model training phase. During deployment, a forward lightweight feature encoder is executed to reduce the time overhead of this step. The gradient-fidelity loss constraint invariant feature retains the original image edge gradient, preventing the network from excessively smoothing features to satisfy the SSIM loss, avoiding the disappearance of small target contours, and ensuring sufficient anchor points for fine local alignment.

[0031] like Figure 3 As shown, the specific operation process of performing deformable convolution on the two modality-invariant features is as follows: the deformable convolution kernel used subsequently has K samples, the output feature modulation mask is processed by the Sigmoid activation function, and its value range is limited to (0,1). This mask represents the confidence weight of the network for the features extracted from these K sampling points. The confidence mask weighted sampling feature is introduced to distinguish the effective target pixels from the background noise pixels, suppress the erroneous offset learning in the background region, and reduce the probability of false matching. Example 3

[0032] like Figures 1 to 3 As shown, the specific process of calculating the finely aligned visible light feature map in step four is as follows: For any center pixel on the target feature map Establish a regular sampling grid R for the standard convolution kernel (the grid of the 3×3 convolution is R={(-1,-1),(-1,0),…,(1,1)}), and denote the k-th standard sampling coordinate offset in the grid as . ; Deformable convolution kernels traverse coarse-aligned visible light invariant feature maps At that time, its actual sampling position is the coordinate: To accurately obtain the feature value at this location ; Multiply the extracted feature values ​​by the corresponding feature modulation weights. Then, combined with the inherent learned weights of the deformable convolution kernel. We perform weighted summation to obtain the finely aligned target output feature values. Its discrete mathematical calculation formula is as follows: , After traversing the entire image and performing the above calculations, the network finally outputs a finely aligned visible light feature map. It integrates the inherent weights of convolution, sampling offset, and confidence mask, and the sampling results take into account semantic features, spatial coordinates, and pixel credibility. The local alignment accuracy is much higher than that of traditional fixed grid convolution.

[0033] like Figures 1 to 3 As shown, the feature fusion and calculation process in step five is specifically as follows: Perform element-by-element addition or channel stitching on the precisely aligned visible light feature map and the original infrared feature map; It is then fed into the subsequent classification and regression detection head to output the final target category and accurate bounding box localization results. Two compatible fusion strategies are provided, which can be flexibly selected according to the scenario. Channel splicing retains all information of both channels, while element-wise addition compresses channels and reduces the amount of computation.

[0034] The above is only used to illustrate the technical solution of the present invention and not to limit it. Any other modifications or equivalent substitutions made by those skilled in the art to the technical solution of the present invention, as long as they do not depart from the spirit and scope of the technical solution of the present invention, should be covered within the scope of the claims of the present invention.

Claims

1. A multimodal image feature matching method combining global affine transformation and local deformable convolution, characterized in that, Includes the following steps: Step 1: Construct a multimodal target detection network architecture. Input the paired raw visible light images and raw infrared images into the dual-stream backbone network for feature extraction to obtain the raw visible light feature map and the raw infrared feature map. Step 2: Use the global affine transformation module to globally control the large-scale rigid misalignment between images to obtain a coarsely aligned visible light feature map; Step 3: Construct a mode-invariant feature encoder. Input the coarsely aligned visible light feature map and the original infrared feature map into the mode-invariant feature encoder respectively. Extract mode-invariant features through the constraints of two loss functions to provide a matching reference for the downstream network without interference from mode-specific information. Step 4: Perform deformable convolution on the two modality-invariant features, combine it with the feature modulation mask, and use the deformable convolution kernel output by the two modality-invariant features to adaptively adjust and fine-tune the coarsely aligned visible light feature map at the pixel level to obtain the final finely aligned visible light feature map. Step 5: After completing all spatial matching work, perform feature fusion, input the fused features into the downstream target detection head, and output the final detection result.

2. The multimodal image feature matching method combining global affine transformation and local deformable convolution according to claim 1, characterized in that: The multimodal target detection network architecture in step one includes a dual-stream backbone network, a coarse alignment module, a fine alignment module, and a detection head; The specific process of building a multimodal object detection network architecture is as follows: Pairs of visible light images to be processed... With infrared thermal imaging images The data are fed into a dual-stream backbone network, where convolutional neural networks are used for forward propagation and feature extraction to obtain the original visible light feature maps. and original infrared feature map .

3. The multimodal image feature matching method combining global affine transformation and local deformable convolution according to claim 1, characterized in that: The global affine transformation module alignment in step two specifically includes the following steps: S1, using the original infrared feature map As a spatial reference, the original visible light feature map within the proposed box area will be used. Compared with the original infrared feature map Perform pixel-by-pixel subtraction to calculate the feature difference map; S2. Input the difference map into a multilayer fully connected network for regression to predict the global positional deviation, size deviation, and angular deviation, and combine them into an affine parameter matrix. S3. Resample the original visible light feature map based on the affine parameter matrix, and output a coarsely aligned visible light feature map through global translation and rotation. .

4. The multimodal image feature matching method combining global affine transformation and local deformable convolution according to claim 1, characterized in that: The modality-invariant encoder in step three internally uses a 5-layer densely connected block, where the output of each layer is concatenated with the outputs of all preceding layers before being input to the next layer; subsequently, it enters a fusion block consisting of 3 consecutive convolutional layers, ultimately outputting visible light invariant feature maps. Infrared Invariant Feature Map .

5. The multimodal image feature matching method combining global affine transformation and local deformable convolution according to claim 4, characterized in that: The modality-invariant feature encoder introduces the following two losses during the training phase: Structural similarity loss , This loss forces the visible light invariant characteristic Infrared invariant features It exhibits high consistency in macroscopic spatial topology and contrast distribution, effectively filtering out color and temperature modality-specific information; Dual-path gradient fidelity loss , in For a two-dimensional discrete spatial gradient operator, the loss constraint is that regardless of whether the features are visible light invariant or infrared invariant, the sharp physical abrupt edges of their respective original input images must be preserved. This mechanism avoids the feature mean smoothing phenomenon, provides high-precision physical boundary anchors for subsequent local alignment, and avoids the feature collapse shortcut where the network reduces all features to 0. Gradient calculation and loss backpropagation are performed during model training. During deployment, a forward lightweight feature encoder is executed to reduce the time overhead of this step.

6. The multimodal image feature matching method combining global affine transformation and local deformable convolution according to claim 1, characterized in that: The specific operation process of performing deformable convolution on the two modality-invariant features is as follows: the deformable convolution kernel used subsequently has K samples, the output feature modulation mask is processed by the Sigmoid activation function, and its value range is limited to (0,1). This mask represents the confidence weight of the network for the features extracted from these K sampling points.

7. The multimodal image feature matching method combining global affine transformation and local deformable convolution according to claim 1, characterized in that: The specific process for calculating the precisely aligned visible light feature map in step four is as follows: For any center pixel on the target feature map Establish a regular sampling grid R for the standard convolution kernel (the grid of the 3×3 convolution is R={(-1,-1),(-1,0),…,(1,1)}), and denote the k-th standard sampling coordinate offset in the grid as . ; Deformable convolution kernels traverse coarse-aligned visible light invariant feature maps At that time, its actual sampling position is the coordinate: To accurately obtain the feature value at this location ; Multiply the extracted feature values ​​by the corresponding feature modulation weights. Then, combined with the inherent learned weights of the deformable convolution kernel. We perform weighted summation to obtain the finely aligned target output feature values. Its discrete mathematical calculation formula is as follows: After traversing the entire image and performing the above calculations, the network finally outputs a finely aligned visible light feature map. .

8. The multimodal image feature matching method combining global affine transformation and local deformable convolution according to claim 1, characterized in that: The feature fusion and calculation process in step five is as follows: Perform element-by-element addition or channel stitching on the precisely aligned visible light feature map and the original infrared feature map; It is then fed into subsequent classification and regression detection heads, which output the final target category and accurate bounding box localization results.

Images