Multi-modal image real-time matching system and method based on lightweight convolutional network

Abstract

The invention discloses a multi-modal image real-time matching system based on a lightweight convolutional network. The multi-modal image real-time matching system comprises a feature extraction module, a rough matching module, a feature fusion and window extraction module and a fine matching module, the feature extraction module is connected with the rough matching module, the rough matching module is connected with the feature fusion and window extraction module, and the feature fusion and window extraction module is connected with the fine matching module; the method comprises the following steps: S1, carrying out feature extraction through a lightweight convolutional neural network capable of being re-parameterized to obtain feature maps of different scales; s2, performing coarse matching on the coarse-level feature map to obtain coarse matching points; s3, fusing the multi-scale feature map, mapping the position of the rough matching point to the fused feature map to serve as a key point, and taking a window around the key point; s4, performing fine matching based on the extracted window; according to the invention, the problems of matching precision and speed in a cross-modal scene are solved.

Description

Technical Field

[0001] This invention belongs to the field of real-time image matching between different modalities, specifically relating to a real-time multimodal image matching system and method based on lightweight convolutional networks. Background Technology

[0002] Multimodal image matching technology, as a core component of remote sensing data processing, has significant application value in fields such as disaster monitoring, UAV autonomous navigation, and satellite mapping. This technology aims to establish precise correspondences between different imaging modalities (such as visible light, infrared, and synthetic aperture radar) to achieve spatial alignment and fusion of multi-source information. Existing technical solutions mainly fall into two categories: traditional manual feature methods and deep learning-driven methods, both of which have significant limitations.

[0003] Traditional feature matching methods (represented by SIFT and SURF) rely on manually designed local feature descriptors. These methods detect salient structures such as corners and edges, construct descriptive vectors using statistical features such as gradient histograms, and finally establish matching relationships through similarity metrics (such as Euclidean distance). In scenarios with the same imaging modality (e.g., visible light vs. visible light matching), these methods are somewhat practical due to their rotation and scale invariance. However, when faced with cross-modal tasks (e.g., visible light vs. infrared, visible light vs. SAR matching), their fundamental flaws become glaringly apparent: the physical mechanisms of different imaging modalities are fundamentally different—visible light images reflect the surface reflectance spectrum characteristics, infrared images characterize thermal radiation distribution, and SAR images rely on microwave scattering characteristics—causing the low-level visual features extracted by traditional methods to completely fail in the cross-modal feature space. This feature space mismatch is particularly severe in weakly textured areas (e.g., deserts, water bodies) or scenarios with nonlinear radiation differences, not only drastically reducing the number of matching points but also leading to numerous mismatches, or even the complete inability to establish effective correspondences. Furthermore, traditional methods employ a serial process of "detection-description-matching," and their computational efficiency increases exponentially with image resolution, making it difficult to meet the stringent requirements of real-time systems for millisecond-level response.

[0004] Deep learning methods attempt to overcome traditional limitations through data-driven approaches, and can be divided into two branches based on whether they rely on explicit keypoint detection. The first category is detector-dependent methods (such as the SuperPoint and SuperGlue combined framework), which use end-to-end convolutional neural networks to jointly optimize feature point detection and descriptor generation, and introduce graph neural networks to model the spatial topological constraints between feature points. While these methods show advantages in intramodal matching, their inherent limitations restrict cross-modal adaptability: the feature detection module is extremely sensitive to differences in imaging physics, leading to a significant reduction in the number of stable feature points in cross-modal tasks such as infrared-visible light; and the computational redundancy generated by the multi-stage processing flow (feature extraction, descriptor generation, graph matching) causes latency on embedded devices to far exceed the real-time processing threshold. The second category is detectorless methods (such as LoFTR and ASpanFormer), which completely abandon the explicit keypoint detection stage and directly establish pixel-level dense matching through the Transformer architecture. This approach utilizes self-attention mechanisms to capture long-range dependencies within the image, combined with cross-attention to achieve feature interaction between images, theoretically exhibiting stronger robustness in weakly textured regions. However, the quadratic computational complexity introduced by the global attention mechanism creates an unbearable computational burden when processing high-resolution remote sensing imagery. More importantly, when the feature space distribution differs significantly (such as between visible light and SAR imagery), the standard attention mechanism struggles to adaptively model the nonlinear mapping relationships between modes, leading to feature interaction failure and a sharp decline in matching accuracy.

[0005] In summary, the existing technology system faces three systemic bottlenecks: 1. Failure of cross-modal feature representation: The low-level features of traditional methods and the feature extraction modules of deep learning methods cannot overcome the feature space mismatch caused by differences in imaging physical mechanisms. 2. Insufficient real-time processing capabilities: The computational efficiency bottleneck of traditional serial architectures and the complex models (graph networks / global attention) in deep learning make it difficult for embedded deployments to meet real-time requirements. 3. Lack of resource adaptability: The high memory consumption and high computational intensity of deep learning models lack dynamic optimization mechanisms, restricting their application in low-power scenarios such as airborne equipment.

[0006] These shortcomings have led to serious consequences in critical scenarios such as disaster emergency response: for example, forest fire monitoring requires real-time fusion of infrared thermal images and visible light maps, but cross-modal matching failures cause delays in thermal source localization; satellite on-orbit services require precise matching of optical and SAR imagery for collision avoidance operations, but computational delays create safety risks. Therefore, developing new matching technologies that combine cross-modal adaptability, low latency, and embedded compatibility has become an urgent need to overcome industry bottlenecks. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention proposes a real-time multimodal image matching system and method based on lightweight convolutional networks, which solves the problems of matching accuracy and speed in cross-modal scenarios.

[0008] The technical solution of the present invention is as follows:

[0009] A real-time multimodal image matching system based on a lightweight convolutional network includes: a feature extraction module, a coarse matching module, a feature fusion and window extraction module, and a fine matching module;

[0010] The feature extraction module is connected to the coarse matching module, the coarse matching module is connected to the feature fusion and window extraction module, and the feature fusion and window extraction module is connected to the fine matching module.

[0011] Preferably, a real-time multimodal image matching method based on a lightweight convolutional network includes the following steps:

[0012] Step S1: Feature extraction is performed using a reparameterizable lightweight convolutional neural network. Each feature extraction module achieves feature extraction by stacking different numbers of improved convolutional modules, resulting in feature maps of different scales.

[0013] Step S2: On the coarse feature map, establish a preliminary spatial correspondence between the two images to be matched to obtain coarse matching points;

[0014] Step S3: Fuse the feature information of different scales of the image to form a fused feature map, map the coarse matching points to the fused feature map to generate key points, and extract local regions around the key points;

[0015] Step S4: Perform similarity calculation and sub-pixel level coordinate regression on the local window features of the coarse matching points obtained by the feature fusion and window extraction module, thereby realizing the transition from coarse matching to fine matching.

[0016] Preferably, step S1 includes the following sub-steps:

[0017] Sub-step S11: Preprocess the data;

[0018] Sub-step S12: Combine the preprocessed data and perform feature extraction based on the lightweight neural network.

[0019] Preferably, step S2 includes the following sub-steps:

[0020] Sub-step S21: Reduce and normalize the data, and calculate the global similarity matrix;

[0021] Sub-step S22: Obtain the confidence matrix based on bidirectional softmax;

[0022] Sub-step S23: Obtain coarse matching results by filtering using the nearest neighbor method.

[0023] Preferably, step S3 includes the following sub-steps:

[0024] Sub-step S31: Perform feature fusion on the coarse matching result through feature dimensionality reduction and bilinear interpolation to obtain the fused features;

[0025] Sub-step S32: Combine the fused features to extract a fine matching window around the coarse matching point.

[0026] Preferably, step S4 includes the following sub-steps:

[0027] Sub-step S41: Calculate the offset position heatmap within the fine matching window;

[0028] Sub-step S42: Combine the heatmap and obtain the fine matching result through subpixel-level regression.

[0029] Preferably, the normalization process of the feature vector sequence in sub-step S21, which eliminates the difference in the magnitude of the feature vectors by scaling the channel dimension, is as follows:

[0030]

[0031] Where Fc is the feature vector after flattening the image space dimensions. It is the normalized feature vector, and d is the number of feature channels;

[0032] The global similarity matrix is:

[0033]

[0034] Among them, S i，j It is the global similarity matrix. These are the normalized feature vectors of a point on the two flattened images, respectively, and τ is the temperature coefficient.

[0035] Preferably, the formula for the confidence matrix in sub-step S22 is:

[0036] F c (i, j) = softmax(S(i, ·)) j ·softmax(S(·,j)) i

[0037] Among them, F c (i, j) represents the confidence matrix, and softmax(S(i, ·)) j This represents the matrix after performing a softmax operation on the global similarity matrix in the j-direction, where softmax(S(·,j)) iThis represents the matrix obtained by performing a Softmax operation on the global similarity matrix in the i-th direction.

[0038] The beneficial effects of the multimodal image real-time matching system and method based on lightweight convolutional networks of this invention are as follows:

[0039] 1. Significantly Improved Matching Accuracy: The algorithm proposed in this invention demonstrates significant comprehensive performance advantages in real-time multimodal image matching tasks. Regarding matching accuracy, the algorithm achieves a comprehensive breakthrough across six cross-modal scenarios, including optical-infrared and optical-SAR, through a physically-aware feature fusion mechanism: the number of correctly matched points (NCM) is nearly twice that of suboptimal methods, maintaining dense matching capability even in weakly textured areas such as deserts and water bodies where traditional methods completely fail; the matching success rate (SR) reaches 98.3% across various modalities and extreme scenarios, very close to 100%, significantly better than the average level of 60% for mainstream deep learning solutions; the geometric accuracy (RMSE) is compressed to 3.062 pixels through sub-pixel-level positioning optimization, exhibiting the smallest error among all compared algorithms, meeting the sub-meter level accuracy requirements of satellite mapping. Typical cases show that in optical-SAR matching tasks, the traditional SIFT method fails completely due to feature mismatch, and the deep learning solution LoFTR generates only 332.5 matching points (success rate of 60%), while the algorithm of this invention achieves 844.3 high-precision matching points (success rate of 100%), with the error controlled within 2.955 pixels (ground scale < 1.5 meters).

[0040] 2. Enhanced real-time processing capabilities: The proposed algorithm demonstrates superior processing speed, achieving a single-frame processing speed of 30.51 milliseconds on a consumer-grade GPU platform (NVIDIA GeForce RTX 4090, 24GB GDDR6X memory), which is over 30% faster than mainstream depth solutions (such as LoFTR's 43.44 milliseconds), completely resolving the efficiency bottleneck of traditional methods in cross-modal tasks. The proposed algorithm's reparameterizable lightweight convolutional neural network and feature fusion module significantly reduce computational load while maintaining high accuracy, thereby significantly improving processing efficiency.

[0041] On embedded platform environments such as NVIDIA Jetson AGX Xavier, GPU 22TOPS (INT8), 2.3TFLOPS (FP32), 137.6GB / s (256-bit LPDDR4x), the proposed algorithm can achieve sub-second processing time per frame, indicating that the algorithm has the ability to be deployed on lightweight devices such as mobile and airborne devices.

[0042] 3. Robustness in complex environments and cross-modal scenarios: The proposed algorithm enhances its adaptability to complex visual scenes (such as illumination variations, scale variations, occlusion, etc.) and cross-modal scenarios through multi-source data training and multi-scale feature fusion mechanisms. For example, in cross-modal scenes with high noise and low texture, the proposed algorithm's feature extraction and feature fusion modules enable it to capture global information while preserving detailed information, thus achieving high-precision matching even under these conditions.

[0043] 4. Energy consumption and computational efficiency optimization: The proposed algorithm significantly reduces system energy consumption and improves computational efficiency through coordinated optimization of model structure and computational process. In the model inference stage, reparameterization design integrates the multi-branch network of the training stage into a highly efficient single-path structure, greatly reducing computational redundancy. At the task execution level, the two-stage strategy quickly identifies key regions through global screening at the coarse matching level, while the fine matching stage focuses only on the fine calculation of local windows, avoiding the resource waste caused by traditional global matching. Experimental results show that this scheme significantly reduces power consumption and optimizes memory usage compared to mainstream deep learning methods, enabling stable operation of UAV-borne equipment in cross-modal real-time processing modes, providing reliable technical support for long-term field operations such as disaster monitoring.

[0044] 5. Task Adaptability and Universality: The proposed algorithm was trained on various cross-modal data, maintaining high matching accuracy under different modalities, radiation distortion, and viewpoint variations. Experimental data show that the proposed algorithm achieves a near 100% success rate even on complex cross-modal video data, demonstrating its strong task adaptability. Furthermore, the proposed algorithm can be applied to various complex scenes and visual matching tasks, exhibiting strong universality.

[0045] In summary, the proposed algorithm's high precision, low energy consumption, real-time performance, and adaptability to complex scenes give it significant technical advantages in real-time multimodal image matching tasks, improving the stability and accuracy of the task and providing solid technical support for subsequent related tasks. Attached Figure Description

[0046] To more clearly illustrate the purpose, design concept, and innovation of the multimodal image real-time matching system and method based on lightweight convolutional networks proposed in this invention, the invention will be described in detail below with reference to the accompanying drawings and tables.

[0047] Figure 1 This is a flowchart of the algorithm of the present invention;

[0048] Figure 2 This is a diagram of the algorithm network structure of the present invention. Detailed Implementation

[0049] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0050] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0051] A real-time multimodal image matching system based on a lightweight convolutional network includes: a feature extraction module, a coarse matching module, a feature fusion and window extraction module, and a fine matching module;

[0052] The feature extraction module is connected to the coarse matching module, the coarse matching module is connected to the feature fusion and window extraction module, and the feature fusion and window extraction module is connected to the fine matching module.

[0053] A real-time multimodal image matching method based on a lightweight convolutional network includes the following steps:

[0054] Step S1: Feature extraction is performed using a reparameterizable lightweight convolutional neural network. Each feature extraction module achieves feature extraction by stacking different numbers of improved convolutional modules, resulting in feature maps of different scales.

[0055] Step S2: On the coarse feature map, establish a preliminary spatial correspondence between the two images to be matched to obtain coarse matching points;

[0056] Step S3: Fuse the feature information of different scales of the image to form a fused feature map, map the coarse matching points to the fused feature map to generate key points, and extract local regions around the key points;

[0057] Step S4: Perform similarity calculation and sub-pixel level coordinate regression on the local window features of the coarse matching points obtained by the feature fusion and window extraction module, thereby realizing the transition from coarse matching to fine matching.

[0058] Step S1 of this implementation plan includes the following sub-steps:

[0059] Sub-step S11: Preprocess the data;

[0060] Sub-step S12: Combine the preprocessed data and perform feature extraction based on the lightweight neural network.

[0061] Step S2 of this implementation plan includes the following sub-steps:

[0062] Sub-step S21: Reduce and normalize the data, and calculate the global similarity matrix;

[0063] Sub-step S22: Obtain the confidence matrix based on bidirectional softmax;

[0064] Sub-step S23: Obtain coarse matching results by filtering using the nearest neighbor method.

[0065] Step S3 of this implementation plan includes the following sub-steps:

[0066] Sub-step S31: Perform feature fusion on the coarse matching result through feature dimensionality reduction and bilinear interpolation to obtain the fused features;

[0067] Sub-step S32: Combine the fused features to extract a fine matching window around the coarse matching point.

[0068] Step S4 of this implementation plan includes the following sub-steps:

[0069] Sub-step S41: Calculate the offset position heatmap within the fine matching window;

[0070] Sub-step S42: Combine the heatmap and obtain the fine matching result through subpixel-level regression.

[0071] In sub-step S21 of this implementation scheme, the feature vector sequence is normalized. The formula for eliminating the difference in feature vector magnitudes through scaling the channel dimension is as follows:

[0072]

[0073] Where Fc is the feature vector after flattening the image space dimensions. It is the normalized feature vector, and d is the number of feature channels;

[0074] The global similarity matrix is:

[0075]

[0076] Among them, S i，j It is the global similarity matrix. These are the normalized feature vectors of a point on the two flattened images, respectively, and τ is the temperature coefficient.

[0077] The formula for the confidence matrix in sub-step S22 of this implementation scheme is as follows:

[0078] F c (i, j) = softmax(S(i, ·)) j ·softmax(S(·,j)) i

[0079] Among them, F c (i, j) represents the confidence matrix, and softmax(S(i, ·)) j This represents the matrix after performing a softmax operation on the global similarity matrix in the j-direction, where softmax(S(·,j)) i This represents the matrix obtained by performing a Softmax operation on the global similarity matrix in the i-th direction.

[0080] 1. When implementing this implementation plan,

[0081] This invention effectively solves the matching accuracy and speed problems in cross-modal scenarios by collaboratively optimizing feature extraction methods and calculation processes.

[0082] The core technology adopts a hierarchical processing architecture. First, it uses an improved reparameterized convolutional network to extract multi-scale visual features. During the training phase, it enhances the feature representation capability through a multi-branch structure. During the inference phase, it is fused into an efficient single-path model to compress computational redundancy. Then, it constructs a coarse-to-fine two-stage matching engine. It narrows down the potential corresponding region by performing global fast matching and localization at the low-resolution level, and then focuses on local windows at the high-resolution level to achieve sub-pixel-level accurate localization, which greatly reduces the computational complexity.

[0083] The feature fusion module integrates shallow texture features with deep semantic information through a multi-scale pyramid structure. It captures cross-modal common structures (such as building outlines and road topology) using 1 / 8-scale feature maps and combines them with edge details preserved by 1 / 2-scale features to form a robust feature representation. The dynamic computation architecture is based on an improved RepVGG network. During training, it uses multi-branch convolutions to mine feature correlations, and during inference, it generates equivalent single-path models through mathematical inference transformations, thereby improving computational efficiency.

[0084] This solution achieves millisecond-level response on consumer-grade GPU platforms and sub-second latency on embedded platforms through hierarchical computational load optimization, while maintaining sub-pixel-level geometric accuracy, providing a practical solution for real-time multimodal image matching tasks.

[0085] This invention proposes a real-time multimodal image matching technique based on lightweight convolutional networks, focusing on solving the problem of real-time image matching between different modalities. The technical solution mainly consists of four modules: a feature extraction module, a coarse matching module, a feature fusion and window extraction module, and a fine matching module.

[0086] 2. This embodiment is further implemented as follows:

[0087] 2.1 Feature Extraction Module

[0088] The proposed algorithm extracts features using a reparameterizable lightweight convolutional neural network. The main body of the network consists of four feature extraction modules, each layer employing a stacked number of improved convolutional modules to achieve feature extraction. Finally, feature maps of different scales are extracted for subsequent operations.

[0089] The network also features reparameterization: after training, reparameterization merges the weights of multiple convolutional paths between layers into an equivalent single convolutional path. This parameter fusion mechanism significantly simplifies computation during the testing phase, greatly reducing matching time and enabling the algorithm to run efficiently on platforms with limited computing power, such as embedded devices.

[0090] 2.2. Coarse Matching Module

[0091] The coarse matching module can establish a preliminary spatial correspondence between two images to be matched: the module first flattens the input coarse feature map, normalizes it and obtains the similarity matrix, then obtains the confidence matrix through bidirectional Softmax operation, and obtains the coarse matching result after mutual nearest neighbor filtering on the confidence matrix.

[0092] 2.3 Feature Fusion and Window Extraction Module

[0093] This module fuses feature information from different scales of the image to form a fused feature map, and extracts local regions around key points in the fused feature map to provide richer detailed features for subsequent fine matching. During the feature fusion stage, the module employs a progressive upsampling strategy from deep to shallow. It fuses feature maps of different scales and dimensions through dimensionality compression and bilinear interpolation to obtain a fused feature map that simultaneously possesses deep overall semantic information and shallow detailed texture information.

[0094] In the local window extraction stage, based on the fused high-resolution feature map, the algorithm uses a sliding window mechanism to locate a window around the coarse matching point. Each window covers a fixed-size area (e.g., 5×5) centered on the coordinates of the coarse matching point. The final output window feature pairs are strictly aligned and contain refined descriptive information of the matching point's neighborhood, providing high-resolution feature support for sub-pixel-level matching.

[0095] 2.4. Fine Matching Module

[0096] The fine matching module performs similarity calculations and sub-pixel level coordinate regression on the local window features of the coarse matching points obtained from the previous module, thereby achieving the transition from coarse matching to fine matching: The module first performs dense similarity calculations on the features of the source window and the target window to generate a heatmap to show the probability distribution of matching, then calculates the sub-pixel level offset through the expectation operation of differentiable space, and finally maps the matching coordinates with the offset to the original image space to output the precise matching point pair.

[0097] 3. This embodiment is further implemented as follows:

[0098] The algorithm proposed in this invention comprises four modules: a feature extraction module, a coarse matching module, a feature fusion and window extraction module, and a fine matching module. The specific implementation process is described in detail below:

[0099] 3.1 Feature Extraction Module

[0100] The proposed algorithm extracts features using a reparameterizable lightweight convolutional neural network. The network consists of four feature extraction modules, each layer employing a stacked number of improved convolutional modules to achieve feature extraction. The input image first passes through a 3×3 convolutional layer (stride 2) at layer 0, reducing the resolution to half the original image and expanding the feature channels to 128 dimensions, forming the initial feature layer. The first layer of the network consists of two cascaded improved convolutional modules, each containing parallel 3×3 convolutions (main path), 1×1 convolutions (auxiliary path), and identity mappings for channel matching. The output maintains half the resolution and 128 feature channels, but further enhances the feature depth. The second layer further reduces the resolution to one-quarter through four improved modules, expanding the feature channels to 192 dimensions. Finally, the third layer consists of 14 stacked modules, outputting a low-resolution feature map at one-eighth resolution, expanding the feature channels to 256 dimensions. Each layer is connected in parallel through a multi-branch structure to achieve multi-scale feature fusion.

[0101] The reparameterization process of this network transforms multi-branch paths into single-path paths through mathematical transformations. After training, the network merges the weights of multiple convolutional paths between layers into an equivalent single convolutional path and bias term through reparameterization. This process uses the principle of linear superposition, padding a 1×1 convolutional kernel with zeros to a 3×3 size and adding it to the main convolutional kernel, while fusing the diagonal matrix parameters corresponding to the identity mapping, ultimately generating a lightweight single-path model that is mathematically equivalent to the original multi-path structure. This parameter fusion mechanism significantly simplifies the computation during the testing phase, thereby greatly reducing the matching time and enabling the algorithm to run efficiently even on platforms with limited computing power, such as embedded devices.

[0102] 3.2 Coarse Matching Module

[0103] The coarse matching module establishes a preliminary spatial correspondence between two images to be matched. After inputting the low-resolution (typically 1 / 8) feature map extracted by the feature extraction network, this module first flattens the feature map spatially, converting the original two-dimensional feature map into a one-dimensional sequence. Then, it normalizes this feature vector sequence, eliminating differences in feature vector magnitudes through channel-level scaling.

[0104]

[0105] Where Fc is the feature vector after flattening the image space dimensions. These are the normalized eigenvectors;

[0106] Here, d represents the number of feature channels. This operation makes the feature matching process focus more on directional consistency rather than numerical magnitude, enhancing the algorithm's robustness to illumination changes and noise interference. The standardized feature sequence retains its original structure, and then the global similarity matrix is ​​calculated:

[0107]

[0108] S i,j It is the global similarity matrix. These are the normalized feature vectors of a point on the two flattened images, respectively. τ is the temperature coefficient. This formula represents the result of taking the inner product of the feature vector of point i on image 0 and the feature vector of point j on image 1, and then dividing by τ, to obtain the point at position (i, j) on the global similarity matrix S.

[0109] The temperature coefficient τ acts as a hyperparameter to control the sharpening of the similarity distribution. Lower temperature values ​​enhance the significance of differences in high-confidence regions, while higher temperature values ​​mitigate distribution differences to retain more potential matches.

[0110] Next, confidence scores are generated using a two-way probability constraint strategy. A joint confidence matrix is ​​generated by independently performing Softmax operations on the row and column dimensions and then taking the element-wise product.

[0111] F c (i,j)=softmax(S(i,·)) j ·softmax(S(·,j)) i

[0112] F c (i,j) represents the confidence matrix, and softmax(S(i,·)) j This represents the matrix after performing a softmax operation on the global similarity matrix in the j-direction, where softmax(S(·,j))i This represents the matrix obtained by performing a Softmax operation on the global similarity matrix in the i-th direction.

[0113] After obtaining the confidence matrix, the module uses the mutual nearest neighbor method to filter reliable matching points. This bidirectional constraint mechanism can be implemented through tensor operations. First, the global maximum value is located in the row and column dimensions of the similarity matrix, generating a Boolean mask indicating the maximum response. Then, the bidirectional conditions are fused through a logical AND operation, retaining only point pairs that simultaneously satisfy the maximum row and column constraints. This method can effectively eliminate erroneous matches with high unidirectional similarity but low reverse similarity.

[0114] Finally, after obtaining the reliable matching points selected by the mutual nearest neighbor method, coordinate mapping is performed using a scaling factor to obtain the position of the corresponding reliable matching points in the high-resolution feature map (usually 1 / 2). Finally, the coordinates of the matching points obtained by coarse matching in the high-resolution feature map and the confidence weights between the corresponding matching point pairs are output, laying a good data foundation for subsequent fine matching.

[0115] 3.3 Feature Fusion and Window Extraction Module

[0116] This module fuses feature information from different scales of the image to form a fused feature map, and extracts local regions around key points in the fused feature map to provide richer detailed features for subsequent fine matching. The module first receives the coordinates of the matching points obtained from coarse matching in the high-resolution feature map, along with the confidence weights between the corresponding matching point pairs. Then, it receives intermediate features from different scales from the feature extraction module and fuses them to form a fused feature map. During the input feature processing stage, the module re-dimensionally restructures the features output from coarse matching, restoring the flattened feature sequence to a two-dimensional spatial structure, facilitating subsequent convolutional operations to capture local spatial relationships. For input image pairs of the same size, the module concatenates the features of the two images along the batch dimension to achieve parameter sharing and parallel processing; for inputs of different sizes, independent dual-branch processing is used.

[0117] In the feature fusion stage, the module employs a progressive upsampling strategy from deep to shallow. Low-resolution features (1 / 8 scale) are first compressed in channel dimension using a 1×1 convolution, bringing their dimension to the same level as the 1 / 4 scale feature map extracted by the feature extraction module. Then, bilinear interpolation is used to upsample the feature map to 1 / 4 scale, facilitating its addition with the 1 / 4 scale feature map extracted by the feature extraction module. The added features are then fused using a 3×3 convolution and batch normalization, and the LeakyReLU activation function is used to enhance the non-linear expressive power of the features. After fusion, the fused feature map is again compressed in channel dimension using a 1×1 convolution, bringing its dimension to the same level as the 1 / 2 scale feature map extracted by the feature extraction module. The fusion result is then further upsampled to 1 / 2 scale and compared with the 1 / 2 scale feature map. Figure 2 The features are added and fused one by one, and the final output is a fused and enhanced feature map that combines information from both the deep and shallow layers of the neural network.

[0118] In the local window extraction stage, based on the fused high-resolution feature map, the algorithm uses a sliding window mechanism to locate a window around the coarse matching point. The module generates multiple local windows on the feature map through an unfolding operation (F.unfold). Each window covers a fixed-size area (e.g., 5×5) centered on the coordinates of the coarse matching point. After obtaining the windows, the dimensions of the windows are reorganized, separating the channels, spatial positions, and window indices to form a well-structured three-dimensional tensor, which facilitates subsequent similarity calculation.

[0119] In the matching point localization stage, the target window is precisely filtered based on the index information output by the coarse matching. The b_ids generated by the coarse matching module represents the batch index, while i_ids and j_ids represent the position indices of matching points on the feature maps of the source and target images, respectively. Through tensor slicing operations, the module quickly retrieves the source and target local windows corresponding to each pair of matching points from the full-image window sequence. i_ids and j_ids, as one-dimensional position indices, can have their two-dimensional coordinates on the feature map recovered through integer division and modulo operations, ensuring the spatial accuracy of window extraction. The final output window feature pairs are strictly aligned and contain refined descriptions of the matching point neighborhood, providing high-resolution feature support for sub-pixel-level matching.

[0120] 3.4 Fine Matching Module

[0121] The fine matching module performs similarity calculations and sub-pixel-level coordinate regression on the local window features of the coarse matching points obtained from the previous module, thus transitioning from coarse to fine matching. This module receives local window feature pairs from the feature fusion stage; each window covers a fixed area centered on the coarse matching point. The module first performs dense similarity calculations on the features of the source and target windows, generating a heatmap to display the probability distribution of matching. Then, it calculates the sub-pixel-level offset through differentiable space expectation operations. Finally, it maps the matching coordinates with the offset to the original image space, outputting precisely matched point pairs.

[0122] In the feature processing stage, the module extracts the feature vector of the center position of the source window as the query benchmark. It then calculates the similarity matrix between this central feature and all positional features within the target window, generating an initial response map. Next, similarity calculations are performed on the initial response map, using the reciprocal of the square root of the channel dimension as a temperature coefficient to scale the results and balance the numerical distribution range in the high-dimensional feature space. The scaled response map is then transformed into a probabilistic heatmap through Softmax normalization, where the value at each position in the target window represents the matching confidence between that position and the center position of the source window.

[0123] The transformation from heatmap to sub-pixel offset is then achieved through spatial expectation calculation. First, a normalized grid coordinate system is constructed within the target window region. The probability distribution of the heatmap is used as weights, and the weighted average position is calculated as the sub-pixel offset of the matching point. Simultaneously, the algorithm calculates the variance of the heatmap as an estimate of the matching uncertainty. A smaller variance indicates more accurate matching and positioning; this metric can be used in subsequent mismatch elimination steps.

[0124] The coordinate mapping stage converts sub-pixel offsets to the scale of the original image. The module calculates a scale factor based on the feature map downsampling rate, scales the normalized offsets to the actual pixel distance, and adds it to the coarse matching coordinates to obtain the final matching point position. For image pairs with scale differences, the algorithm dynamically adjusts the target image's coordinate mapping using pre-stored scale transformation parameters to ensure geometric consistency across scales. The output includes refined matching point pairs between the source and target images, as well as confidence and uncertainty estimates for each match, providing directly usable correspondence data for subsequent applications.

[0125] 4. Key points of this invention:

[0126] 4.1 Lightweight Convolutional Neural Network Feature Extraction Module with Reparameterizable Configuration This module employs an improved convolutional neural network architecture to achieve efficient feature extraction from multimodal images. Its core lies in introducing a dynamic structure reparameterization mechanism: during model training, multiple convolutional paths (including 3×3 convolutions, 1×1 convolutions, and identity mappings) are arranged in parallel to fully explore the common feature representations of different modalities; during inference deployment, the multi-branch structure is fused into a single 3×3 convolutional layer through mathematical equivalence transformation, significantly reducing computational complexity. This "multi-branch training - single-path inference" design pattern reduces computation while preserving feature discriminative power, significantly improving real-time performance. The module constructs a multi-level feature pyramid (1 / 2, 1 / 4, 1 / 8 scale), effectively extracting features from images at various levels, providing a foundation for subsequent matching that combines semantic abstraction and textural detail.

[0127] 4.2 Feature Fusion Module

[0128] This module fuses features at different scales and levels, achieving efficient utilization of cross-level feature information. Its core lies in a progressive feature fusion mechanism from deep to shallow: first, deep semantic features at the 1 / 8 scale are passed to the 1 / 4 level through channel compression and upsampling operations, and fused with feature maps of the same scale; then, the fused result is upsampled a second time to the 1 / 2 scale and fused again with the original high-resolution features, forming an enhanced feature map that combines global structure awareness with local detail preservation. Through multi-layer convolution and activation functions, feature interaction enhancement is achieved, effectively bridging the differences in semantic expression between different modalities of imagery and significantly improving the matching robustness of weakly textured regions.

[0129] 4.3 Coarse-fine two-stage matching process

[0130] This invention employs a two-stage matching process from coarse to fine:

[0131] In the coarse matching stage, bidirectional joint confidence screening is implemented at the 1 / 8 low-resolution feature layer: first, the flattened feature sequence is normalized to eliminate interference from intermodal radiation differences; then, a joint confidence matrix is ​​generated through row and column-independent similarity calculations, and high-reliability matching points are selected using the nearest neighbor principle. This stage compresses the computational region to 1 / 64 of the original image through a downsampling strategy, significantly reducing global search overhead. The fine matching stage focuses on sub-pixel optimization in the local neighborhood: based on the coordinates of the coarse matching points, a local window is extracted from the 1 / 2 high-resolution feature map; a heatmap probability distribution is generated through feature interactions within the window, and a spatially weighted average algorithm is used to calculate sub-pixel-level offsets, ultimately outputting high-precision matching coordinates. This design concentrates computational resources on key regions, maintaining sub-pixel accuracy while avoiding the burden of global computation.

[0132] This matching process achieves a balance between accuracy and efficiency through the synergy of rapid global filtering and fine-grained local positioning.

Claims

1. A real-time multimodal image matching system based on a lightweight convolutional network, characterized in that, include: The module includes a feature extraction module, a coarse matching module, a feature fusion and window extraction module, and a fine matching module. The feature extraction module is connected to the coarse matching module, the coarse matching module is connected to the feature fusion and window extraction module, and the feature fusion and window extraction module is connected to the fine matching module.

2. A real-time multimodal image matching method based on a lightweight convolutional network, characterized in that, Includes the following steps: Step S1: Feature extraction is performed using a reparameterizable lightweight convolutional neural network. Each feature extraction module achieves feature extraction by stacking different numbers of improved convolutional modules, resulting in feature maps of different scales. Step S2: On the coarse feature map, establish a preliminary spatial correspondence between the two images to be matched to obtain coarse matching points; Step S3: Fuse the feature information of different scales of the image to form a fused feature map, map the coarse matching points to the fused feature map to generate key points, and extract local regions around the key points; Step S4: Perform similarity calculation and sub-pixel level coordinate regression on the local window features of the coarse matching points obtained by the feature fusion and window extraction module, thereby realizing the transition from coarse matching to fine matching.

3. The multimodal image real-time matching method based on lightweight convolutional networks according to claim 2, characterized in that, Step S1 includes the following sub-steps: Sub-step S11: Preprocess the data; Sub-step S12: Combine the preprocessed data and perform feature extraction based on the lightweight neural network.

4. The multimodal image real-time matching method based on lightweight convolutional networks according to claim 2, characterized in that, Step S2 includes the following sub-steps: Sub-step S21: Reduce and normalize the data, and calculate the global similarity matrix; Sub-step S22: Obtain the confidence matrix based on bidirectional softmax; Sub-step S23: Obtain coarse matching results by filtering using the nearest neighbor method.

5. The multimodal image real-time matching method based on lightweight convolutional networks according to claim 2, characterized in that, Step S3 includes the following sub-steps: Sub-step S31: Perform feature fusion on the coarse matching result through feature dimensionality reduction and bilinear interpolation to obtain the fused features; Sub-step S32: Combine the fused features to extract a fine matching window around the coarse matching point.

6. The multimodal image real-time matching method based on lightweight convolutional networks according to claim 2, characterized in that, Step S4 includes the following sub-steps: Sub-step S41: Calculate the offset position heatmap within the fine matching window; Sub-step S42: Combine the heatmap and obtain the fine matching result through subpixel-level regression.

7. The multimodal image real-time matching method based on a lightweight convolutional network according to claim 4, characterized in that, The formula for normalizing the feature vector sequence in sub-step S21, which eliminates the difference in feature vector magnitudes through scaling the channel dimension, is as follows: Where Fc is the feature vector after flattening the image space dimensions. It is the normalized feature vector, and d is the number of feature channels; The global similarity matrix is: Among them, S i，j It is the global similarity matrix. These are the normalized feature vectors of a point on the two flattened images, respectively, and τ is the temperature coefficient.

8. The multimodal image real-time matching method based on lightweight convolutional networks according to claim 4, characterized in that, The formula for the confidence matrix in sub-step S22 is: F c (i,j)=softmax(S(i,·)) j ·softmax(S(·,j)) i Among them, F c (i, j) represents the confidence matrix, and softmax(S(i, ·)) j This represents the matrix after performing a softmax operation on the global similarity matrix in the j-direction, where softmax(S(i,·)) j This represents the matrix obtained by performing a Softmax operation on the global similarity matrix in the i-th direction.

Images