A Multimodal Image Matching Method and System Based on Sub-Pixel Deviation Estimation

Abstract

The present invention discloses a multi-modal image matching method and system based on sub-pixel deviation estimation. The method of the present invention includes inputting two input images of different modalities into a pre-trained multi-modal image matching network to obtain the offset between the two input images, including: respectively extracting feature tensors of multiple scales for the two input images, fusing them to obtain a coarse-grained descriptor of image key points and further extracting a fine-grained descriptor; calculating the similarity between the coarse-grained descriptors of the image key points of the two input images to generate coarse-grained matching point pairs; combining the coarse-grained matching point pairs of the two input images with the fine-grained descriptors to crop the corresponding feature point blocks, calculating the correlation heat map between the feature point blocks, and estimating the offset between the two input images. The purpose of the present invention is to address the problems of modal changes and geometric distortions in images from different sensors, achieve matching with sub-pixel-level accuracy, and improve the accuracy of image matching.

Description

Technical Field

[0001] The present invention relates to visual image processing technology in the field of computers, and more particularly to a multi-modal image matching method and system based on sub-pixel deviation estimation. Background Art

[0002] Image matching is a basic technology in computer vision and is widely used in fields such as image fusion, image restoration, and depth estimation. In each field, methods for accurately matching images from different sensors have important application values. For example, in power line inspection, by combining visible light images and infrared images, information such as the shape, state, and working temperature of power equipment can be comprehensively obtained, providing more reliable data support for fault detection. However, how to accurately match images from different sensors has become a key problem in technical implementation. Multi-modal image matching has become an important research topic. Although significant progress has been made in multi-modal image matching in the past few decades, the position deviation between the estimated matching points and the ground truth points remains a problem to be solved.

[0003] The positional deviation in multimodal image matching is mainly caused by two reasons: the detector extracts non-corresponding key points, which may lead to the key points being matched to adjacent pixels of the ground truth, thus generating positional deviation; the extracted descriptors may lack sufficient robustness to the geometric distortion and modal changes of multimodal image pairs, resulting in many incorrect matches and generating positional deviation. To solve this problem, there are two major categories of existing research works, each with different limitations: The first category of methods is based on manually designed methods. Manually designed methods mainly focus on designing feature detectors and descriptors that are robust to geometric distortion or modal changes to address the multimodal image matching problem. Typical methods such as 3MRS fall into this category. This method refines the coarse-grained matching with positional deviation. The refinement module then calculates the template features through the phase consistency (PC) map for matching to improve the matching performance. Although this method can significantly improve the matching accuracy in some cases, its robustness to modal changes is limited. And the MSFT method, which uses the fast sample consensus (FSC) method and a hard elimination threshold to obtain the initial transformation matrix for the coarse-grained matching, and then adjusts the position of the coarse-grained matching to minimize the transformation error based on the initial matrix, thereby enhancing the transformation consistency of the adjusted matching and reducing the positional deviation. Although this method can reduce the positional deviation, it depends on the accuracy of the initial transformation matrix. If the initial matrix is inaccurate, the adjustment effect may be greatly reduced. The second category of methods is based on learning methods. Among them, detector-based methods such as Keypt2Subpx can refine any detector by predicting the offset vectors of the detected key points to obtain sub-pixel-level matching. However, when used in conjunction with sub-pixel-level detectors (such as SIFT), the performance of the method will decline and may not achieve the best results. Detector-free methods use the self-attention and cross-attention layers in the Transformer to obtain feature descriptors, thereby obtaining a global receptive field to generate dense matches. However, the dense matches and attention layers in the Transformer involve a large number of pixels, resulting in a significant increase in the computational requirements during training. Summary of the Invention

[0004] The technical problem to be solved by the present invention: Aiming at the above problems of the prior art, a multimodal image matching method and system based on sub-pixel deviation estimation are provided. The present invention aims to achieve sub-pixel-level accuracy matching and improve the image matching accuracy for the problems of modal changes and geometric distortion of different sensor images.

[0005] To solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0006] A multimodal image matching method based on sub-pixel deviation estimation, which includes inputting two input images of different modalities into a pre-trained multimodal image matching network to obtain the offset between the two input images. The processing of the two input images by the multimodal image matching network includes: extracting feature tensors of multiple scales from the two input images respectively, fusing to obtain a coarse-grained descriptor of image key points and further extracting a fine-grained descriptor; calculating the similarity between the coarse-grained descriptors of the image key points of the two input images to generate coarse-grained matching point pairs; combining the coarse-grained matching point pairs of the two input images with the fine-grained descriptors to crop the corresponding feature point blocks, calculating the correlation heat map between the feature point blocks and estimating the offset between the two input images.

[0007] Optionally, the functional expression for respectively extracting feature tensors of multiple scales from the two input images, fusing to obtain a coarse-grained descriptor of image key points and further extracting a fine-grained descriptor is:

[0008] ,

[0009] ,

[0010] Wherein, is the input image, is the encoder for extracting feature tensors of multiple scales, and the encoders of the two input images of different modalities do not share network parameters, is the operation of fusing to obtain a coarse-grained descriptor of image key points, is the coarse-grained descriptor of image key points, is the descriptor head composed of a multi-layer perceptron, is the fine-grained descriptor of image key points.

[0011] Optionally, the feature tensors of multiple scales respectively include three feature tensors with sizes of , , where and are the height and width of the input image respectively; the fusion to obtain a coarse-grained descriptor of image key points includes bilinearly upsampling the two feature tensors of , to the size of and then element-wise summing with the feature tensor with a size of to obtain the coarse-grained descriptor of image key points.

[0012] Optionally, the calculation of the similarity between the coarse-grained descriptors of the image key points of the two input images to generate coarse-grained matching point pairs includes: respectively using the detection head for the coarse-grained descriptors of the image key points of the two input images Obtain the key point detection score map , the detection head is composed of a convolutional layer and an activation function; for the key point detection score map The top K image key points with higher scores are obtained through a non-maximum suppression detector, and a confidence matrix is generated according to the coarse-grained descriptors of the top K image key points with the highest scores in the two input images, and the point pairs with the largest mutual confidence in the confidence matrix are used as the coarse-grained matching point pairs.

[0013] Optionally, the functional expression for generating the confidence matrix is:[[]]

[0014] ,

[0015] where is the confidence matrix, and are the coarse-grained descriptors of the input images and the input image respectively, is the transpose operation.

[0016] Optionally, the combining of the coarse-grained matching point pairs of the two input images with the fine-grained descriptors to crop the corresponding feature point blocks, calculate the correlation heat map between the feature point blocks, and estimate the offset between the two input images includes: for each coarse-grained matching point pair of the two input images , at one of the coarse-grained matching points a feature point block is cropped with a given radius r, and the fine-grained descriptor points are cropped with a given radius r in the area centered on another coarse-grained matching point to obtain a feature point block ; calculate the correlation heat map , the feature point block between the feature point blocks according to the following formula:[[]]

[0017] ,

[0018] where is the softmax activation function, is the transpose operation;

[0019] Estimate the offset between the two input images according to the following formula:[[]]

[0020] ,

[0021] where is the offset between the two input images, is the feature point block area with radius r, are the coordinates in the feature point block area is the associated heat map coordinates in the score value at.

[0022] Optionally, the functional expression of the loss function adopted by the multi-modal image matching network during training is:

[0023] ,

[0024] ,

[0025] wherein, is the loss function is the coarse-grained loss function, is the detection loss function, is the fine-grained loss function, is the sufficiency loss, is the peak loss, is the coupling loss function, and the functional expression of the coarse-grained loss function is:

[0026] ,

[0027] wherein, and respectively represent the calculated losses in the row and column directions, and represent the confidence matrix the similarity of the coarse-grained descriptions of the row column, the row column; the functional expression of the fine-grained loss function is:

[0028] ,

[0029] wherein, is the number of coarse-grained matching point pairs of two input images, is the variance of the associated heat map corresponding to the jth coarse-grained matching point pair the inverse of, represents the operation of taking the average value, is the inverse of the variance of the associated heat maps corresponding to all coarse-grained matching point pairs the inverse of, is the true offset between two input images, is the offset between two input images, and there is:

[0030] ,

[0031] ,

[0032] Among them, and are the image key points belonging to the input image and the input image respectively in a coarse-grained matching point pair, is the image key point in the input image , is the corresponding point of the image key point in the input image is the transformation matrix between the input image and the input image ; The functional expression of the coupling loss function is:

[0033] ,

[0034] Among them, , are the pixel sets of two input images respectively, is the confidence matching probability, is the score of the pixel in the key point detection score map corresponding to the input image , is the score of the pixel in the key point detection score map corresponding to the input image , and there is:

[0035] ,

[0036] Among them, and are the maximum values calculated in the row and column directions respectively.

[0037] In addition, the present invention also provides a multi-modal image matching system based on sub-pixel deviation estimation, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the multi-modal image matching method based on sub-pixel deviation estimation.

[0038] In addition, the present invention also provides a computer-readable storage medium, in which a computer program or instruction is stored, and the computer program or instruction is programmed or configured to execute the multi-modal image matching method based on sub-pixel deviation estimation through a processor.

[0039] In addition, the present invention also provides a computer program product, including a computer program or instruction, and the computer program or instruction is programmed or configured to execute the multi-modal image matching method based on sub-pixel deviation estimation through a processor.

[0040] Compared with the prior art, the present invention can mainly achieve the following beneficial effects: The present invention includes extracting feature tensors of multiple scales from two input images respectively to cope with modality changes, fusing the feature tensors of multiple scales to obtain a coarse-grained descriptor of image key points and further extracting a fine-grained descriptor to handle geometric distortion, calculating the similarity between the coarse-grained descriptors of the image key points of the two input images to generate coarse-grained matching point pairs; on the basis of the coarse-grained matching point pairs, a sub-pixel offset estimation method is designed, including: combining the coarse-grained matching point pairs of the two input images with the fine-grained descriptors to crop the corresponding feature point blocks, calculating the correlation heat map between the feature point blocks and estimating the offset between the two input images, so as to accurately adjust the position deviation in the coarse-grained matching, thereby improving the matching accuracy. The present invention can achieve accurate matching in multi-modal image matching, and helps to solve the problem of position offset between the estimated matching points and the ground truth points. Description of the Drawings

[0041] Figure 1 It is a schematic flowchart of the method of the embodiment of the present invention.

[0042] Figure 2 It is a schematic diagram of the network structure of the multi-modal image matching network in the embodiment of the present invention. Detailed Embodiments

[0043] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention will be further described in detail below with reference to the drawings in the embodiments of the present invention.

[0044] As Figure 1 shown, this embodiment provides a multi-modal image matching method based on sub-pixel deviation estimation, including inputting two input images of different modalities into a pre-trained multi-modal image matching network to obtain the offset between the two input images. The processing of the two input images by the multi-modal image matching network includes: extracting feature tensors of multiple scales from the two input images respectively, fusing them to obtain a coarse-grained descriptor of image key points and further extracting a fine-grained descriptor; calculating the similarity between the coarse-grained descriptors of the image key points of the two input images to generate coarse-grained matching point pairs; combining the coarse-grained matching point pairs of the two input images with the fine-grained descriptors to crop the corresponding feature point blocks, calculating the correlation heat map between the feature point blocks and estimating the offset between the two input images.

[0045] See Figure 2, in this embodiment, the multi-modal image matching network can be divided into three stages: multi-scale feature extraction, pixel-level feature matching, and sub-pixel level offset estimation. In the multi-scale feature extraction stage, feature tensors of multiple scales are extracted from two input images respectively, fused to obtain a coarse-grained descriptor of the image key points, and further a fine-grained descriptor is extracted. In the pixel-level feature matching stage, the similarity between the coarse-grained descriptors of the image key points of the two input images is calculated to generate coarse-grained matching point pairs. In the sub-pixel level offset estimation stage, the coarse-grained matching point pairs of the two input images are combined with the fine-grained descriptors to crop the corresponding feature point blocks, calculate the correlation heat map between the feature point blocks, and estimate the offset between the two input images.

[0046] In this embodiment, the functional expression for extracting feature tensors of multiple scales from two input images respectively, fusing to obtain a coarse-grained descriptor of the image key points, and further extracting a fine-grained descriptor is as follows:

[0047] ,

[0048] ,

[0049] Among them, is the input image, is the encoder for extracting feature tensors of multiple scales, and the encoders of two input images of different modalities do not share network parameters, is the operation of fusing to obtain a coarse-grained descriptor of the image key points, is the coarse-grained descriptor of the image key points, is the descriptor head composed of multi-layer perceptrons, is the fine-grained descriptor of the image key points. The encoder for extracting feature tensors of multiple scales can adopt the required encoder type according to needs. For example, as an optional implementation manner, see Figure 2 , in this embodiment, an adaptive encoder is adopted; the multi-layer perceptron constituting the descriptor head is composed of multiple fully connected layers plus batch normalization and activation functions.

[0050] Two input images of different modalities can be expressed as: , and their corresponding pixels are and , and The geometric transformation between

[0051] is as follows:

[0052] Among them, represents the geometric transformation matrix between image pairs.

[0053] As an alternative implementation, in this embodiment, the feature tensors of multiple scales respectively include three feature tensors with sizes of , , , where and are the height and width of the input image respectively; the coarse-grained descriptor of the image key points obtained by fusion includes bilinearly upsampling the two feature tensors of , to the size of and then element-wise summing with the feature tensor of size to obtain the coarse-grained descriptor of the image key points. As an alternative implementation, see Figure 2 , in this embodiment, the coarse-grained descriptor of the image key points obtained by fusion is specifically implemented through a multi-scale fusion module. For a bimodal input image, the common information of different modal images is extracted through two adaptive encoders with non-shared weights, and then the multi-scale feature fusion module is used to improve the robustness of the descriptor to geometric distortion. The coarse-grained descriptor is obtained through the multi-scale feature fusion module and input into the descriptor head to obtain the fine-grained descriptor.

[0054] In this embodiment, calculating the similarity between the coarse-grained descriptors of the image key points of two input images to generate coarse-grained matching point pairs includes: respectively using the detection head for the coarse-grained descriptors of the image key points of the two input images to obtain the key point detection score map , the detection head is composed of a convolutional layer plus an activation function, and the 128-dimensional coarse-grained descriptor becomes a one-dimensional score matrix after passing through the detection head ; for the key point detection score map , the top K image key points with higher scores are extracted through a non-maximum suppression detector, and a confidence matrix is generated according to the coarse-grained descriptors of the top K image key points with the highest scores in the two input images, and the point pairs with the maximum mutual confidence in the confidence matrix are used as the coarse-grained matching point pairs. In this embodiment, the functional expression for generating the confidence matrix is:

[0055] ,

[0056] where is the confidence matrix, and are the coarse-grained descriptors of the input image and the input image respectively, and is the transpose operation.

[0057] In this embodiment, a sub-pixel offset estimation method is designed based on coarse-grained matching point pairs combined with fine-grained descriptors to accurately adjust the position deviation in coarse-grained matching and improve the matching accuracy. Specifically, for each pair of coarse-grained matching points in the two input images, the steps of combining the coarse-grained matching point pairs of the two input images with fine-grained descriptors, cropping the corresponding feature point blocks, calculating the correlation heat map between the feature point blocks, and estimating the offset between the two input images include: , crop out the feature point block at a given radius r centered at one of the coarse-grained matching points , and crop out the feature point block at a given radius r centered at the other coarse-grained matching point with the fine-grained descriptor points in the area; calculate the correlation heat map between the feature point block and the feature point block according to the following formula:

[0058] ,

[0059] where, is the softmax activation function, is the transpose operation;

[0060] Estimate the offset between the two input images according to the following formula:

[0061] ,

[0062] where, is the offset between the two input images, is the feature point block area (patch) with radius r, are the coordinates in the feature point block area, is the correlation heat map and is the score value at the coordinate

[0063] In this embodiment, the functional expression of the loss function adopted by the multi-modal image matching network during training is:

[0064] ,

[0065] ,

[0066] where, is the loss function is the coarse-grained loss function, is the detection loss function, is the fine-grained loss function, is the sufficiency loss, is the peak loss, is the coupling loss function.

[0067] For any pair of bimodal images in the dataset, in order to obtain a coarse-grained match at the pixel level, it is necessary to align the descriptors of the corresponding pixels. Even in the case of geometric deformation and modality change, the similarity of the descriptors of the corresponding pixels should be maximized. To ensure that the similarity of the descriptors of the corresponding pixels is mutually maximized, the system uses the dual-softmax loss in the confidence matrix. Therefore, the functional expression of the coarse-grained loss function in this embodiment is:

[0068] ,

[0069] where, and respectively represent the calculated losses in the row and column directions, and represent the confidence matrix on the diagonal of the -th row -th column, the -th row -th column of the similarity of the coarse-grained descriptions; the functional expression of the fine-grained loss function is:

[0070] ,

[0071] where, is the number of coarse-grained matching point pairs of the two input images, is the variance of the associated heat map corresponding to the j-th coarse-grained matching point pair inverse, represents the operation of taking the average value, is the variance of the associated heat maps corresponding to all coarse-grained matching point pairs inverse, is the true offset between the two input images, is the offset between the two input images, and there is:

[0072] ,

[0073] ,

[0074] where, and are the image key points belonging to the input image and the input image respectively in a coarse-grained matching point pair, is the image key point in the input image in the input image corresponding point,​ is the input image and the input image between the transformation matrices. In order to estimate the sub-pixel level position offset of the coarse-grained matching, it is necessary to establish a sub-pixel level alignment for the coarse-grained matching key point pairs for supervision. In addition, considering that there are some low-quality corresponding relationships in the coarse-grained matching, the system introduces an interpretable uncertainty measure as the weight of the fine-grained loss function . For each coarse-grained matching, calculate the variance of the corresponding heat map as the uncertainty measure. A high variance indicates the existence of multiple or scattered patterns, reflecting low-quality or fuzzy predictions. Therefore, the fine-grained loss function optimizes the distance between the estimated position offset and the ground truth offset, achieving sub-pixel level accuracy, thereby improving the matching performance.

[0075] In order to detect repeatable key points at the corresponding positions, the detection score maps of the input image and should be similar. In addition, to ensure that key points in the local area can be detected, the scores of the key points should be significant. Therefore, the detection loss function includes a repeatability loss, a peak loss, and a coupling loss function. The function expression of the repeatability loss is:

[0076] ,

[0077] where is the number of corresponding pixel pairs , and are the detection score maps of the input image and respectively. are the feature point blocks centered at with a radius of respectively. The repeatability loss optimizes the scores of the corresponding patches to generate repeatable key points. The peak loss enhances the significance of the key points. The function expression of the peak loss is:

[0078] ,

[0079] where is the set of overlapping patches of size with a stride of 8 in the image, is the number of patches. The peak loss promotes the introduction of significant points in the local area by minimizing the mean of the patch scores and maximizing the maximum value.

[0080] Although repeatable key points are extracted by optimizing the repeatability loss and peak loss, low-quality key points with low matching success rates are also detected, which reduce the matching effect. Previous studies have shown that combining detection and description can suppress these low-quality key points. Therefore, the system designs a coupled loss function based on the confidence matrix and the detection score map to couple feature detection and description. The functional expression of this coupled loss function is:

[0081] ,

[0082] where , are the pixel sets of two input images respectively, is the confidence matching probability, is the score of pixel in the key point detection score map corresponding to the input image , is the score of pixel in the key point detection score map corresponding to the input image , and there is:

[0083] ,

[0084] where and calculate the maximum values in the row and column directions respectively. It should be noted that 's gradient only backpropagates through . The coupled loss function improves the detection scores of pixels with high confidence matching probabilities, which is helpful for suppressing low-quality key points and detecting key points with high matching success rates.

[0085] To verify the multi-modal image matching method based on sub-pixel deviation estimation in this embodiment, the development language used in this embodiment is Python, and the deep learning framework is Pytorch. The model is trained on a machine equipped with an i9-11900k CPU (3.50GHz×16) and an NVIDIA GeForce RTX 3090 graphics card. A multi-modal image dataset collected by ReDFeat is used. This multi-modal image dataset includes three cross-modal images, namely visible light-synthetic aperture radar (VIS-SAR) images, visible light-infrared (VIS-IR) images, and visible light-near infrared (VIS-NIR) images: The VIS-SAR dataset includes 2011 pairs and 424 pairs of aligned images, with a unified size of 512×512, which are used for training and testing. This dataset is captured by satellites and includes scenes from fields and urban areas. The VIS-IR images come from the RGB-LWIR dataset and the RoadScene dataset. The RGB-LWIR dataset provides 44 pairs of static images, mainly taken of buildings during the day. The RoadScene dataset provides 221 pairs of images, containing many objects such as people and cars, covering both day and night. The VIS-IR dataset has a total of 265 pairs of aligned images, with an average size of 533×321. 47 images are randomly selected as the test set, and the remaining image pairs are used for training. The VIS-NIR images are provided by the RGB-NIR scene dataset. This dataset contains 9 scenes, forming 477 pairs of aligned images, with an average size of 983×686, including scenes such as rural areas, fields, forests, indoors, mountains, old buildings, streets, cities, and waters. The VIS-NIR images are randomly divided into a training set and a test set according to a ratio of 3:1. The training set has 345 pairs, and the test set has 128 pairs. It should be noted that the test set has been manually verified and screened to ensure more reliable ground truth. During training, the Adam optimizer is used for training, with a total of 120,000 steps, an initial learning rate of 3.0×10 -4 , and the batch size is 16. In this embodiment, the matching success rate (SR), the number of correct matches (NCM), the mean error (ME), and the root mean square error (RMSE) are used as evaluation metrics, and the fine-grained matching point pairs are calculated residual , and the residual less than or equal to three pixels is regarded as a correct match. The expression of the residual is:

[0086] ,

[0087] where and are fine-grained matching point pairs in the input image and the input image of the image key points is the input image and the input image is the transformation matrix between them. The number of correct matches is the number of correct matches NCM. If the number of correct matches NCM > 10, it is considered a successful match. The matching success rate SR represents the percentage of the number of successfully matched image pairs in the total number of image pairs. The average error ME and the root mean square error RMSE respectively represent the matching error and the error fluctuation. Their functional expressions are:

[0088] ,

[0089] ,

[0090] where is the th residual .

[0091] In the sub-pixel level offset estimation stage, the radius r of the feature point block region (patch) is an important parameter that determines the offset estimation range, thus affecting the performance of the system. In this embodiment, training with different radii r was carried out on the aforementioned dataset, and the final results are shown in Table 1.

[0092] Table 1: Training results with different radii r

[0093]

[0094] In Table 1, "↑" indicates that the larger the index, the better, and "↓" indicates that the smaller the index, the better. Referring to Table 1, it can be seen that when the radius r is 2px, the matching success rate (SR), the number of correct matches (NCM), the average error (ME), and the root mean square error (RMSE) all achieve the optimal results.

[0095] To verify the effectiveness of each module in the multi-modal image matching network of the method in this embodiment, in this embodiment, the method in this embodiment and the following variants of the method in this embodiment were trained and tested on the aforementioned dataset: the method without the multi-scale feature fusion block (W / O-FB), the method without the detector and descriptor coupling (W / O-CP), and the method without the sub-pixel level offset estimation (W / O-OE). Among them, the multi-scale feature fusion module performs a feature fusion operation on the image after passing through the adaptive encoder, and fuses , , the feature tensors of three scales. Multi-scale feature fusion can improve the robustness and accuracy of matching when dealing with complex scenes such as scale changes, insufficient texture, and occlusion. The method without the multi-scale feature fusion block (W / O-FB) directly The feature tensor passes through a weight - shared fully - convolutional encoder to replace the multi - scale feature fusion module and outputs a coarse - grained descriptor, thus reducing the adaptability to geometric distortion and resulting in a decline in the results compared to the method of this example. The detector - descriptor coupling method generates a detection score map by performing feature detection on the coarse - grained descriptor. Through the detection score map, unreliable key points with a low matching success rate can be suppressed. The method without detector - descriptor coupling (W / O - CP) abandons the coupling of the detector and the descriptor, and the feature descriptor is not filtered by the score map generated by the detector. This means that all feature points will participate in the matching, which causes many key points with a low matching success rate to be detected, resulting in a performance decline. The method without sub - pixel level offset estimation (W / O - OE) deletes the sub - pixel level offset estimation module and directly uses the pixel - level rough matching result for feature matching, making the obtained pixel - level rough matching result not fine enough, thus reducing the matching result. The final results are shown in Table 2.

[0096] Table 2: Training results of the method of this example (this method) and its variants on the VIS - SAR dataset

[0097]

[0098] In Table 2, "↑" indicates that the larger the index, the better, and "↓" indicates that the smaller the index, the better. Referring to Table 2, it can be seen that the method of this example has achieved the optimal results in terms of the matching success rate (SR), the number of correct matches (NCM), the mean error (ME), and the root - mean - square error (RMSE) compared to its variants.

[0099] To quantify the improvement of sub - pixel level offset estimation on sub - pixel accuracy, in this example, the performance of the method of this example and the method without sub - pixel level offset estimation (W / O - OE) is compared on the VIS - SAR dataset, the VIS - IR dataset, and the VIS - NIR dataset. The number of matches representing sub - pixel accuracy, and the final results are shown in Table 3.

[0100] Table 3: Comparison results between the method of this example (this method) and the method without sub - pixel level offset estimation

[0101]

[0102] In Table 3, "↑" indicates that the larger the index, the better, and "↓" indicates that the smaller the index, the better. Referring to Table 3, it can be seen that the sub - pixel level offset estimation in the multi - modal image matching network of the method of this example improves the results of all the indicators listed in Table 3, and the number of matches with sub - pixel accuracy also increases significantly, especially on the VIS - NIR dataset, increasing by approximately 125%.

[0103] In addition, this embodiment also provides a multimodal image matching system based on sub-pixel deviation estimation, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the multimodal image matching method based on sub-pixel deviation estimation.

[0104] In addition, this embodiment also provides a computer-readable storage medium, in which a computer program or instruction is stored, and the computer program or instruction is programmed or configured to execute the multimodal image matching method based on sub-pixel deviation estimation through a processor.

[0105] In addition, this embodiment also provides a computer program product, including a computer program or instruction, and the computer program or instruction is programmed or configured to execute the multimodal image matching method based on sub-pixel deviation estimation through a processor.

[0106] Those skilled in the art should understand that the technical solution provided by the present invention can be in the form of a method, a system, or a computer program product. Therefore, the present invention can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes. The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device realizes the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide means for realizing the functions specified in one processFigure 1 one process or multiple processes and / or blocks Figure 1 steps of the functions specified in one block or multiple blocks

[0107] The above are only the preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the concept of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A multimodal image matching method based on sub-pixel deviation estimation, characterized in that Including inputting two input images of different modalities into a pre-trained multi-modal image matching network to obtain the offset between the two input images. The processing of the two input images by the multi-modal image matching network includes: respectively extracting feature tensors of multiple scales for the two input images, fusing to obtain a coarse-grained descriptor of the image key points and further extracting a fine-grained descriptor; calculating the similarity between the coarse-grained descriptors of the image key points of the two input images to generate coarse-grained matching point pairs; combining the coarse-grained matching point pairs of the two input images with the fine-grained descriptors to crop the corresponding feature point blocks, calculating the correlation heat map between the feature point blocks and estimating the offset between the two input images, including: for each coarse-grained matching point pair of the two input images , at one of the coarse-grained matching points , crop out a feature point block with a given radius r, and dot the fine-grained descriptor at the other coarse-grained matching point as the center of the region to crop out a feature point block with a given radius r; calculate the correlation heat map between the feature point block and the feature point block according to the following formula: ， Among them, is the softmax activation function, is the transpose operation; Estimate the offset between two input images according to the following formula: ， Among them, is the offset between two input images, is the feature point block area with radius r, is the coordinate in the feature point block area, is the associated heat map and the coordinate is the score value at that position.

2. The multimodal image matching method based on sub-pixel deviation estimation according to claim 1, wherein, The function expressions for respectively extracting feature tensors of multiple scales from the two input images, fusing to obtain a coarse-grained descriptor of the image key points, and further extracting a fine-grained descriptor are: ， ， Among them, is the input image, is the encoder for extracting feature tensors of multiple scales, and the network parameters are not shared between the encoders of two input images of different modalities, is the operation for fusing to obtain the coarse-grained descriptor of the image key points, is the coarse-grained descriptor of the image key points, is the descriptor head composed of a multi-layer perceptron, is the fine-grained descriptor of the image key points.

3. The multimodal image matching method based on sub-pixel deviation estimation according to claim 1, wherein The feature tensors of multiple scales respectively include three feature tensors with sizes of , , , where and are the height and width of the input image respectively; The fused coarse-grained descriptor of the image key points includes , of the two feature tensors are bilinearly upsampled to in size and then element-wise summed with the one of size to obtain the coarse-grained descriptor of the image key points.

4. The multimodal image matching method based on sub-pixel deviation estimation according to claim 1, wherein Calculating the similarity between the coarse-grained descriptors of the image key points of the two input images to generate coarse-grained matching point pairs includes: respectively using the detection head for the coarse-grained descriptors of the image key points of the two input images to obtain a key point detection score map , where the detection head is composed of a convolutional layer plus an activation function; for the key point detection score map , the top K image key points with higher scores are extracted through a non-maximum suppression detector, a confidence matrix is generated according to the coarse-grained descriptors of the top K image key points with the highest scores of the two input images, and the point pairs with the maximum mutual confidence in the confidence matrix are used as coarse-grained matching point pairs.

5. The multimodal image matching method based on sub-pixel deviation estimation according to claim 4, characterized in that, The function expression for generating the confidence matrix is: ， Among them, is the confidence matrix, and are the coarse-grained descriptors of the input image and the input image respectively, is the transpose operation.

6. The multimodal image matching method based on sub-pixel deviation estimation according to claim 1, characterized in that The function expression of the loss function adopted by the multi-modal image matching network during training is: ， ， Among them, is the loss function is the coarse-grained loss function, is the detection loss function, is the fine-grained loss function, is the sufficiency loss, is the peak loss, is the coupling loss function, and the functional expression of the coarse-grained loss function is: ， Among them, and respectively represent the calculated losses in the row and column directions, and represent the confidence matrix on the diagonal of row column, the similarity of the coarse-grained description of the row column; the functional expression of the fine-grained loss function is: ， Among them, is the number of coarse-grained matching point pairs of two input images, is the variance of the associated heat map corresponding to the j-th coarse-grained matching point pair inverse, represents the operation of taking the average value, is the variance of the associated heat maps corresponding to all coarse-grained matching point pairs inverse, is the true offset between two input images, is the offset between two input images, and there is: ， ， Among them, and are the image key points belonging to the input image and the input image respectively in a coarse-grained matching point pair. is the image key point in the input image , and the corresponding point in the input image . is the transformation matrix between the input image and the input image . The functional expression of the coupling loss function is: ， Among them, and are respectively the pixel sets of two input images, is the confidence matching probability, is the input image corresponding key point detection score map in the pixel score, is the input image corresponding key point detection score map in the pixel score, and there is: ， Among them, and are respectively the maximum values in the row and column directions.

7. A multimodal image matching system based on sub-pixel deviation estimation, comprising a microprocessor and a memory connected to each other, characterized in that, The microprocessor is programmed or configured to execute the multi-modal image matching method based on sub-pixel deviation estimation according to any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instruction is programmed or configured to execute the multi-modal image matching method based on sub-pixel deviation estimation according to any one of claims 1 to 6 through a processor.

9. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instruction is programmed or configured to execute the multi-modal image matching method based on sub-pixel deviation estimation according to any one of claims 1 to 6 through a processor.

Images