Unified multi-modal image matching method and system based on disentangled representation learning
By explicitly separating semantic and domain features through the unentangled representation learning method, the stability and applicability issues in multimodal image matching are solved, and the stability and accuracy of cross-modal image matching are improved.
Patent Information
- Application Number
- CN202611554760.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-09
- Publication Date
- 2026-11-10
AI Technical Summary
Existing technologies suffer from insufficient stability in key point localization, weak descriptor discrimination, and high mismatch rate in multimodal image matching. Furthermore, existing methods are typically developed for specific modal pairs and are difficult to apply to various cross-modal image matching tasks.
We employ a de-entangled representation learning method, which extracts shallow features by sharing a shallow encoder, and uses a semantic encoder and a domain encoder to de-entangle and reconstruct features. We then train the network model by combining modality classification loss, reconstruction loss, consistency loss, and matching loss, explicitly separating semantic and domain features to generate stable modality-invariant features.
It enables the application of a single model to multiple cross-modal image matching tasks, improves the ability of image recognition or understanding across modalities, reduces interference from modality-specific domain information, and enhances the stability and accuracy of matching.
Smart Images

Figure CN123071685A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image recognition or understanding technology, and specifically relates to a unified multimodal image matching method and system based on unentangled representation learning. Background Technology
[0002] Multimodal image matching is a fundamental task in computer vision, widely applied in image recognition and understanding tasks such as image fusion, 3D reconstruction, visual localization, depth estimation, and remote sensing image registration. Its aim is to achieve cross-alignment of corresponding pixels between images obtained from different sensors or imaging mechanisms. With the widespread use of multi-source visual data such as optical, SAR, infrared, and map data in practical systems, image matching methods not only need to remain stable in single-modal combinations but also need the ability to process multiple modal combinations uniformly. Traditional hand-designed methods typically design feature detectors and descriptors by manually constructing prior knowledge to mitigate nonlinear radiation distortion caused by modal differences. These methods use more robust phase-consistency features instead of image intensity information for feature point detection, do not rely on training data, possess a certain degree of cross-modal adaptability, and exhibit corresponding robustness to nonlinear radiation distortion. However, because their core reliance on low-level hand-designed visual cues limits their ability to handle complex cross-modal variations, generally suffering from insufficient stability in keypoint localization, weak descriptor discriminability, and increased mismatch rates. Learning-based methods utilize deep convolutional neural networks to learn shared representations across modalities. Existing paradigms typically perform direct feature extraction and alignment within a shared embedding space, hoping that corresponding representations from different modalities will become similar in the feature space. While effective to some extent, this direct alignment strategy is still subject to two significant limitations. First, the shared features learned through these methods may contain modality-specific information, which weakens feature discriminability and thus limits matching accuracy. Second, these methods are usually developed for specific modality pairs and cannot be directly applied to other cross-modal images; instead, they require separate training for different modality combinations. Furthermore, when multiple cross-modal datasets are directly mixed for training instead of using specific modality pairs, the model becomes more susceptible to complex modality differences, which can lead to slow convergence or even degraded matching performance. Therefore, how to explicitly separate features into modality-invariant semantic information and modality-specific domain information in a single model is a key technical problem that needs to be addressed for unified multimodal image matching. Summary of the Invention
[0003] The technical problem to be solved by the present invention is to provide a unified multimodal image matching method and system based on unwrapped representation learning, which addresses the above-mentioned problems in the prior art. The present invention aims to learn stable modality-invariant semantic features from multimodal images through explicit feature unwrapping and cross-recombination constraints, suppress the interference of modality-specific domain information, and enable a single model to be applicable to multiple cross-modal image matching tasks, thereby improving the ability of image recognition or understanding across modalities.
[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A unified multimodal image matching method based on unwrapped representation learning includes the following steps: S101, during the training phase, images of different modalities are input into a shared shallow encoder to extract shallow features. The semantic encoder and domain encoder are used to unwrap features to extract semantic and domain features. The decoder performs homologous reconstruction and cross-reconstruction on the semantic and domain features. The cross-reconstructed shallow features are re-encoded by the semantic encoder and domain encoder respectively to generate recoded features. Based on the obtained features, the network model consisting of a shared shallow encoder, semantic encoder, domain encoder and decoder is jointly trained using modality classification loss, reconstruction loss, consistency loss and matching loss. S102, In the inference stage, semantic features are extracted from the image pairs to be matched using the shared shallow encoder and semantic encoder in the trained network model. Then, the semantic features of each key point in the image pairs to be matched are used as feature descriptors. Key point matching is performed based on the feature descriptors of the key points to obtain a set of key point pairs.
[0005] Optionally, the function expression for feature unwrapping in step S101 to extract semantic features and domain features is: ; ; ; ; in, and Images and semantic features and Images and Domain characteristics, For semantic encoders, For a domain encoder, and Images and Shallow features.
[0006] Optionally, the function expression for reconstructing semantic features and domain features from the same source and cross-reconstruction by the decoder in step S101 is as follows: ; ; ; ; in, and Images and Homologous reconstruction shallow features, For image semantic features and The shallow features obtained by concatenating the domain features are cross-reconstructed. image semantic features and images The shallow features obtained by concatenating the domain features are cross-reconstructed. For decoder, This indicates feature splicing.
[0007] Optionally, in step S101, the function expression for re-encoding the shallow cross-reconstructed features obtained from the cross-reconstruction using a semantic encoder and a domain encoder respectively to generate recoded features is as follows: ; ; in, and Images and Re-encoded semantic features and Images and Features of the recoding domain For semantic encoders, For a domain encoder, For image semantic features and The shallow features obtained by concatenating the domain features are cross-reconstructed. image semantic features and images The shallow cross-reconstruction features are obtained by concatenating the domain features, and the recoded semantic features and recoded domain features constitute the recoded features.
[0008] Optionally, when training the network model consisting of a shared shallow encoder, semantic encoder, domain encoder, and decoder using modality classification loss, reconstruction loss, consistency loss, and matching loss in step S101, the expression for the calculation function of the modality classification loss is: ; in, For modal classification loss, and For image and modal tags, Represents the modality classification head. Represents cross-entropy loss, and Images and Domain characteristics; The expression for the calculation function of the reconstruction loss is: ; in, To reconstruct the loss, and Images and Shallow features, and Images and Homologous reconstruction shallow features; The expression for the calculation function of the consistency loss is: ; in, For consistency loss, and Images and semantic features and Images and Re-encoded semantic features and Images and The recoding domain features, and the recoding semantic features and the recoding domain features together constitute the recoding features; The matching loss consists of some or all of the descriptor loss, repeatability loss, and local peak loss. The calculation function expression for the descriptor loss is as follows: ; ; in, For descriptor loss, and These represent the softmax operations along the row and column directions, respectively. and Represent the confidence matrix respectively Similarity of matching descriptors on the diagonal and Confidence matrix index number, and Images and The semantic features of the superscript For transpose; The expression for the calculation function of the repeatability loss is: ; in, For repetitive loss, For image and The number of pixels in and Images and The set of coordinates of the same pixel in the detection score image obtained by the detection head. and Images and The coordinates of the i-th identical pixel in the detection score image. and They are respectively and The detection score, in superscript This is a transpose operation; The expression for the calculation function of the local peak loss is: ; in, For local peak loss, For a set of overlapping image patches Size, For image and The set of overlapping image patches in the detection score map. and These are taking the average and taking the maximum value, respectively. For the first Overlapping image patches The detection score, and For the first Overlapping image patches In the image and The sum of the detection scores in the detection score graph.
[0009] Optionally, in step S101, when jointly training the network model consisting of a shared shallow encoder, semantic encoder, domain encoder, and decoder using modality classification loss, reconstruction loss, consistency loss, and matching loss, the total loss function used is: ; ; in, For the total loss function, To match the loss, For descriptor loss, For repetitive loss, For local peak loss, For modal classification loss, To reconstruct the loss, This results in a loss of consistency.
[0010] Optionally, before inputting the images of different modalities into the sample to extract shallow features using the shared shallow encoder in step S101, the method further includes applying a random affine transformation to the images of different modalities in the sample. The random affine transformation includes changing part or all of the orientation, scale, shape, and spatial position of the image.
[0011] Optionally, step S102 includes: S201, use the shared shallow encoder in the trained network model to extract shallow features from the image pair to be matched consisting of image a and image b, and then use the shallow features to extract semantic features through the semantic encoder. S202, the semantic features of each key point in image a and image b are used as feature descriptors, and key point matching is performed based on the feature descriptors of the key points to obtain a set of key point pairs.
[0012] Optionally, after step S201 and before step S202, the method further includes: detecting the semantic features of image a and image b using the detection head in the trained network model to obtain detection score maps, and performing non-maximum suppression on the feature point positions in the detection score maps for each image in image a and image b to obtain key point positions.
[0013] A unified multimodal image matching system based on unentangled representation learning includes an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the unified multimodal image matching method based on unentangled representation learning.
[0014] Compared with existing technologies, the present invention mainly achieves the following beneficial effects: The method of the present invention includes extracting shallow features from images of different modalities by sharing a shallow encoder, using a semantic encoder and a domain encoder to unwrap features to extract semantic and domain features, performing homologous reconstruction and cross-reconstruction of semantic and domain features through a decoder, and jointly training the network using modality classification loss, reconstruction loss, consistency loss, and matching loss; in the inference stage, generating detection score maps and feature descriptors using semantic features, and outputting keypoint matching results through keypoint extraction, descriptor sampling, and nearest neighbor matching. The present invention can explicitly interpret modality changes as domain differences and separate features into semantic and domain components. This design reduces modality-induced interference and promotes the realization of a unified method that can handle multiple modality combinations in a single model. It can learn stable modality-invariant semantic features from multimodal images, suppress the interference of modality-specific domain information, and enable a single model to be applicable to multiple cross-modal image matching tasks, improving the ability of image recognition or understanding across modalities. Attached Figure Description
[0015] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.
[0016] Figure 2 This is a schematic diagram of the network structure and loss function of the network model in an embodiment of the present invention.
[0017] Figure 3 The present invention provides visualization matching results of multiple sets of samples in two modalities: optical image and SAR image. (a) is the visualization matching result of the first set of samples, (b) is the visualization matching result of the second set of samples, and (c) is the visualization matching result of the third set of samples.
[0018] Figure 4 The above are the visualization matching results of multiple samples in two modalities, optical image and infrared image, according to the embodiment of the present invention. (a) is the visualization matching result of the first group of samples, (b) is the visualization matching result of the second group of samples, and (c) is the visualization matching result of the third group of samples.
[0019] Figure 5 The present invention provides visualization matching results of multiple sets of samples in two modalities: optical image and map. (a) is the visualization matching result of the first set of samples, (b) is the visualization matching result of the second set of samples, and (c) is the visualization matching result of the third set of samples.
[0020] Figure 6This invention presents a comparison of the visual matching results of multiple samples under two modalities: optical image and SAR image. (a1) shows the visual matching result of the ReDFeat method, (a2) shows the checkerboard pattern used for the ReDFeat method samples, and (b1) shows the MINIMA method. LG The method's visual matching results, (b2) are for MINIMA LG The sample used by the method is a checkerboard pattern; (c1) is the visual matching result of the SGPFeat method, (c2) is the checkerboard pattern used by the SGPFeat method; (d1) is the visual matching result of the SPEN method, (d2) is the checkerboard pattern used by the SPEN method; (e1) is the visual matching result of the method in this embodiment, (e2) is the checkerboard pattern used by the method in this embodiment.
[0021] Figure 7 This invention presents a comparison of the visual matching results of multiple samples under two modalities: optical image and infrared image. (a1) shows the visual matching result of the ReDFeat method, (a2) shows the checkerboard pattern used for the ReDFeat method samples, and (b1) shows the MINIMA method. LG The method's visual matching results, (b2) are for MINIMA LG The sample used by the method is a checkerboard pattern; (c1) is the visual matching result of the SGPFeat method, (c2) is the checkerboard pattern used by the SGPFeat method; (d1) is the visual matching result of the SPEN method, (d2) is the checkerboard pattern used by the SPEN method; (e1) is the visual matching result of the method in this embodiment, (e2) is the checkerboard pattern used by the method in this embodiment.
[0022] Figure 8 This invention presents a comparison of the visualization matching results of multiple samples under two modalities: optical image and map. (a1) shows the visualization matching result of the ReDFeat method, (a2) shows the checkerboard pattern used for the ReDFeat method samples, and (b1) shows the MINIMA method. LG The method's visual matching results, (b2) are for MINIMA LG The sample used by the method is a checkerboard pattern; (c1) is the visual matching result of the SGPFeat method, (c2) is the checkerboard pattern used by the SGPFeat method; (d1) is the visual matching result of the SPEN method, (d2) is the checkerboard pattern used by the SPEN method; (e1) is the visual matching result of the method in this embodiment, (e2) is the checkerboard pattern used by the method in this embodiment. Detailed Implementation
[0023] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.
[0024] like Figure 1 As shown, the unified multimodal image matching method based on unwrapped representation learning in this embodiment includes the following steps: S101, during the training phase, images of different modalities are input into a shared shallow encoder to extract shallow features. The semantic encoder and domain encoder are used to unwrap features to extract semantic and domain features. The decoder performs homologous reconstruction and cross-reconstruction on the semantic and domain features. The cross-reconstructed shallow features are re-encoded by the semantic encoder and domain encoder respectively to generate recoded features. Based on the obtained features, the network model consisting of a shared shallow encoder, semantic encoder, domain encoder and decoder is jointly trained using modality classification loss, reconstruction loss, consistency loss and matching loss. S102, In the inference stage, semantic features are extracted from the image pairs to be matched using the shared shallow encoder and semantic encoder in the trained network model. Then, the semantic features of each key point in the image pairs to be matched are used as feature descriptors. Key point matching is performed based on the feature descriptors of the key points to obtain a set of key point pairs.
[0025] like Figure 2 As shown, the function expression for feature unwrapping in step S101 to extract semantic and domain features is: ; ; ; ; in, and Images and semantic features and Images and Domain characteristics, For semantic encoders, For a domain encoder, and Images and The shallow features are extracted from the input image. The shared shallow encoder E is used to extract shallow features containing structural and modal appearance information; the semantic encoder... Used to extract modality-invariant semantic features; domain encoder Used to extract modality-specific features. Among them, the image... and images It can be used for different modal image pairs, such as optical images and SAR images, optical images and infrared images, and optical images and map images.
[0026] Image and The semantic and domain features are reconstructed using decoder D to obtain shallow features. Furthermore, by exchanging domain features from different modalities, homologous reconstructed shallow features and cross-reconstructed shallow features after exchanging domain features are obtained. For example... Figure 2 As shown, the function expression for reconstructing semantic features and domain features from the same source and cross-reconstruction by the decoder in step S101 is as follows: ; ; ; ; in, and Images and Homologous reconstruction shallow features, For image semantic features and The shallow features obtained by concatenating the domain features are cross-reconstructed. image semantic features and images The shallow features obtained by concatenating the domain features are cross-reconstructed. For decoder, This represents feature concatenation. Decoder D is used to recover shallow features based on semantic and domain features, and supports domain feature exchange between different images. This embodiment employs both homologous reconstruction and cross-reconstruction to jointly constrain the feature unwrapping process. In homologous reconstruction, the semantic and domain features of the same image are concatenated and input into decoder D to recover the corresponding shallow features; in cross-reconstruction, the image... semantic features and images Concatenate the domain features to create the image semantic features and images The domain features are concatenated to obtain the cross-reconstructed shallow features. These cross-reconstructed shallow features are then input again. and If the semantic features and domain features can remain consistent with the original unwrapped features, it indicates that the unwrapping process can stably distinguish between semantic information and modal information.
[0027] like Figure 2 As shown, the function expression for re-encoding the shallow cross-reconstructed features obtained in step S101 using a semantic encoder and a domain encoder respectively to generate recoded features is as follows: ; ; in, and Images and Re-encoded semantic features and Images and Features of the recoding domain For semantic encoders, For a domain encoder, For image semantic features and The shallow features obtained by concatenating the domain features are cross-reconstructed. image semantic features and images The shallow cross-reconstruction features are obtained by concatenating the domain features, and the recoded semantic features and recoded domain features constitute the recoded features.
[0028] like Figure 2 As shown, in step S101, when jointly training the network model consisting of a shared shallow encoder, semantic encoder, domain encoder, and decoder using modality classification loss, reconstruction loss, consistency loss, and matching loss, the modality classification loss is used to supervise the expression of modality-related appearance information by domain features, thereby reducing the residual domain information in semantic features; the reconstruction loss is calculated directly in the shallow feature space instead of reconstructing the original image, thus reducing the optimization difficulty and retaining image information useful for matching; the consistency loss constrains the semantic and domain features to remain consistent before and after recombination by exchanging domain features and then re-encoding them; the matching loss further constrains the semantic features to generate reliable descriptors and stable keypoints.
[0029] In this embodiment, the expression for the modality classification loss is: ; in, For modal classification loss, and For image and modal tags, Represents the modality classification head. Represents cross-entropy loss, and Images and Domain characteristics; The expression for the calculation function of the reconstruction loss is: ; in, To reconstruct the loss, and Images and Shallow features, and Images and Homologous reconstruction shallow features; The expression for the calculation function of the consistency loss is: ; in, For consistency loss, and Images and semantic features and Images and Re-encoded semantic features and Images and The recoding domain features, and the recoding semantic features and the recoding domain features together constitute the recoding features; The matching loss consists of some or all of the following: descriptor loss, repeatability loss, and local peak loss. Repeatability loss is used to maintain consistency in the local blocks of the detection score map at corresponding locations. Local peak loss is used to enhance the peak response within a local region. The calculation function expression for the descriptor loss is: ; ; in, For descriptor loss, and These represent the softmax operations along the row and column directions, respectively. and Represent the confidence matrix respectively Similarity of matching descriptors on the diagonal and Confidence matrix index number, and Images and The semantic features of the superscript For transpose; The expression for the calculation function of the repeatability loss is: ; in, For repetitive loss, For image and The number of pixels in and Images and The set of coordinates of the same pixel in the detection score image obtained by the detection head. and Images and The coordinates of the i-th identical pixel in the detection score image. and They are respectively and The detection score, in superscript This is a transpose operation; The expression for the calculation function of the local peak loss is: ; in, For local peak loss, For a set of overlapping image patches Size, For image and The set of overlapping image patches in the detection score map. and These are taking the average and taking the maximum value, respectively. For the first Overlapping image patches The detection score, and For the first Overlapping image patches In the image and The sum of the detection scores in the detection score graph.
[0030] In step S101 of this embodiment, when jointly training the network model consisting of a shared shallow encoder, semantic encoder, domain encoder, and decoder using modality classification loss, reconstruction loss, consistency loss, and matching loss, the total loss function used is: ; ; in, For the total loss function, To match the loss, For descriptor loss, For repetitive loss, For local peak loss, For modal classification loss, To reconstruct the loss, This results in a loss of consistency.
[0031] In this embodiment, before inputting the image pairs of different modalities into the shared shallow encoder to extract shallow features in step S101, a random affine transformation is applied to the image pairs of different modalities. This random affine transformation includes changing part or all of the image's orientation, scale, shape, and spatial position. Considering that the image pairs in the dataset have already been registered, to enhance the model's robustness to geometric changes, this embodiment applies a random affine transformation to the image pairs, introducing geometric perturbations such as rotation and scaling, thereby changing the image's orientation, scale, shape, and spatial position while maintaining the correspondence between the image pairs. Then, the two images are input into the shared shallow encoder E to obtain... and Then enter them separately. and Semantic features and domain features are obtained. Since semantic features mainly represent cross-modal stable structural and content information, they are given priority in subsequent detection and description processes to reduce the impact of modal appearance differences on matching.
[0032] In this embodiment, step S102 includes: S201, use the shared shallow encoder in the trained network model to extract shallow features from the image pair to be matched consisting of image a and image b, and then use the shallow features to extract semantic features through the semantic encoder. S202, the semantic features of each key point in image a and image b are used as feature descriptors. Key point matching is performed based on the feature descriptors of the key points to obtain a set of key point pairs. For example, the sampled feature descriptors are matched with each other to obtain the key point matching results.
[0033] In this embodiment, after step S201 and before step S202, the method further includes: detecting the semantic features of image a and image b using the detection head in the trained network model to obtain detection score maps, and performing non-maximum suppression on the feature point positions in the detection score maps for each image in image a and image b to obtain key point positions.
[0034] In this embodiment, the unified multimodal image matching method based on unwrapped representation learning is implemented using PyTorch and trained for 160,000 steps using the Adam optimizer on a machine equipped with an Intel i9-11900K CPU and an NVIDIA GeForce RTX 3090 GPU. After training, the image pair to be matched (images a and b) is input into the trained unified multimodal image matching network to obtain the detection score map and feature descriptor map from the semantic features. Non-maximum suppression is performed on the detection score map to obtain the key point positions; corresponding descriptors are sampled in the descriptor map according to the key point positions; the similarity between the descriptors of the two images is calculated, and the key point matching result is obtained by using nearest neighbor matching. ; Where M is the set of matched keypoint pairs, and For the first pair of matching key points, and This is the second pair of matching keypoints, and so on. It should be noted that the two images in the image pair to be matched are cross-modal images obtained from different sensors, and the image pairs in the dataset have already completed the registration training phase. Random affine transformations are applied to the image pairs to introduce geometric perturbations such as rotation and scaling, thereby changing the orientation, scale, shape, and spatial position of the images, while maintaining the correspondence between the image pairs; in the inference phase, multi-scale feature extraction can be performed on the images to be matched.
[0035] To verify the performance of the unified multimodal image matching method based on unwrapped representation learning in this embodiment, experiments were conducted using the OSdataset, SEN1-2, MU-Net, Google Maps, MRSR, WHU-SEN-City, and SIRF datasets. Homography estimation accuracy was used as the evaluation metric to assess the performance of the proposed method. Homography estimation accuracy was evaluated using the average corner projection error for each image pair. Performance was reported using the area under the curve (AUC) of the cumulative corner projection error distribution curve within the 3px, 5px, and 10px thresholds, denoted as @3px, @5px, and @10px, respectively. All comparison methods used the same RANSAC algorithm to estimate the homography matrix, and the reprojection error threshold was uniformly set to 2 pixels. This evaluation metric comprehensively reflects the keypoint detection quality and the estimation performance of the homography matrix corresponding to the matching results. To verify the effectiveness of the method in this embodiment, it was compared with representative methods of three matching paradigms: semi-dense, dense, and sparse matching. For semi-dense matching, the comparison methods included LoFTR, XoFTR, and MINIMA. LoFTRAnd AMS-Former; for dense matching, RoMa and MINIMA were evaluated. RoMa For sparse matching, SuperGlue, LightGlue (LG), Redefeat, and Minima were selected. LG SGPFeat and SPEN are used for comparison. Furthermore, all detector-based methods (including the method in this embodiment) set a uniform upper limit of 5000 keypoints per image to ensure consistent comparison and yield results as shown below. Figures 3-8 As shown in Tables 1 to 3.
[0036] pass Figure 3 , Figure 4 and Figure 5 It can be seen that the method in this embodiment can effectively achieve unified multimodal image matching in optical image-SAR image, optical image-infrared image, and optical image-map scene. Figure 6 , Figure 7 and Figure 8 The method of this embodiment and the existing ReDFeat method, MINIMA are further described. LG The comparison of the visualization matching results of the method, SGPFeat method and SPEN method further verifies that the method of this embodiment can effectively achieve unified multimodal image matching in optical image-SAR image, optical image-infrared image and optical image-map scene.
[0037] Table 1: Homography estimation results of different methods on optical-SAR images
[0038] As shown in Table 1, for the sparse matching task, the method in this embodiment achieves the best and second-best performance on both the MRSR and WHU-SEN-City datasets in terms of average matching success rate (SR) and area under the homography error curve (AUC). Notably, among all the compared methods, when the error threshold is set to 3px, the method in this embodiment performs second only to the SPEN method on the WHU-SEN-City dataset, while at error thresholds of 5px and 10px, it achieves the best performance on both datasets. These results highlight that the overall performance of the method in this embodiment outperforms the domain-specific models used by the compared methods.
[0039] Table 2: Homography estimation results of different methods on optical-infrared images
[0040] As shown in Table 2, for sparse matching tasks, the method in this embodiment achieves the best performance in terms of area under the homography error curve (AUC) at error thresholds of 3px, 5px, and 10px, while also outperforming other methods in terms of average successful match rate (SR). Notably, most methods significantly outperform the optical-infrared dataset in terms of performance on the optical-SAR dataset, primarily due to the relatively smaller modal differences between optical and infrared images.
[0041] Table 3: Homography estimation results of different methods on optical-map images
[0042] As shown in Table 3, for sparse matching tasks, the method in this embodiment achieves the best performance in terms of area under the homography error curve (AUC) at error thresholds of 3px, 5px, and 10px, and also outperforms other comparative methods in terms of average successful matching rate (SR). It is worth noting that the overall performance on the optical-map dataset is significantly lower than that on the optical-SAR and optical-infrared datasets. This is mainly due to the fact that SAR and infrared images are essentially based on sensor imaging observations of real-world scenes, while map images are symbolic abstractions of the environment. The rich local textures, appearance patterns, and target boundaries in optical images typically do not have direct visual correspondences in map images. This significantly weakens the feature similarity across modalities and introduces stronger ambiguity into correspondence estimation, thus making optical-map cross-modal matching significantly more difficult.
[0043] To verify the effectiveness of the various losses in the method of this embodiment, experiments were conducted on the WHU-SEN-City dataset to evaluate their impact on performance and generalization ability, including the matching loss. Modal classification loss Reconstruction loss and consistency loss The results of the ablation experiments on the WHU-SEN-City dataset are shown in Table 4.
[0044] Table 4: Results of ablation experiments on the WHU-SEN-City dataset
[0045] As can be observed from Table 4, only matching loss is used The baseline method achieves relatively limited performance, and removing any loss function leads to a performance degradation. Specifically, the modality classification loss... The aim is to provide supervisory constraints for domain branches, achieving effective separation of modality-specific information from semantic features. When modality classification loss... When a domain branch is removed, the supervision it receives weakens, and modality-specific domain information becomes difficult to decouple from semantic features, ultimately leading to a decline in model matching performance. (Reconstruction loss) Used to preserve complete feature information and remove reconstruction loss during feature unwrapping. This leads to the loss of feature information during untangling, resulting in the largest performance degradation among all ablation terms, fully demonstrating the crucial importance of preserving feature information during untangling. Consistency loss It helps maintain stable semantics and modality-specific representations after feature refactoring, and removes consistency loss. This can disrupt the consistency of feature representations, thereby reducing matching accuracy. Overall, each loss term contributes positively to the final performance, and their joint optimization achieves optimal results across all evaluation thresholds, validating the effectiveness of each loss component in the method of this embodiment.
[0046] The experiments described above demonstrate that the unified multimodal image matching method based on unwrapped representation learning in this embodiment explicitly decomposes modality-invariant semantic features and modality-specific domain features. This allows the matching process to primarily rely on stable cross-modal structural semantic information. Stable feature unwrapping constraints are constructed using homologous reconstruction, cross-reconstruction, and re-encoding. The semantic features are then used for keypoint detection and descriptor learning, reducing the impact of appearance differences caused by different imaging mechanisms on image matching, thereby improving the robustness and generalization ability of unified multimodal image matching. Furthermore, this unified multimodal image matching method uses modality classification loss to centrally express modal appearance through domain features, shallow feature reconstruction loss to preserve the integrity of shallow image features, and cross-reconstruction consistency loss to ensure the stability of semantics and modality-specific representations after cross-modal domain information exchange. Finally, this unified multimodal image matching method jointly optimizes detection and description using descriptor loss, repeatability loss, and local peak loss, enabling the method to output reliable keypoint matching results on multimodal image pairs such as optical-SAR, optical-infrared, and optical-map.
[0047] Furthermore, this embodiment also provides a unified multimodal image matching system based on unentangled representation learning, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the unified multimodal image matching method based on unentangled representation learning.
[0048] Those skilled in the art will understand that the technical solutions provided by this invention may take the form of a method, system, or computer program product. Therefore, this invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this invention may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce an implementation of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0049] The above description is merely a preferred embodiment of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should also be considered within the scope of protection of the present invention.
Claims
1. A unified multimodal image matching method based on unentangled representation learning, characterized in that, Includes the following steps: S101, during the training phase, images of different modalities are input into a shared shallow encoder to extract shallow features. The semantic encoder and domain encoder are used to unwrap features to extract semantic and domain features. The decoder performs homologous reconstruction and cross-reconstruction on the semantic and domain features. The cross-reconstructed shallow features are re-encoded by the semantic encoder and domain encoder respectively to generate recoded features. Based on the obtained features, the network model consisting of a shared shallow encoder, semantic encoder, domain encoder and decoder is jointly trained using modality classification loss, reconstruction loss, consistency loss and matching loss. S102, In the inference stage, semantic features are extracted from the image pairs to be matched using the shared shallow encoder and semantic encoder in the trained network model. Then, the semantic features of each key point in the image pairs to be matched are used as feature descriptors. Key point matching is performed based on the feature descriptors of the key points to obtain a set of key point pairs.
2. The unified multimodal image matching method based on unentangled representation learning according to claim 1, characterized in that, The function expression for feature unwrapping in step S101 to extract semantic and domain features is as follows: ; ; ; ; in, and Images and semantic features and Images and Domain characteristics, For semantic encoders, For a domain encoder, and Images and Shallow features.
3. The unified multimodal image matching method based on unentangled representation learning according to claim 1, characterized in that, In step S101, the function expression for reconstructing semantic features and domain features from the same source and through cross-reconstruction using the decoder is as follows: ; ; ; ; in, and Images and Homologous reconstruction shallow features, For image semantic features and The shallow features obtained by concatenating the domain features are cross-reconstructed. image semantic features and images The shallow features obtained by concatenating the domain features are cross-reconstructed. For decoder, This indicates feature splicing.
4. The unified multimodal image matching method based on unentangled representation learning according to claim 1, characterized in that, In step S101, the shallow cross-reconstruction features obtained from the cross-reconstruction are re-encoded using a semantic encoder and a domain encoder, respectively, to generate recoded features. The function expression is as follows: ; ; in, and Images and Re-encoded semantic features and Images and Features of the recoding domain For semantic encoders, For a domain encoder, For image semantic features and The shallow features obtained by concatenating the domain features are cross-reconstructed. image semantic features and images The shallow cross-reconstruction features are obtained by concatenating the domain features, and the recoded semantic features and recoded domain features constitute the recoded features.
5. The unified multimodal image matching method based on unentangled representation learning according to claim 1, characterized in that, In step S101, when jointly training the network model consisting of a shared shallow encoder, semantic encoder, domain encoder, and decoder using modality classification loss, reconstruction loss, consistency loss, and matching loss, the expression for the calculation function of the modality classification loss is: ; in, For modal classification loss, and For image and modal tags, Represents the modality classification head. Represents cross-entropy loss, and Images and Domain characteristics; The expression for the calculation function of the reconstruction loss is: ; in, To reconstruct the loss, and Images and Shallow features, and Images and Homologous reconstruction shallow features; The expression for the calculation function of the consistency loss is: ; in, For consistency loss, and Images and semantic features and Images and Re-encoded semantic features and Images and The recoding domain features, and the recoding semantic features and the recoding domain features together constitute the recoding features; The matching loss consists of some or all of the descriptor loss, repeatability loss, and local peak loss. The calculation function expression for the descriptor loss is as follows: ; ; in, For descriptor loss, and These represent the softmax operations along the row and column directions, respectively. and Represent the confidence matrix respectively Similarity of matching descriptors on the diagonal and Confidence matrix index number, and Images and The semantic features of the superscript For transpose; The expression for the calculation function of the repeatability loss is: ; in, For repetitive loss, For image and The number of pixels in and Images and The set of coordinates of the same pixel in the detection score image obtained by the detection head. and Images and The coordinates of the i-th identical pixel in the detection score image. and They are respectively and The detection score, in superscript This is a transpose operation; The expression for the calculation function of the local peak loss is: ; in, For local peak loss, For a set of overlapping image patches Size, For image and The set of overlapping image patches in the detection score map. and These are taking the average and taking the maximum value, respectively. For the first Overlapping image patches The detection score, and For the first Overlapping image patches In the image and The sum of the detection scores in the detection score graph.
6. The unified multimodal image matching method based on unentangled representation learning according to claim 5, characterized in that, In step S101, when jointly training the network model consisting of a shared shallow encoder, semantic encoder, domain encoder, and decoder using modality classification loss, reconstruction loss, consistency loss, and matching loss, the total loss function used is: ; ; in, For the total loss function, To match the loss, For descriptor loss, For repetitive loss, For local peak loss, For modal classification loss, To reconstruct the loss, This results in a loss of consistency.
7. The unified multimodal image matching method based on unentangled representation learning according to claim 1, characterized in that, Before extracting shallow features from images of different modalities by inputting them into a shared shallow encoder in step S101, the method further includes applying random affine transformations to the images of different modalities in the samples. These random affine transformations include changing some or all of the orientation, scale, shape, and spatial position of the images.
8. The unified multimodal image matching method based on unentangled representation learning according to claim 1, characterized in that, Step S102 includes: S201, use the shared shallow encoder in the trained network model to extract shallow features from the image pair to be matched consisting of image a and image b, and then use the shallow features to extract semantic features through the semantic encoder. S202, the semantic features of each key point in image a and image b are used as feature descriptors, and key point matching is performed based on the feature descriptors of the key points to obtain a set of key point pairs.
9. The unified multimodal image matching method based on unentangled representation learning according to claim 8, characterized in that, After step S201 and before step S202, the method further includes: using the detection head in the trained network model to detect the semantic features of image a and image b respectively to obtain detection score maps; and performing non-maximum suppression on the feature point positions in the detection score maps for each image in image a and image b to obtain key point positions.
10. A unified multimodal image matching system based on unentangled representation learning, comprising interconnected microprocessors and a memory, characterized in that, The microprocessor is programmed or configured to execute the unified multimodal image matching method based on unwrapped representation learning as described in any one of claims 1 to 9.
