A multimodal image matching method and system based on learned features and epipolar geometric constraints

Abstract

This invention relates to a multimodal image matching method and system based on learned features and epipolar geometric constraints. The method includes edge enhancement processing of the input image using wavelet transform; extraction of multi-scale dense feature maps based on a modified convolutional neural network, combined with principal direction normalization to generate feature descriptors with rotation and scale invariance; preliminary feature matching using the FLANN algorithm and dynamic distance constraints; and the introduction of a fundamental matrix construction and epipolar geometric consistency verification mechanism, combined with a RANSAC affine constraint model to eliminate mismatched point pairs. This invention integrates image enhancement, deep learning, and geometric verification strategies, effectively addressing the radiation nonlinearity and geometric distortion problems caused by differences in imaging mechanisms between multimodal images, improving matching accuracy and robustness. It is suitable for remote sensing applications such as optical-SAR registration, multi-source image fusion, and land surface change detection.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a multimodal image matching method and system based on learned features and epipolar geometric constraints. Background Technology

[0002] Multimodal remote sensing imagery, as a crucial component of remote sensing imagery, plays a vital role in fields such as land surveys, environmental monitoring, and high-precision mapping. Existing multimodal remote sensing imagery is mainly divided into passive optical imagery and active optical imagery. Passive optical imagery relies on passive sensing systems to collect electromagnetic wave information reflected from ground objects in the visible and near-infrared bands, accurately characterizing the color attributes and texture features of targets and intuitively presenting detailed surface features. Active optical imagery constructs images by actively emitting electromagnetic pulses in the infrared, visible, or ultraviolet bands, offering unique advantages in long-distance ground object measurement and regional coverage. Collaborative processing and application of these multi-source data can improve the accuracy of land cover mapping, the reliability of change detection, and the effectiveness of data fusion.

[0003] Existing multimodal remote sensing image matching techniques mainly employ methods based on hand-designed approaches and those based on learned features. Hand-designed multimodal image matching techniques rely heavily on researchers' expertise to design feature detectors, descriptors, and matching strategies for efficient matching. Learned-based methods, on the other hand, leverage the data-driven nature of deep learning, utilizing multi-level nonlinear transformations to autonomously extract features from low-level visual data to high-level semantic data, achieving better matching results. Each of these methods has its limitations in implementation. Hand-designed methods struggle to construct accurate feature detection and description models that represent common features across multimodal images when dealing with significant geometric deformations and radiometric differences. Learned-feature-based methods exhibit significant performance limitations when processing large-scale affine transformation images, and due to the scarcity of publicly available training datasets, their generalization capabilities are insufficient for cross-regional applications.

[0004] Regarding the implementation techniques in the field of multimodal remote sensing image matching, existing methods still have unresolved issues. First, the matching performance of multimodal images is limited by different imaging modes, making it very difficult to obtain accurate corresponding feature points and achieve stable, invariant feature extraction. Second, existing methods have poor adaptability when facing large-scale geometric distortions and radiometric differences, making it difficult to suppress differences between modes. Third, the lack of ground truth references between multimodal images makes it difficult to guarantee the geometric reliability of the matching results. Fourth, the lack of publicly available multimodal image training datasets limits the effectiveness of feature representation and model performance. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multimodal image matching method and system based on learned features and epipolar geometric constraints, aiming to solve the technical problems of invariant feature extraction, modal difference suppression, geometric correspondence matching, and lack of public datasets in the field of multimodal remote sensing image matching.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a multimodal image matching method based on learned features and epipolar geometric constraints, comprising the following steps:

[0007] S1. Preprocess cross-source multimodal remote sensing image pairs from different sources to obtain the original image pairs;

[0008] S2. Perform Haar wavelet transform decomposition and weighting on the original image pairs respectively, and then perform weighted reconstruction through inverse wavelet transform to obtain an enhanced image containing multi-scale detailed structure, which strengthens structural details and improves the stability and discriminability of feature points in weak texture regions.

[0009] S3. Input the enhanced image containing multi-scale detailed structure into the improved VGG-16 convolutional neural network for feature extraction, construct a local orientation histogram to estimate the main orientation of the feature points, and perform orientation normalization on the feature descriptor of each feature point; during the feature extraction process, combine the multi-scale feature fusion strategy to generate a multi-scale dense feature map, and then obtain a feature descriptor with rotation and scale invariance.

[0010] S4. The FLANN algorithm is used to perform initial matching of feature points and construct difference vectors. A dynamic threshold constraint discrimination mechanism is designed based on cosine similarity to realize dynamic adaptive distance constraint, eliminate false matching point pairs, and introduce a one-way uniqueness constraint rule to retain only the optimal matching result of each feature point in the source image in the target image, thus forming a preliminary feature point matching pair set.

[0011] S5. Construct a basic matrix based on the initial set of feature point matching pairs, calculate the epipolar error of the initial feature point matching pairs, and judge their geometric consistency; construct a RANSAC affine constraint model, combine affine constraints and RANSAC algorithm to perform interior point estimation and model optimization, eliminate erroneous matching point pairs, and finally obtain a stable and reliable high-quality feature matching point set.

[0012] Furthermore, in step S2, the specific process includes the following steps:

[0013] S21. Perform two-level Daubechies 2 wavelet transform decomposition on the original image pairs to obtain the second-level low-frequency subband coefficients and the high-frequency subband coefficients of each level.

[0014] S22. The low-frequency and high-frequency sub-band coefficients are weighted based on the enhancement weighting factor to simultaneously improve the overall contrast of the image and the ability to express local edge texture details.

[0015] S23. Keeping the high-frequency subband coefficients of the first layer unchanged, perform inverse wavelet transform on the weighted low-frequency and high-frequency subband coefficients of the second layer to reconstruct an enhanced image containing multi-scale detailed structures, which serves as the input basis for subsequent feature extraction, thereby improving the stability and distribution balance of feature points in weak texture regions.

[0016] Furthermore, the enhanced weighting factor is dynamically set according to the energy characteristics of different sub-bands after Haar wavelet transform decomposition, prioritizing the retention of alignment contours and directional structural features to improve the structural consistency matching capability between cross-source multimodal remote sensing image pairs.

[0017] Furthermore, in step S3, the specific process includes the following steps:

[0018] S31. A VGG-16 convolutional neural network with the last fully connected layer removed is used as the feature extraction backbone network, while its first four convolutional modules are retained to extract feature maps and obtain feature points.

[0019] S32. Construct a local orientation histogram based on the neighborhood gradient direction of the feature point, estimate the main orientation of the feature point, and rotate and align the feature descriptor of the feature point with the main orientation as the reference to complete the orientation normalization of the feature descriptor.

[0020] S33. The multi-scale feature fusion strategy constructs a multi-scale image pyramid model, which consists of L levels, where the first level represents the original image, and each subsequent level is a scaling layer of the previous level, with a scaling factor of . The last layer is the initial image. times.

[0021] Furthermore, in step S3, the feature extraction backbone network performs L2 normalization on the feature maps output by each layer to ensure that the features have a uniform amplitude and orientation distribution at different scales.

[0022] Furthermore, step S4 specifically includes the following steps:

[0023] S41. Using the feature descriptors obtained in step S3 as input for initial matching, the FLANN algorithm is used to perform nearest neighbor search on the feature descriptors to obtain initial matching point pairs.

[0024] S42. Construct the difference vector between the initial matching point pairs based on the feature descriptors, and calculate the cosine similarity based on the direction and distance distribution of the difference vector;

[0025] S43. Design a dynamic threshold constraint discrimination mechanism based on the cosine similarity between initial matching point pairs, and set a threshold to filter out pseudo-matching point pairs with low consistency scores.

[0026] S44. Introduce a one-way uniqueness constraint rule. For each feature point in the source image, only retain the unique matching result with the highest cosine similarity in the target image. When multiple source feature points correspond to the same target feature point, retain the nearest neighbor matching pair according to the principle of minimizing the Euclidean distance of the feature descriptor, and remove the other matching pairs to form a preliminary feature point matching pair set.

[0027] Furthermore, in step S43, the dynamic threshold constraint discrimination mechanism defines a consistency score between each pair of initial matching points, the mathematical expression of which is:

[0028] ;

[0029] in Cosine similarity is represented by the initial matching point pairs. and Transform the differences into a difference vector , A quantitative evaluation of its consistency is conducted, describing the degree of consistency in the differences of feature descriptors between any two initial matching point pairs i and k, where... , , , Representing feature points respectively , , , The corresponding feature descriptor.

[0030] Furthermore, step S5 specifically includes the following steps:

[0031] S51. Construct the basic matrix between the source image and the target image based on the preliminary feature point matching pairs. F Solving the fundamental matrix using the eight-point method F To obtain the epipolar geometric relationship between preliminary feature point matching pairs;

[0032] S52, Based on the fundamental matrix F Calculate the geometric distance between each pair of preliminary feature point matches and its corresponding epipolar line, and use the Sampson error as the epipolar error to evaluate the geometric consistency of the preliminary feature point matches.

[0033] S53. Model the local spatial geometry between preliminary feature point matching pairs based on affine transformation;

[0034] S54. Optimize the model constructed in step S53 using the RANSAC algorithm and perform iterative internal point estimation. Eliminate incorrect matching point pairs with excessive epipolar errors or those that do not conform to affine constraints, and output a high-quality feature matching point set that passes the geometric consistency check.

[0035] Further, in step S5, the mathematical form of the RANSAC affine constraint model is:

[0036] ;

[0037] Where a, b, c, and d represent linear transformation parameters, and e and f are translation components. The above six parameters of the RANSAC affine constraint model are estimated by randomly sampling the smallest sample set from the initial feature point matching pair set. The residuals of the remaining matching pairs are calculated and a threshold is set to determine whether they belong to the inlier set of the current model. Feature point matching pairs with high consistency are selected to form a high-quality feature matching point set output by the model, which serves as the basis for cross-source multimodal remote sensing image registration.

[0038] This application also provides a multimodal image matching system based on learned features and epipolar geometric constraints, used to apply the above-mentioned multimodal image matching method based on learned features and epipolar geometric constraints, including:

[0039] It consists of four parts: image acquisition subsystem, feature extraction and matching subsystem, geometric verification subsystem, and matching output subsystem;

[0040] The image acquisition subsystem includes a cross-source multimodal image acquisition device, a data interface module, and an image preprocessing module, used to acquire and import cross-source multimodal remote sensing image pairs from different sensors, and to perform scale unification, format conversion, and grayscale normalization processing on the images.

[0041] The feature extraction and matching subsystem includes an image enhancement module, a deep feature extraction module, a feature point filtering module, and an initial matching module: the image enhancement module performs Haar wavelet transform decomposition on the input image and extracts high-frequency subband features, and enhances edge texture information by weighted fusion of the high-frequency subband features; the deep feature extraction module extracts multi-scale dense feature maps based on convolutional neural networks; the feature point filtering module constructs feature descriptors with rotation and scale invariance by combining a direction normalization mechanism; the initial matching module constructs a preliminary set of feature point matching pairs by combining the FLANN algorithm with a dynamic threshold constraint discrimination mechanism and a unidirectional uniqueness constraint rule;

[0042] The geometric verification subsystem includes a epipolar geometric consistency module and a RANSAC robust model estimation module. It uses the fundamental matrix to construct epipolar geometric relations, calculates the Sampson error, and combines the RANSAC affine constraint model to filter and optimize feature point matching pairs.

[0043] The matching output subsystem includes a matching visualization unit and an evaluation module, which are used to output the final registration results, draw matching connection diagrams, and calculate various evaluation indicators, supporting the quantitative analysis and comparative verification of cross-source multimodal remote sensing image registration results.

[0044] By employing the above technical solution, the present invention provides a multimodal image matching method based on learned features and epipolar geometric constraints, which has at least the following beneficial effects:

[0045] (1) This invention effectively enhances the ability to express image edges and texture details through wavelet enhancement and main direction normalization. Combined with the multi-scale feature learning mechanism of convolutional neural network, the matching accuracy is improved by more than 20% compared with traditional methods. Compared with existing multimodal remote sensing image matching technology, it significantly improves the accuracy of cross-modal matching.

[0046] (2) By integrating the fundamental matrix constraint and the affine RANSAC elimination mechanism, this invention significantly reduces the proportion of false matches, ensures that the final matching point set has geometric consistency, and can maintain high stability even in scenarios with large-scale affine transformation and radiation differences, thereby improving the stability and robustness of the matching process.

[0047] (3) The algorithm of this invention does not rely on the same source data or specific area samples. It is applicable to a variety of remote sensing image combinations such as SAR-optical and infrared-visible light. It can be widely adapted to a variety of remote sensing application scenarios such as land use classification, disaster monitoring, and target recognition. It has good cross-scenario and cross-regional generalization ability.

[0048] (4) This invention constructs a standardized multimodal matching process and evaluation system, and comprehensively evaluates the matching effect through multiple indicators such as NCM, SR, and RMSE, providing an integrable, scalable, and highly reliable basic module for multimodal image processing systems. Attached Figure Description

[0049] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0050] Figure 1 This is the workflow of the multimodal image matching method based on learned features and epipolar geometric constraints proposed in this invention;

[0051] Figure 2 This is a schematic diagram comparing the wavelet enhancement effects in embodiments of the present invention;

[0052] Figure 3 This is a schematic diagram of the feature extraction network structure in an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of feature extraction based on multimodal remote sensing images in an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of the optical-SAR image matching results in an embodiment of the present invention. Detailed Implementation

[0055] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This will allow for a full understanding of how the present application uses technical means to solve technical problems and achieve technical effects, and to facilitate its implementation.

[0056] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0057] Please refer to Figures 1-5 This illustration shows a specific implementation of the present embodiment. By fusing image enhancement, deep learning, and epipolar geometry verification strategies, this embodiment effectively solves the problems of radiation nonlinearity and geometric distortion caused by differences in imaging mechanisms between multimodal images, improves matching accuracy and robustness, and is suitable for remote sensing application scenarios such as accurate matching between visible light images and multimodal remote sensing images such as SAR images and thermal infrared images, multi-source image fusion, and surface change detection.

[0058] Please refer to Figure 1 This embodiment proposes a multimodal image matching method based on learned features and epipolar geometric constraints. The method includes the following steps:

[0059] S1. Preprocess cross-source multimodal remote sensing image pairs from different sources (taking visible light-SAR image as an example) to obtain the original image pairs;

[0060] As a preferred implementation of step S1, the specific process includes: acquiring multimodal remote sensing image pairs with spatially overlapping regions, the image sources including passive optical remote sensing images and active synthetic aperture radar (SAR) images, and uniformly performing grayscale normalization, resolution adjustment and format standardization processing on the differences between them in spectral characteristics and structural information, so as to provide basic input for subsequent feature extraction and matching.

[0061] S2. Perform Haar wavelet transform decomposition and weighting on the original image pairs respectively, and then perform weighted reconstruction through inverse wavelet transform to obtain an enhanced image containing multi-scale detailed structure, which strengthens structural details and improves the stability and discriminability of feature points in weak texture regions.

[0062] As a preferred embodiment of step S2, the specific process includes the following steps:

[0063] S21. The multimodal remote sensing images processed in this embodiment cover SAR and optical images with 10-meter and 1-meter resolutions, exhibiting significant differences in structure, texture, and radiometric properties. To alleviate these modal differences, wavelet transform enhancement is first applied to both the optical and SAR images to enhance their local structure and edge features. In wavelet enhancement, two layers of Daubechies 2 (db2) wavelet transform decomposition are performed on the original image pairs to obtain the second-layer low-frequency subband coefficients and the high-frequency subband coefficients of each layer, enhancing detail information and suppressing noise. The wavelet enhancement effect is illustrated below. Figure 2 As shown;

[0064] S22. The low-frequency and high-frequency sub-band coefficients obtained from the Daubechies 2 (db2) wavelet transform decomposition are weighted based on the enhancement weighting factor to simultaneously improve the overall image contrast and local edge texture expression capability; the enhancement weighting factor includes the low-frequency enhancement coefficient. With high frequency enhancement factor The weighted processing mathematical formula is as follows: This formula is used to control the enhancement intensity of the low-frequency and high-frequency sub-bands respectively.

[0065] ;

[0066] in Indicates the first The high-frequency sub-band coefficients of the layer represent the detail information of the image in the horizontal, vertical and diagonal directions, respectively. Indicates the first The low-frequency subband coefficient of the layer characterizes the overall brightness and large-scale structural information of the image; Indicates the weighted processing of the first... Low-frequency and high-frequency subband coefficients;

[0067] S23. The enhanced image is reconstructed through inverse wavelet transform, and its mathematical formula is:

[0068] ;

[0069] in, To enhance the image, the first layer of high-frequency subband coefficients are used during the reconstruction process. No weighting is applied;

[0070] More specifically, the weighting factor is dynamically set based on the energy characteristics of different sub-bands after Haar wavelet transform decomposition, prioritizing the preservation of alignment contours and directional structural features to improve the structural consistency matching capability between cross-source multimodal remote sensing image pairs.

[0071] In this embodiment, by enhancing the high-frequency components and reconstructing the image, the structural clarity and boundary distinguishability of the image can be improved without introducing artifacts, thereby enhancing the adaptability of subsequent feature descriptors among heterogeneous images.

[0072] S3. Input the enhanced image containing multi-scale detailed structures into the improved VGG-16 convolutional neural network, extract dense feature maps, construct local orientation histograms to estimate the main orientation of feature points, and perform orientation normalization on the feature descriptors of each feature point; combine multi-scale feature fusion strategy during feature extraction to generate multi-scale dense feature maps, and then obtain feature descriptors with rotation and scale invariance.

[0073] As a preferred embodiment of step S3, the specific process includes the following steps:

[0074] S31. A multi-scale convolutional feature extraction network with an improved classic VGG-16 structure is used, combined with a multi-scale feature fusion strategy, to extract semantic features of the image from convolutional layers with different receptive fields, and to detect local extrema as feature points in the multi-scale dense feature map. The feature extraction network structure is as follows: Figure 3 As shown.

[0075] The extracted features include multi-scale dense feature maps and their corresponding spatial location, principal orientation information, and 128-dimensional descriptor feature vectors. Feature extraction based on multimodal remote sensing images, such as... Figure 4 As shown.

[0076] To ensure the uniform distribution of feature points on the image, this embodiment adopts the "Top-K response + uniform grid sampling" strategy, which prioritizes the retention of the K key points with the largest response intensity as feature points, while uniformly sampling within the grid-divided area on the image to balance information representation capability and spatial coverage.

[0077] S32. Construct a local orientation histogram based on the neighborhood gradient direction of the feature point, estimate the main orientation of the feature point, and rotate and align the feature descriptor of the feature point with the main orientation as the reference to complete the orientation normalization of the feature descriptor.

[0078] S33. The multi-scale feature fusion strategy constructs a multi-scale image pyramid model, which consists of L levels, where the first level represents the original image, and each subsequent level is a scaling layer of the previous level, with a scaling factor of . The last layer is the initial image. The mathematical expression for "times" is:

[0079] ;

[0080] in, , These represent feature maps at different levels. This represents the fused multi-scale dense feature map.

[0081] More specifically, to enhance the consistency of features at different scales, this embodiment introduces a feature normalization strategy to perform L2 normalization on dense feature maps at each scale, so as to eliminate the influence of numerical differences between scales on the expressive power of descriptors and ensure that features have a uniform amplitude and orientation distribution at different scales.

[0082] In this embodiment, a convolutional neural network is used to extract multi-scale deep features, capturing semantic and textural information at different scales through dense feature maps output by specific layers. During feature extraction, a principal orientation estimation mechanism is introduced. The principal orientation of each feature point (i.e., the principal orientation of each feature region) is calculated statistically using local gradient directions, and then the feature descriptors are rotated and aligned to achieve orientation normalization. This strategy effectively improves the rotation and scale invariance of feature descriptors and enhances matching stability under multimodal conditions.

[0083] S4. The FLANN algorithm is used to perform initial matching of feature points and construct difference vectors. A dynamic threshold constraint discrimination mechanism is designed based on cosine similarity to realize dynamic adaptive distance constraint, eliminate false matching point pairs, and introduce a one-way uniqueness constraint rule to retain only the optimal matching result of each feature point in the source image in the target image, thus forming a preliminary feature point matching pair set.

[0084] As a preferred embodiment of step S4, the specific process includes the following steps:

[0085] S41. Using the feature descriptors obtained in step S3 as input for initial matching, the FLANN algorithm is used to perform nearest neighbor search on the feature descriptors; specifically, for the source image Each feature point Corresponding feature descriptor Nearest neighbor search in the target image Search for its nearest neighbor in a subset of feature descriptors Construct initial matching point pairs;

[0086] S42. Construct the difference vector between the initial matching point pairs based on the feature descriptors, and calculate the cosine similarity based on the direction and distance distribution of the difference vector;

[0087] S43. Design a dynamic threshold constraint discrimination mechanism based on the cosine similarity between initial matching point pairs, and set a threshold to filter out pseudo-matching point pairs with low consistency scores.

[0088] Specifically, the dynamic threshold constraint discrimination mechanism defines a consistency score between each pair of initial matching points, and its mathematical expression is:

[0089] ;

[0090] in Cosine similarity is represented by the initial matching point pairs. and Transform the differences into a difference vector , A quantitative evaluation of its consistency is conducted, describing the degree of consistency in the differences of feature descriptors between any two initial matching point pairs i and k, where... , , , Representing feature points respectively , , , The corresponding feature descriptor.

[0091] When the consistency score between the initial matching point pairs is lower than the preset threshold τ, the system will automatically identify it as an outlier and filter it. This method reduces the interference of feature descriptor offset caused by modal differences on the matching accuracy and effectively improves the robustness of feature matching at the global geometric structure level.

[0092] S44. Introduce a one-way uniqueness constraint rule, which retains only the unique matching result with the best cosine similarity in the target image for each feature point in the source image, thus forming a preliminary set of feature point matching pairs.

[0093] In this embodiment, the FLANN fast approximate nearest neighbor search algorithm is used to perform pairwise matching of the target feature descriptors. Simultaneously, a dynamic threshold constraint discrimination mechanism based on cosine similarity and consistency scores is introduced to dynamically adjust the threshold to adapt to the differences in feature distribution between different image pairs, thereby eliminating a large number of low-quality pseudo-matches. Furthermore, a one-way uniqueness constraint is used to retain the unique matching pair in the target image that has the highest similarity to each feature point in the source image, further improving the accuracy of the matching pairs.

[0094] S5. Construct a basic matrix based on the initial set of feature point matching pairs, calculate the epipolar error of the initial feature point matching pairs, and judge their geometric consistency; construct a RANSAC affine constraint model, combine affine constraints and RANSAC algorithm to perform interior point estimation and model optimization, eliminate erroneous matching point pairs, and finally obtain a stable and reliable high-quality feature matching point set.

[0095] As a preferred embodiment of step S5, the specific process includes the following steps:

[0096] S51. Construct the basic matrix between the source image and the target image based on the preliminary feature point matching pairs. F Solving the fundamental matrix using the eight-point method F To obtain the epipolar geometric relationship between preliminary feature point matching pairs;

[0097] S52, Based on the fundamental matrix F The geometric distance between each initial feature point matching pair and its corresponding epipolar line is calculated, and the Sampson error is used as the epipolar error to evaluate the geometric consistency of the initial feature point matching pairs; the mathematical expression for the Sampson error is:

[0098] ;

[0099] Among them, the fundamental matrix F Based on the relative projection relationship between image pairs The solution was obtained. as well as These represent matching points in optical and SAR images, respectively. By quantifying this deviation index, matching points that do not meet the constraint are eliminated.

[0100] S53. Model the local spatial geometry between preliminary feature point matching pairs based on affine transformation;

[0101] S54. Optimize the model constructed in step S53 using the RANSAC algorithm and perform iterative internal point estimation. Remove incorrectly matched point pairs with excessive epipolar errors or those that do not conform to affine constraints. The mathematical form of the constructed RANSAC affine constraint model is as follows:

[0102] ;

[0103] Where a, b, c, and d represent linear transformation parameters, and e and f are translation components. The above six parameters of the RANSAC affine constraint model are estimated by randomly sampling the smallest sample set from the initial feature point matching pair set. The residuals of the remaining matching pairs are calculated and a threshold is set to determine whether they belong to the inlier set of the current model. Feature point matching pairs with high consistency are selected to form a high-quality feature matching point set output by the model, which serves as the basis for cross-source multimodal remote sensing image registration.

[0104] In this embodiment, to address the potential geometric errors and spurious correspondences that may remain in the initial matching process, this invention proposes an epipolar geometric constraint mechanism. Based on the initial feature point matching pairs, the fundamental matrix is ​​calculated, and the Sampson error between the point and its corresponding epipolar line is used as the geometric consistency criterion. Furthermore, a RANSAC affine constraint model is employed for robust fitting and error point removal, ultimately obtaining a high-precision feature matching point set that satisfies the geometric consistency requirements in both spatial location and structure, achieving stable registration between multimodal images.

[0105] This embodiment also provides a multimodal remote sensing image matching system based on learned features and epipolar geometric constraints, including:

[0106] It consists of four parts: image acquisition subsystem, feature extraction and matching subsystem, geometric verification subsystem, and matching output subsystem;

[0107] The image acquisition subsystem includes optical image acquisition equipment and SAR image acquisition device, data interface module and image preprocessing module. It acquires and imports cross-source multimodal remote sensing image pairs with spatial texture details and structural intensity information, and performs scale unification, format conversion and grayscale normalization on the images to provide a data foundation for subsequent image matching.

[0108] It should be noted that optical images in the image acquisition subsystem are generally multispectral or high-resolution visible light images, possessing excellent texture and shape information representation capabilities; SAR images, on the other hand, employ side-looking radar imaging, possessing cloud penetration and all-weather acquisition capabilities, and can provide intensity information and ground feature outlines, making them suitable for remote sensing applications in complex terrain and climate regions. The two types of images exhibit varying degrees of modal differences, including radiometric response, imaging geometry, and scale transformation.

[0109] The feature extraction and matching subsystem includes an image enhancement module, a deep feature extraction module, a feature point filtering module, and an initial matching module. These modules respectively perform detail enhancement of multimodal images, multi-scale feature extraction based on CNN, principal orientation estimation and orientation normalization, and initial feature point matching pair filtering using the FLANN algorithm combined with a dynamic threshold constraint discrimination mechanism and a unidirectional uniqueness constraint rule.

[0110] It should be noted that: the image enhancement module uses a wavelet multi-level reconstruction method to enhance the edges and textures of the optical image and SAR image respectively, in order to improve the consistency of feature extraction; the depth feature extraction module uses an improved VGG-16 network as the backbone and selects the activation feature maps of the intermediate convolutional layers in the multi-scale output; the extracted feature points are normalized after the principal direction is estimated to obtain rotation- and scale-invariant descriptors; in the feature point selection module, the Top-K response value selection method and the uniform grid hybrid sampling method are used to ensure spatial distribution; the initial matching module uses the FLANN fast approximate nearest neighbor algorithm and combines it with a dynamic threshold constraint discrimination mechanism to initially remove erroneous point pairs.

[0111] The geometric verification subsystem includes a polar geometric consistency module and a RANSAC robust model estimation module, which are used to perform geometric consistency verification and error elimination on the initial matching point pairs to obtain the final high-confidence matching results.

[0112] It should be noted that: the epipolar geometric consistency module verifies whether the preliminary feature point matching pairs satisfy the epipolar geometric constraint relationship between the source image and the target image by calculating the fundamental matrix, and removes matching pairs that obviously violate geometric consistency; the RANSAC robust model estimation module fits the affine model and removes outliers using RMSE as the error index, thereby improving the matching geometric accuracy and stability.

[0113] The matching output subsystem includes a matching visualization unit and an evaluation module, which are used to output the final registration results, draw the matching connection diagram, and calculate evaluation indicators such as the number of in-matching points (NCM), matching accuracy (SR), root mean square error (RMSE), and matching time (MT), supporting the quantitative analysis and comparative verification of cross-source multimodal remote sensing image registration results. Figure 5 The image shows the matching effect of this embodiment on optical-SAR images, wherein... Figure 5 (a) shows the validation results on the WHU-SEN-City dataset. Figure 5 (b) shows the results of validation on the OSdataset dataset. Figure 5 (c) in the figure is the result of validation on the SEN1-2 dataset.

[0114] The system provided in this embodiment has the following advantages:

[0115] It achieves highly robust registration of multimodal remote sensing images under complex radiometric differences and geometric distortions; it comprehensively utilizes the feature representation capabilities of deep learning and traditional geometric consistency constraints to balance expressiveness and accuracy; the evaluation mechanism is complete and supports batch processing and quantitative output, which facilitates subsequent map production and monitoring analysis.

[0116] This system provides an effective means for high-precision collaborative processing of multimodal remote sensing data. It is applicable to various scenarios such as land change monitoring, urban modeling, and disaster response, and has practical application significance for improving the level of intelligent interpretation and cross-modal integration of remote sensing images.

[0117] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0118] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0119] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multimodal image matching method based on learned features and epipolar geometric constraints, characterized in that, Includes the following steps: S1. Preprocess cross-source multimodal remote sensing image pairs from different sources to obtain the original image pairs; S2. Perform Haar wavelet transform decomposition and weighting on the original image pairs respectively, and then perform weighted reconstruction through inverse wavelet transform to obtain an enhanced image containing multi-scale detailed structure, which strengthens structural details and improves the stability and discriminability of feature points in weak texture regions. S3. Input the enhanced image containing multi-scale detailed structure into the improved VGG-16 convolutional neural network for feature extraction, construct a local orientation histogram to estimate the main orientation of the feature points, and perform orientation normalization on the feature descriptor of each feature point; during the feature extraction process, combine the multi-scale feature fusion strategy to generate a multi-scale dense feature map, and then obtain a feature descriptor with rotation and scale invariance. S4. The FLANN algorithm is used to perform initial matching of feature points and construct difference vectors. A dynamic threshold constraint discrimination mechanism is designed based on cosine similarity to realize dynamic adaptive distance constraint, eliminate false matching point pairs, and introduce a one-way uniqueness constraint rule to retain only the optimal matching result of each feature point in the source image in the target image, thus forming a preliminary feature point matching pair set. S5. Construct a basic matrix based on the initial set of feature point matching pairs, calculate the epipolar error of the initial feature point matching pairs, and judge their geometric consistency; construct a RANSAC affine constraint model, combine affine constraints and RANSAC algorithm to perform interior point estimation and model optimization, eliminate erroneous matching point pairs, and finally obtain a stable and reliable high-quality feature matching point set.

2. The multimodal image matching method based on learned features and epipolar geometric constraints according to claim 1, characterized in that: Step S2 specifically includes the following steps: S21. Perform two-level Daubechies 2 wavelet transform decomposition on the original image pairs to obtain the second-level low-frequency subband coefficients and the high-frequency subband coefficients of each level. S22. The low-frequency and high-frequency sub-band coefficients are weighted based on the enhancement weighting factor to simultaneously improve the overall contrast of the image and the ability to express local edge texture details. S23. Keeping the high-frequency subband coefficients of the first layer unchanged, perform inverse wavelet transform on the weighted low-frequency and high-frequency subband coefficients of the second layer to reconstruct an enhanced image containing multi-scale detailed structures, which serves as the input basis for subsequent feature extraction, thereby improving the stability and distribution balance of feature points in weak texture regions.

3. The multimodal image matching method based on learned features and epipolar geometric constraints according to claim 2, characterized in that: The enhancement weighting factor is dynamically set according to the energy characteristics of different sub-bands after Haar wavelet transform decomposition, and prioritizes the preservation of alignment contour and directional structural features to improve the structural consistency matching capability between cross-source multimodal remote sensing image pairs.

4. The multimodal image matching method based on learned features and epipolar geometric constraints according to claim 1, characterized in that: Step S3 specifically includes the following steps: S31. A VGG-16 convolutional neural network with the last fully connected layer removed is used as the feature extraction backbone network, while its first four convolutional modules are retained to extract feature maps and obtain feature points. S32. Construct a local orientation histogram based on the neighborhood gradient direction of the feature point, estimate the main orientation of the feature point, and rotate and align the feature descriptor of the feature point with the main orientation as the reference to complete the orientation normalization of the feature descriptor. S33. The multi-scale feature fusion strategy constructs a multi-scale image pyramid model, which consists of L levels, where the first level represents the original image, and each subsequent level is a scaling layer of the previous level, with a scaling factor of . The last layer is the initial image. times.

5. The multimodal image matching method based on learned features and epipolar geometric constraints according to claim 4, characterized in that: In step S3, the feature extraction backbone network performs L2 normalization on the feature maps output by each layer to ensure that the features have a uniform amplitude and orientation distribution at different scales.

6. The multimodal image matching method based on learned features and epipolar geometric constraints according to claim 1, characterized in that: Step S4 specifically includes the following steps: S41. Using the feature descriptors obtained in step S3 as input for initial matching, the FLANN algorithm is used to perform nearest neighbor search on the feature descriptors to obtain initial matching point pairs. S42. Construct the difference vector between the initial matching point pairs based on the feature descriptors, and calculate the cosine similarity based on the direction and distance distribution of the difference vector; S43. Design a dynamic threshold constraint discrimination mechanism based on the cosine similarity between initial matching point pairs, and set a threshold to filter out pseudo-matching point pairs with low consistency scores. S44. Introduce a one-way uniqueness constraint rule. For each feature point in the source image, only retain the unique matching result with the highest cosine similarity in the target image. When multiple source feature points correspond to the same target feature point, retain the nearest neighbor matching pair according to the principle of minimizing the Euclidean distance of the feature descriptor, and remove the other matching pairs to form a preliminary feature point matching pair set.

7. The multimodal image matching method based on learned features and epipolar geometric constraints according to claim 6, characterized in that: In step S43, the dynamic threshold constraint discrimination mechanism defines a consistency score between each pair of initial matching points, and its mathematical expression is: ； in Cosine similarity is represented by the initial matching point pairs. and Transform the differences into a difference vector , A quantitative evaluation of its consistency is conducted, describing the degree of consistency in the differences of feature descriptors between any two initial matching point pairs i and k, where... , , , Representing feature points respectively , , , The corresponding feature descriptor.

8. The multimodal image matching method based on learned features and epipolar geometric constraints according to claim 1, characterized in that: Step S5 specifically includes the following steps: S51. Construct the basic matrix between the source image and the target image based on the preliminary feature point matching pairs. F Solving the fundamental matrix using the eight-point method F To obtain the epipolar geometric relationship between preliminary feature point matching pairs; S52, Based on the fundamental matrix F Calculate the geometric distance between each pair of preliminary feature point matches and its corresponding epipolar line, and use the Sampson error as the epipolar error to evaluate the geometric consistency of the preliminary feature point matches. S53. Model the local spatial geometry between preliminary feature point matching pairs based on affine transformation; S54. Optimize the model constructed in step S53 using the RANSAC algorithm and perform iterative internal point estimation. Eliminate incorrect matching point pairs with excessive epipolar errors or those that do not conform to affine constraints, and output a high-quality feature matching point set that passes the geometric consistency check.

9. The multimodal image matching method based on learned features and epipolar geometric constraints according to claim 8, characterized in that: In step S5, the mathematical form of the RANSAC affine constraint model is: ； Where a, b, c, and d represent linear transformation parameters, and e and f are translation components. The above six parameters of the RANSAC affine constraint model are estimated by randomly sampling the smallest sample set from the initial feature point matching pair set. The residuals of the remaining matching pairs are calculated and a threshold is set to determine whether they belong to the inlier set of the current model. Feature point matching pairs with high consistency are selected to form a high-quality feature matching point set output by the model, which serves as the basis for cross-source multimodal remote sensing image registration.

10. A multimodal image matching system based on learned features and epipolar geometric constraints, used to apply the multimodal image matching method based on learned features and epipolar geometric constraints as described in any one of claims 1-9, characterized in that, include: It consists of four parts: image acquisition subsystem, feature extraction and matching subsystem, geometric verification subsystem, and matching output subsystem; The image acquisition subsystem includes a cross-source multimodal image acquisition device, a data interface module, and an image preprocessing module, used to acquire and import cross-source multimodal remote sensing image pairs from different sensors, and to perform scale unification, format conversion, and grayscale normalization processing on the images. The feature extraction and matching subsystem includes an image enhancement module, a deep feature extraction module, a feature point filtering module, and an initial matching module: the image enhancement module performs Haar wavelet transform decomposition on the input image and extracts high-frequency subband features, and enhances edge texture information by weighted fusion of the high-frequency subband features; the deep feature extraction module extracts multi-scale dense feature maps based on convolutional neural networks; the feature point filtering module constructs feature descriptors with rotation and scale invariance by combining a direction normalization mechanism; the initial matching module constructs a preliminary set of feature point matching pairs by combining the FLANN algorithm with a dynamic threshold constraint discrimination mechanism and a unidirectional uniqueness constraint rule; The geometric verification subsystem includes a epipolar geometric consistency module and a RANSAC robust model estimation module. It uses the fundamental matrix to construct epipolar geometric relations, calculates the Sampson error, and combines the RANSAC affine constraint model to filter and optimize feature point matching pairs. The matching output subsystem includes a matching visualization unit and an evaluation module, which are used to output the final registration results, draw matching connection diagrams, and calculate various evaluation indicators, supporting the quantitative analysis and comparative verification of cross-source multimodal remote sensing image registration results.

Images