Multi-modal image matching method and system based on learning features and epipolar geometric constraints

Abstract

The invention relates to a multi-modal image matching method and system based on learning features and epipolar geometric constraints. The method comprises the following steps: carrying out edge enhancement processing on an input image through wavelet transform; extracting a multi-scale dense feature map based on the transformed convolutional neural network, and generating a feature descriptor with rotation and scale invariance in combination with principal direction normalization; adopting an FLANN algorithm and dynamic distance constraint to realize preliminary feature matching; and introducing a basic matrix construction and epipolar geometric consistency verification mechanism, and eliminating mismatching point pairs in combination with an RANSAC affine constraint model. According to the method, image enhancement, deep learning and geometric verification strategies are fused, the problems of radiation nonlinearity and geometric distortion caused by imaging mechanism differences among multi-modal images are effectively solved, the matching precision and robustness are improved, and the method is suitable for remote sensing application scenes such as optical-SAR registration, multi-source image fusion and earth surface change detection.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a multimodal image matching method and system based on learning features and epipolar geometry constraints. Background Art

[0002] Multimodal remote sensing imagery, as an essential component of remote sensing imagery, is crucial in fields such as land surveys, environmental monitoring, and high-precision mapping. Existing multimodal remote sensing imagery is primarily divided into passive optical imagery and active optical imagery. Passive optical imagery relies on passive sensing systems to collect information about electromagnetic waves reflected from objects in the visible and near-infrared bands. It can accurately characterize the target's color attributes and texture characteristics, visually presenting detailed surface features. Active optical imagery constructs images by actively emitting electromagnetic pulses in the infrared, visible, or ultraviolet bands, offering unique advantages in long-distance object measurement and regional coverage. The collaborative processing and application of these multi-source data can improve the accuracy of land cover mapping, the reliability of change detection, and the effectiveness of data fusion.

[0003] Existing multimodal remote sensing image matching technologies mainly use methods based on manual design and learning features. Multimodal image matching technologies based on manual design mainly rely on the professional knowledge of researchers to design feature detectors, descriptors and matching strategies to achieve efficient matching; learning-based methods mainly rely on the data-driven characteristics of deep learning technology, using multi-level nonlinear transformations to achieve autonomous extraction from low-level visual features to high-level semantic features to achieve better matching results. These methods have their own limitations in technical implementation. When the geometric deformation and radiation differences of manual methods are large, it is difficult for existing methods to construct feature detection and description models that can accurately represent the common features between multimodal images. The performance of learning feature-based methods is obviously limited when processing large-scale affine transformed images. In addition, due to the lack of public training data sets, their generalization capabilities cannot meet the needs of cross-regional applications.

[0004] Regarding the implementation technology in the field of multimodal remote sensing image matching, existing methods still have problems that need to be solved. First, the matching performance of multimodal images is limited by the different imaging modes, making it very difficult to obtain accurate corresponding feature points and achieve stable invariant feature extraction. Second, when faced with large-scale geometric distortion and radiation differences, existing methods have poor adaptability and cannot suppress the differences between modalities. Third, there is a lack of real-value references between multimodal images, making it difficult to ensure the geometric reliability of matching results. Fourth, the lack of publicly available multimodal image training datasets limits the effectiveness of feature expression and model performance. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides a multimodal image matching method and system based on learning features and epipolar geometry constraints, aiming to solve the technical problems of the existing technology in the field of multimodal remote sensing image matching, such as invariant feature extraction, modal difference suppression, geometric correspondence matching, and lack of public data sets.

[0006] To solve the above technical problems, the present invention provides the following technical solution: a multimodal image matching method based on learning features and epipolar geometry constraints, comprising the following steps: S1. Preprocessing cross-source multimodal remote sensing image pairs from different sources to obtain original image pairs; S2. Perform Haar wavelet transform decomposition and weighting on the original image pairs, and then perform weighted reconstruction through inverse wavelet transform to obtain an enhanced image containing multi-scale detailed structures, strengthen structural details and improve the stability and distinguishability of feature points in weak texture areas; S3. Input the enhanced image containing multi-scale detail structures into the improved VGG-16 convolutional neural network for feature extraction, construct a local direction histogram to estimate the main direction of the feature points, and perform direction normalization on the feature descriptor of each feature point. In the feature extraction process, a multi-scale feature fusion strategy is combined to generate a multi-scale dense feature map, thereby obtaining a feature descriptor with rotation and scale invariance. S4. Use the FLANN algorithm to perform initial matching of feature points and construct difference vectors. Design a dynamic threshold constraint discrimination mechanism based on cosine similarity to implement dynamic adaptive distance constraint and eliminate false matching point pairs. At the same time, introduce a one-way uniqueness constraint rule to retain only the best matching result of each feature point in the source image in the target image to form a preliminary feature point matching pair set. S5. Construct a basic matrix based on the preliminary feature point matching pair set, calculate the epipolar error of the preliminary feature point matching pair, and judge its geometric consistency; construct a RANSAC affine constraint model, combine the affine constraint with the RANSAC algorithm to perform inlier estimation and model optimization, eliminate incorrect matching point pairs, and finally obtain a stable and reliable high-quality feature matching point set.

[0007] Furthermore, in step S2, the specific process includes the following steps: S21, performing two-layer Daubechies 2 wavelet transform decomposition on the original image pair to obtain the second layer low-frequency sub-band coefficients and each layer high-frequency sub-band coefficients; S22, weighting the low-frequency and high-frequency sub-band coefficients based on the enhancement weight factor to simultaneously improve the overall image contrast and the ability to express local edge texture details; S23. Keeping the high-frequency subband coefficients of the first layer unchanged, reconstruct the weighted low-frequency and high-frequency subband coefficients of the second layer by inverse wavelet transform to generate an enhanced image containing multi-scale detail structures as the input basis for subsequent feature extraction, thereby improving the stability and distribution balance of feature points in weak texture areas.

[0008] Furthermore, the enhancement weight factor is dynamically set according to the energy characteristics of different sub-bands after Haar wavelet transform decomposition, giving priority to retaining the alignment contour and directional structure features to improve the structural consistency matching capability between cross-source multimodal remote sensing image pairs.

[0009] Furthermore, in step S3, the specific process includes the following steps: S31, using the VGG-16 convolutional neural network with the last fully connected layer removed as the feature extraction backbone network, retaining its first four convolutional modules to extract feature maps and obtain feature points; S32, constructing a local direction histogram based on the neighborhood gradient direction of the feature point, estimating the main direction of the feature point, and rotating and aligning the feature descriptor of the feature point based on the main direction, completing the direction normalization processing of the feature descriptor; S33, the multi-scale feature fusion strategy is to construct a multi-scale image pyramid model, which consists of L levels, where the first level represents the original image, and the subsequent level is a scaled level of the previous level, with a scaling factor of , the last layer is the initial image times.

[0010] Furthermore, in step S3, the feature extraction backbone network performs an L2 normalization operation on the feature map output by each layer to ensure that the features have a uniform amplitude distribution and direction distribution at different scales.

[0011] Furthermore, the specific process of step S4 includes the following steps: S41, using the feature descriptor obtained in step S3 as the input for initial matching, performing a nearest neighbor search on the feature descriptor using the FLANN algorithm to obtain an initial matching point pair; S42, constructing a difference vector between the initial matching point pairs based on the feature descriptor, and calculating the cosine similarity according to the direction and distance distribution of the difference vector; S43. Design a dynamic threshold constraint discrimination mechanism based on the cosine similarity between the initial matching point pairs, and set a threshold to filter out false matching point pairs with low consistency scores; S44. Introduce a one-way uniqueness constraint rule. For each feature point in the source image, only the unique matching result with the highest cosine similarity in the target image is retained. When multiple source feature points correspond to the same target feature point at the same time, the nearest neighbor matching pairs are retained based on the principle of minimizing the Euclidean distance of the feature descriptor, and the remaining matching pairs are eliminated, thus forming a preliminary feature point matching pair set.

[0012] Furthermore, in step S43, the dynamic threshold constraint judgment mechanism defines the consistency score between each pair of initial matching points, and its mathematical expression is: ; in Represents cosine similarity, by initial matching point pair and The difference is converted into a difference vector 、 The consistency is quantitatively evaluated to describe the degree of consistency between any two initial matching points i and k in terms of the difference in feature descriptors, where 、 、 、 Represent feature points 、 、 、 The corresponding feature descriptor.

[0013] Furthermore, the specific process of step S5 includes the following steps: S51, constructing a basic matrix between the source image and the target image based on the preliminary feature point matching F , use the eight-point method to solve the basic matrix F , obtain the epipolar geometric relationship between the preliminary feature point matching pairs; S52, based on the basic matrix F Calculate the geometric distance between each pair of preliminary feature point matching pairs and their corresponding epipolar lines, and use Sampson error as the epipolar error to evaluate the geometric consistency of the preliminary feature point matching pairs; S53, modeling the local spatial geometric structure between the preliminary feature point matching pairs based on affine transformation; S54, optimizing the model constructed in step S53 by the RANSAC algorithm and performing iterative inlier estimation, eliminating erroneous matching point pairs with excessively large epipolar errors or that do not conform to affine constraints, and outputting a high-quality feature matching point set that passes the geometric consistency check.

[0014] Furthermore, in step S5, the mathematical form of the RANSAC affine constraint model is: ; Among them, a, b, c, and d represent linear transformation parameters, and e and f are translation components. The above six parameters of the RANSAC affine constraint model are estimated by randomly extracting a minimum sample set from the preliminary feature point matching pair set. The residuals of the remaining matching pairs are calculated and a threshold is set to determine whether they belong to the internal point set of the current model. Feature point matching pairs with high consistency are screened out to constitute a high-quality feature matching point set output by the model, which serves as the basis for cross-source multimodal remote sensing image registration.

[0015] The present application also provides a multimodal image matching system based on learning features and epipolar geometry constraints, which is used to apply the multimodal image matching method based on learning features and epipolar geometry constraints, including: It consists of four parts: image acquisition subsystem, feature extraction and matching subsystem, geometry verification subsystem and matching output subsystem; The image acquisition subsystem includes a cross-source multimodal image acquisition device, a data interface module, and an image preprocessing module, which are used to acquire and import cross-source multimodal remote sensing image pairs from different sensors, and perform scale unification, format conversion, and grayscale normalization on the images; The feature extraction and matching subsystem includes an image enhancement module, a deep feature extraction module, a feature point screening module, and an initial matching module: the image enhancement module is used to perform Haar wavelet transform decomposition on the input image and extract high-frequency subband features, and enhance edge texture information by weighted fusion of high-frequency subband features; the deep feature extraction module extracts multi-scale dense feature maps based on a convolutional neural network; the feature point screening module combines a direction normalization mechanism to construct a feature descriptor with rotation and scale invariance; the initial matching module uses the FLANN algorithm combined with a dynamic threshold constraint judgment mechanism and a one-way uniqueness constraint rule to form a preliminary feature point matching pair set; The geometric verification subsystem includes an epipolar geometry consistency module and a RANSAC robust model estimation module, which uses the basic matrix to construct the epipolar geometry relationship, calculates the Sampson error, and combines the RANSAC affine constraint model to filter and optimize the feature point matching pairs; The matching output subsystem includes a matching visualization unit and an evaluation module, which are used to output the final registration results, draw a matching connection diagram, and calculate various evaluation indicators to support quantitative analysis and comparative verification of cross-source multimodal remote sensing image registration results.

[0016] By means of the above technical solution, the present invention provides a multimodal image matching method based on learning features and epipolar geometry constraints, which has at least the following beneficial effects: (1) This invention effectively enhances the ability to express image edges and texture details through wavelet enhancement and main direction normalization processing. Combined with the multi-scale feature learning mechanism of convolutional neural networks, the matching accuracy is improved by more than 20% compared with traditional methods. Compared with existing multimodal remote sensing image matching technology, the accuracy of cross-modal matching is significantly improved. (2) By integrating the basic matrix constraint and the affine RANSAC elimination mechanism, the present invention significantly reduces the proportion of false matches, ensures that the final matching point set has geometric consistency, and maintains high stability in the presence of large-scale affine transformations and radiation differences, thereby improving the stability and robustness of the matching process. (3) The algorithm of the present invention does not rely on homologous data or samples from specific regions. It is applicable to a variety of remote sensing image combinations such as SAR-optical, infrared-visible light, etc. It can be widely adapted to various remote sensing application scenarios such as land use classification, disaster monitoring, and target identification, and has good cross-scenario and cross-regional generalization capabilities. (4) The present invention constructs a standardized multimodal matching process and evaluation system, and comprehensively evaluates the matching effect through multiple indicators such as NCM, SR, and RMSE, providing an integrable, scalable, and highly reliable basic module for the multimodal image processing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 The workflow of the multimodal image matching method based on learning features and epipolar geometry constraints proposed in this invention; Figure 2 Schematic diagram of the comparison of wavelet enhancement effects in an embodiment of the present invention; Figure 3 Schematic diagram of the feature extraction network structure in an embodiment of the present invention; Figure 4 Schematic diagram of feature extraction based on multimodal remote sensing images in an embodiment of the present invention; Figure 5 Schematic diagram of the optical-SAR image matching results in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] To make the above-mentioned objectives, features, and advantages of the present invention more clearly understood, the present invention is further described below in detail with reference to the accompanying drawings and specific embodiments. This will enable a full understanding of how this application uses technical means to solve technical problems and achieve technical effects, and to implement the invention accordingly.

[0019] Those skilled in the art will appreciate that all or part of the steps in the above-mentioned embodiment methods can be accomplished by instructing the relevant hardware through a program. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0020] Please refer to Figure 1-Figure 5 , shows a specific implementation method of this embodiment. This embodiment effectively solves the radiation nonlinearity and geometric distortion problems caused by differences in imaging mechanisms between multimodal images by fusing image enhancement, deep learning and epipolar geometry verification strategies, improves matching accuracy and robustness, and is suitable for remote sensing application scenarios such as precise matching between visible light images and multimodal remote sensing images such as SAR images and thermal infrared images, multi-source image fusion, and surface change detection.

[0021] Please refer to Figure 1 This embodiment proposes a multimodal image matching method based on learning features and epipolar geometry constraints, which includes the following steps: S1. Preprocess cross-source multimodal remote sensing image pairs (taking visible light-SAR images as an example) from different sources to obtain original image pairs; As a preferred implementation method of step S1, the specific process includes: obtaining a multimodal remote sensing image pair with a spatially overlapping area, the image sources including passive optical remote sensing images and active synthetic aperture radar (SAR) images, and uniformly performing grayscale normalization, resolution adjustment and format standardization processing based on the differences in their spectral characteristics and structural information, to provide basic input for subsequent feature extraction and matching.

[0022] S2. Perform Haar wavelet transform decomposition and weighted processing on the original image pairs, and then perform weighted reconstruction through inverse wavelet transform to obtain an enhanced image containing multi-scale detailed structures, strengthen structural details and improve the stability and distinguishability of feature points in weak texture areas; As a preferred implementation of step S2, the specific process includes the following steps: S21. The multimodal remote sensing images processed in this embodiment include SAR and optical images with a resolution of 10 meters and 1 meter, and there are significant differences in structure, texture and radiation. In order to alleviate the modal differences between images, the optical image and the SAR image are first subjected to wavelet transformation enhancement processing to enhance their local structure and edge features. In wavelet enhancement, the original image pair is subjected to two layers of Daubechies 2 (db2) wavelet transformation decomposition to obtain the second layer of low-frequency sub-band coefficients and each layer of high-frequency sub-band coefficients, thereby enhancing detail information and suppressing noise; the wavelet enhancement effect is compared. Figure 2 As shown; S22, based on the enhancement weight factor, the low-frequency and high-frequency sub-band coefficients obtained by Daubechies 2 (db2) wavelet transform decomposition are weighted respectively to simultaneously improve the overall image contrast and local edge texture expression capabilities; the enhancement weight factor includes the low-frequency enhancement coefficient and high frequency enhancement factor , used to control the enhancement strength of low-frequency and high-frequency sub-bands respectively. The mathematical formula of the weighted processing is as follows: ; in Indicates the The high-frequency sub-band coefficients of the layer represent the detail information of the horizontal, vertical and diagonal directions of the image respectively; Indicates the The low-frequency sub-band coefficients of the layer represent the overall brightness and large-scale structural information of the image; Represents the weighted Layer low-frequency and high-frequency sub-band coefficients; S23, the enhanced image is reconstructed by inverse wavelet transform, and its mathematical formula is: ; in, To enhance the image, the first layer high frequency subband coefficients are reconstructed No weighting is performed; More specifically, the enhancement weight factor is dynamically set according to the energy characteristics of different sub-bands after Haar wavelet transform decomposition, and the alignment contour and directional structure features are retained first to improve the structural consistency matching capability between cross-source multimodal remote sensing image pairs.

[0023] In this embodiment, by enhancing the high-frequency part and reconstructing the image, the structural clarity and boundary distinguishability of the image can be improved without introducing artifacts, thereby improving the adaptability of subsequent feature descriptors between heterogeneous images.

[0024] S3. Input the enhanced image containing multi-scale detail structures into the improved VGG-16 convolutional neural network, extract the dense feature map, construct the local direction histogram to estimate the main direction of the feature points, and perform direction normalization on the feature descriptor of each feature point. In the feature extraction process, a multi-scale feature fusion strategy is combined to generate a multi-scale dense feature map, and then obtain a feature descriptor with rotation and scale invariance. As a preferred implementation of step S3, the specific process includes the following steps: S31, using the improved classic VGG-16 structure of the multi-scale convolution feature extraction network, combined with the multi-scale feature fusion strategy, extracting image semantic features in the convolution layer of multiple different receptive fields, and detecting local extreme points as feature points in the multi-scale dense feature map. The feature extraction network structure is as follows Figure 3 shown.

[0025] The extracted features include multi-scale dense feature maps and their corresponding spatial positions, main direction information and 128-dimensional descriptor feature vectors. Feature extraction based on multimodal remote sensing images is as follows: Figure 4 shown.

[0026] To ensure uniform distribution of feature points on the image, this embodiment adopts the "Top-K response + uniform grid sampling" strategy, which prioritizes retaining the K key points with the largest response intensity as feature points, and uniformly samples the grid areas on the image to balance information expression capability and spatial coverage.

[0027] S32, constructing a local direction histogram based on the neighborhood gradient direction of the feature point, estimating the main direction of the feature point, and rotating and aligning the feature descriptor of the feature point based on the main direction, completing the direction normalization processing of the feature descriptor; S33, the multi-scale feature fusion strategy is to construct a multi-scale image pyramid model, which consists of L levels, where the first level represents the original image, and the subsequent level is a scaled level of the previous level, with a scaling factor of , the last layer is the initial image times, mathematically expressed as: ; in, 、 Represent the feature maps at different levels respectively. Represents the fused multi-scale dense feature map.

[0028] More specifically, in order to enhance the consistency of features at different scales, this embodiment introduces a feature normalization strategy to perform L2 normalization on the dense feature maps of each scale to eliminate the impact of numerical differences between scales on the expressive power of the descriptor, thereby ensuring that the features have uniform amplitude distribution and direction distribution at different scales.

[0029] In this embodiment, a convolutional neural network is used to extract multi-scale deep features. The dense feature maps output by specific layers capture semantic and texture information at different scales. During the feature extraction process, a principal direction estimation mechanism is introduced. The principal direction of each feature point (i.e., the principal direction of each feature region) is calculated using local gradient direction statistics. The feature descriptors are then rotationally aligned to achieve directional normalization. This strategy effectively improves the rotation and scale invariance of the feature descriptors, enhancing matching stability under multimodal conditions.

[0030] S4. Use the FLANN algorithm to perform initial matching of feature points and construct difference vectors. Design a dynamic threshold constraint discrimination mechanism based on cosine similarity to implement dynamic adaptive distance constraint and eliminate false matching point pairs. At the same time, introduce a one-way uniqueness constraint rule to retain only the best matching result of each feature point in the source image in the target image to form a preliminary feature point matching pair set. As a preferred implementation of step S4, the specific process includes the following steps: S41, taking the feature descriptor obtained in step S3 as the input of the initial matching, and performing the nearest neighbor search on the feature descriptor using the FLANN algorithm; specifically, for the source image Each feature point The corresponding feature descriptor , through the nearest neighbor search in the target image Search for its nearest neighbor in the feature descriptor set , construct the initial matching point pair; S42, constructing a difference vector between the initial matching point pairs based on the feature descriptor, and calculating the cosine similarity according to the direction and distance distribution of the difference vector; S43. Design a dynamic threshold constraint discrimination mechanism based on the cosine similarity between the initial matching point pairs, and set a threshold to filter out false matching point pairs with low consistency scores; Specifically, the dynamic threshold constraint discrimination mechanism defines the consistency score between each pair of initial matching points, and its mathematical expression is: ; in Represents cosine similarity, by initial matching point pair and The difference is converted into a difference vector 、 The consistency is quantitatively evaluated to describe the degree of consistency between any two initial matching points i and k in terms of the difference in feature descriptors, where 、 、 、 Represent feature points 、 、 、 The corresponding feature descriptor.

[0031] When the consistency score between the initial matching point pairs is lower than the preset threshold τ, the system will automatically judge it as an outlier and filter it out. This method reduces the interference of feature descriptor offset caused by modal differences on matching accuracy, and effectively improves the robustness of feature matching at the global geometric structure level.

[0032] S44. Introduce a one-way uniqueness constraint rule, and for each feature point in the source image, retain only the unique matching result with the best cosine similarity in the target image to form a preliminary feature point matching pair set.

[0033] In this example, the FLANN fast approximate nearest neighbor search algorithm is used to perform pairwise matching of target feature descriptors. A dynamic threshold constraint based on cosine similarity and consistency score is also introduced. The threshold is dynamically adjusted to accommodate differences in feature distribution between image pairs, thereby eliminating a large number of low-quality false matches. Furthermore, a one-way uniqueness constraint is implemented to retain the unique matching pairs in the target image with the highest similarity for each feature point in the source image, further improving the accuracy of matching pairs.

[0034] S5. Construct a basic matrix based on the preliminary feature point matching pair set, calculate the epipolar error of the preliminary feature point matching pair, and judge its geometric consistency; construct a RANSAC affine constraint model, combine the affine constraint with the RANSAC algorithm to perform inlier estimation and model optimization, eliminate incorrect matching point pairs, and finally obtain a stable and reliable high-quality feature matching point set.

[0035] As a preferred implementation of step S5, the specific process includes the following steps: S51, constructing a basic matrix between the source image and the target image based on the preliminary feature point matching F , use the eight-point method to solve the basic matrix F , obtain the epipolar geometric relationship between the preliminary feature point matching pairs; S52, based on the basic matrix FThe geometric distance between each pair of preliminary feature point matches and their corresponding epipolar lines is calculated, and the Sampson error is used as the epipolar error to evaluate the geometric consistency of the preliminary feature point matches. The mathematical expression of the Sampson error is: ; Among them, the basic matrix F Based on the relative projection relationship between the image pairs Find the solution, as well as Represent the matching points in the optical image and the SAR image respectively. By quantifying this deviation index, matching points that do not meet the constraint conditions are eliminated.

[0036] S53, modeling the local spatial geometric structure between the preliminary feature point matching pairs based on affine transformation; S54, optimize the model constructed in step S53 by the RANSAC algorithm and perform iterative inlier estimation to eliminate incorrect matching point pairs with excessively large epipolar errors or that do not conform to the affine constraint. The mathematical form of the constructed RANSAC affine constraint model is: ; Among them, a, b, c, and d represent linear transformation parameters, and e and f are translation components. The above six parameters of the RANSAC affine constraint model are estimated by randomly extracting a minimum sample set from the preliminary feature point matching pair set. The residuals of the remaining matching pairs are calculated and a threshold is set to determine whether they belong to the internal point set of the current model. Feature point matching pairs with high consistency are screened out to constitute a high-quality feature matching point set output by the model, which serves as the basis for cross-source multimodal remote sensing image registration.

[0037] In this embodiment, to address the geometric errors and false correspondences that may remain after the initial matching, the present invention introduces an epipolar geometry constraint mechanism. This mechanism calculates a fundamental matrix based on preliminary feature point matching pairs, incorporating the Sampson error between the points and the corresponding epipolar lines as a geometric consistency criterion. Furthermore, a RANSAC affine constraint model is employed for robust fitting and error point removal. Ultimately, a high-precision feature matching point set is obtained whose spatial position and structure meet the geometric consistency requirements, enabling stable registration between multimodal images.

[0038] This embodiment further provides a multimodal remote sensing image matching system based on learning features and epipolar geometry constraints, including: It consists of four parts: image acquisition subsystem, feature extraction and matching subsystem, geometry verification subsystem and matching output subsystem; The image acquisition subsystem includes optical image acquisition equipment and SAR image acquisition devices, data interface modules and image preprocessing modules. It acquires and imports cross-source multimodal remote sensing image pairs with spatial texture details and structural strength information, and performs scale unification, format conversion and grayscale normalization on the images, providing a data basis for subsequent image matching.

[0039] It should be noted that optical imagery in the image acquisition subsystem is generally multispectral or high-resolution visible light imagery, which has excellent texture and shape information. SAR imagery, on the other hand, uses side-looking radar imaging, which has the ability to penetrate clouds and acquire data in all weather conditions. It provides intensity information and terrain structure, making it suitable for remote sensing applications in areas with complex terrain and climates. These two types of imagery exhibit varying degrees of modal differences, including radiometric response, imaging geometry, and scale transformation.

[0040] The feature extraction and matching subsystem includes an image enhancement module, a deep feature extraction module, a feature point screening module, and an initial matching module, which respectively complete the detail enhancement of multimodal images, multi-scale feature extraction based on CNN, main direction estimation and direction normalization, and preliminary feature point matching pair screening using the FLANN algorithm combined with a dynamic threshold constraint judgment mechanism and a one-way uniqueness constraint rule.

[0041] It should be noted that: the image enhancement module uses the wavelet multi-level reconstruction method to perform edge and texture enhancement on the optical image and SAR image respectively to improve the consistency of feature extraction; the deep feature extraction module uses the improved VGG-16 network as the backbone, and selects the activated feature map of the intermediate convolutional layer in the multi-scale output; the extracted feature points are normalized after the main direction estimation to obtain a descriptor that is invariant to rotation and scale; in the feature point screening module, the Top-K response value selection point and the uniform grid mixed sampling method are used to ensure spatial distribution; the initial matching module adopts the FLANN fast approximate nearest neighbor algorithm, and combines the dynamic threshold constraint discrimination mechanism to preliminarily eliminate erroneous point pairs.

[0042] The geometric verification subsystem includes an epipolar geometric consistency module and a RANSAC robust model estimation module, which are used to perform geometric consistency verification and error elimination on the initial matching point pairs to obtain the final high-confidence matching results.

[0043] It should be noted that the epipolar geometric consistency module verifies whether the preliminary feature point matching pairs satisfy the epipolar geometric constraint relationship between the source image and the target image through basic matrix calculation, and eliminates matching pairs that obviously violate geometric consistency; the RANSAC robust model estimation module fits the affine model and uses RMSE as the error indicator to eliminate outliers, thereby improving the matching geometric accuracy and stability.

[0044] The matching output subsystem includes a matching visualization unit and an evaluation module, which are used to output the final registration results, draw the matching connection diagram, and calculate evaluation indicators such as the number of matching points (NCM), matching accuracy (SR), root mean square error (RMSE) and matching time (MT), supporting quantitative analysis and comparative verification of cross-source multimodal remote sensing image registration results. Figure 5 The following diagram shows the matching effect of optical-SAR image in this embodiment, where: Figure 5 (a) is the result of verification on the WHU-SEN-City dataset. Figure 5 (b) is the result of verification on the OSdataset dataset. Figure 5 (c) in the figure is the verification result on the SEN1-2 dataset.

[0045] The system provided in this embodiment has the following advantages: It achieves highly robust registration of multimodal remote sensing images under complex radiation differences and geometric distortion conditions; it comprehensively utilizes the feature expression capabilities of deep learning and traditional geometric consistency constraints to balance expressiveness and accuracy; it has a complete evaluation mechanism that supports batch processing and quantitative output, facilitating subsequent map production and monitoring analysis.

[0046] This system provides an effective means for high-precision collaborative processing of multimodal remote sensing data. It is suitable for various scenarios such as land change monitoring, urban modeling, and disaster response. It has practical application significance for improving the level of intelligent interpretation and cross-modal integration of remote sensing images.

[0047] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0048] The logic and / or steps represented in the flowchart or otherwise described herein may be considered, for example, as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0049] The above embodiments provide a detailed introduction to the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A multimodal image matching method based on learning features and epipolar geometry constraints, characterized in that: The following steps are involved: S1. Preprocessing cross-source multimodal remote sensing image pairs from different sources to obtain original image pairs; S2. Perform Haar wavelet transform decomposition and weighted processing on the original image pairs, and then perform weighted reconstruction through inverse wavelet transform to obtain an enhanced image containing multi-scale detailed structures, strengthen structural details and improve the stability and distinguishability of feature points in weak texture areas; S3. Input the enhanced image containing multi-scale detail structures into the improved VGG-16 convolutional neural network for feature extraction, construct a local direction histogram to estimate the main direction of the feature points, and perform direction normalization on the feature descriptor of each feature point. In the feature extraction process, a multi-scale feature fusion strategy is combined to generate a multi-scale dense feature map, thereby obtaining a feature descriptor with rotation and scale invariance. S4. Use the FLANN algorithm to perform initial matching of feature points and construct difference vectors. Design a dynamic threshold constraint discrimination mechanism based on cosine similarity to implement dynamic adaptive distance constraint and eliminate false matching point pairs. At the same time, introduce a one-way uniqueness constraint rule to retain only the best matching result of each feature point in the source image in the target image to form a preliminary feature point matching pair set. S5. Construct a basic matrix based on the preliminary feature point matching pair set, calculate the epipolar error of the preliminary feature point matching pair, and judge its geometric consistency; construct a RANSAC affine constraint model, combine the affine constraint with the RANSAC algorithm to perform inlier estimation and model optimization, eliminate incorrect matching point pairs, and finally obtain a stable and reliable high-quality feature matching point set.

2. The multimodal image matching method based on learning features and epipolar geometry constraints according to claim 1, characterized in that: The specific process of step S2 includes the following steps: S21, performing two-layer Daubechies 2 wavelet transform decomposition on the original image pair to obtain the second layer low-frequency sub-band coefficients and each layer high-frequency sub-band coefficients; S22, weighting the low-frequency and high-frequency sub-band coefficients based on the enhancement weight factor to simultaneously improve the overall image contrast and the ability to express local edge texture details; S23. Keeping the high-frequency subband coefficients of the first layer unchanged, reconstruct the weighted low-frequency and high-frequency subband coefficients of the second layer by inverse wavelet transform to generate an enhanced image containing multi-scale detail structures as the input basis for subsequent feature extraction, thereby improving the stability and distribution balance of feature points in weak texture areas.

3. The multimodal image matching method based on learning features and epipolar geometry constraints according to claim 2, characterized in that: The enhancement weight factor is dynamically set according to the energy characteristics of different sub-bands after Haar wavelet transform decomposition, and prioritizes retaining the alignment contour and directional structure features to improve the structural consistency matching capability between cross-source multimodal remote sensing image pairs.

4. The multimodal image matching method based on learning features and epipolar geometry constraints according to claim 1, characterized in that: The specific process of step S3 includes the following steps: S31, using the VGG-16 convolutional neural network with the last fully connected layer removed as the feature extraction backbone network, retaining its first four convolutional modules to extract feature maps and obtain feature points; S32, constructing a local direction histogram based on the neighborhood gradient direction of the feature point, estimating the main direction of the feature point, and rotating and aligning the feature descriptor of the feature point based on the main direction, completing the direction normalization processing of the feature descriptor; S33, the multi-scale feature fusion strategy is used to construct a multi-scale image pyramid model, which consists of L levels, where the first level represents the original image, and the subsequent levels are scaled levels of the previous level, with a scaling factor of , the last layer is the initial image times.

5. The multimodal image matching method based on learning features and epipolar geometry constraints according to claim 4, characterized in that: In step S3, the feature extraction backbone network performs an L2 normalization operation on the feature map output by each layer to ensure that the features have a uniform amplitude distribution and direction distribution at different scales.

6. The multimodal image matching method based on learning features and epipolar geometry constraints according to claim 1, characterized in that: The specific process of step S4 includes the following steps: S41, using the feature descriptor obtained in step S3 as input for initial matching, performing nearest neighbor search on the feature descriptor using the FLANN algorithm to obtain initial matching point pairs; S42, constructing a difference vector between the initial matching point pairs based on the feature descriptor, and calculating the cosine similarity according to the direction and distance distribution of the difference vector; S43. Design a dynamic threshold constraint discrimination mechanism based on the cosine similarity between the initial matching point pairs, and set a threshold to filter out false matching point pairs with low consistency scores; S44. Introduce a one-way uniqueness constraint rule. For each feature point in the source image, only the unique matching result with the highest cosine similarity in the target image is retained. When multiple source feature points correspond to the same target feature point at the same time, the nearest neighbor matching pairs are retained based on the principle of minimizing the Euclidean distance of the feature descriptor, and the remaining matching pairs are eliminated, thus forming a preliminary feature point matching pair set.

7. The multimodal image matching method based on learning features and epipolar geometry constraints according to claim 6, characterized in that: In step S43, the dynamic threshold constraint judgment mechanism defines the consistency score between each pair of initial matching points, and its mathematical expression is: ； in Represents cosine similarity, by initial matching point pair and The difference is converted into a difference vector 、 The consistency is quantitatively evaluated to describe the degree of consistency between any two initial matching points i and k in terms of the difference in feature descriptors, where 、 、 、 Represent feature points 、 、 、 The corresponding feature descriptor.

8. The multimodal image matching method based on learning features and epipolar geometry constraints according to claim 1, characterized in that: The specific process of step S5 includes the following steps: S51, constructing a basic matrix between the source image and the target image based on the preliminary feature point matching F , use the eight-point method to solve the basic matrix F , obtain the epipolar geometric relationship between the preliminary feature point matching pairs; S52, based on the basic matrix F Calculate the geometric distance between each pair of preliminary feature point matching pairs and their corresponding epipolar lines, and use Sampson error as the epipolar error to evaluate the geometric consistency of the preliminary feature point matching pairs; S53, modeling the local spatial geometric structure between the preliminary feature point matching pairs based on affine transformation; S54, optimizing the model constructed in step S53 by the RANSAC algorithm and performing iterative inlier estimation, eliminating erroneous matching point pairs with excessively large epipolar errors or that do not conform to affine constraints, and outputting a high-quality feature matching point set that passes the geometric consistency check.

9. The multimodal image matching method based on learning features and epipolar geometry constraints according to claim 8, characterized in that: In step S5, the mathematical form of the RANSAC affine constraint model is: ； Among them, a, b, c, and d represent linear transformation parameters, and e and f are translation components. The above six parameters of the RANSAC affine constraint model are estimated by randomly extracting a minimum sample set from the preliminary feature point matching pair set. The residuals of the remaining matching pairs are calculated and a threshold is set to determine whether they belong to the internal point set of the current model. Feature point matching pairs with high consistency are screened out to constitute a high-quality feature matching point set output by the model, which serves as the basis for cross-source multimodal remote sensing image registration.

10. A multimodal image matching system based on learning features and epipolar geometry constraints, for applying the multimodal image matching method based on learning features and epipolar geometry constraints according to any one of claims 1 to 9, characterized in that: include: It consists of four parts: image acquisition subsystem, feature extraction and matching subsystem, geometry verification subsystem and matching output subsystem; The image acquisition subsystem includes a cross-source multimodal image acquisition device, a data interface module, and an image preprocessing module, which are used to acquire and import cross-source multimodal remote sensing image pairs from different sensors, and perform scale unification, format conversion, and grayscale normalization on the images; The feature extraction and matching subsystem includes an image enhancement module, a deep feature extraction module, a feature point screening module, and an initial matching module: the image enhancement module is used to perform Haar wavelet transform decomposition on the input image and extract high-frequency subband features, and enhance edge texture information by weighted fusion of high-frequency subband features; the deep feature extraction module extracts multi-scale dense feature maps based on a convolutional neural network; the feature point screening module combines a direction normalization mechanism to construct a feature descriptor with rotation and scale invariance; the initial matching module uses the FLANN algorithm combined with a dynamic threshold constraint judgment mechanism and a one-way uniqueness constraint rule to form a preliminary feature point matching pair set; The geometric verification subsystem includes an epipolar geometry consistency module and a RANSAC robust model estimation module, which uses the basic matrix to construct the epipolar geometry relationship, calculates the Sampson error, and combines the RANSAC affine constraint model to filter and optimize the feature point matching pairs; The matching output subsystem includes a matching visualization unit and an evaluation module, which are used to output the final registration results, draw a matching connection diagram, and calculate various evaluation indicators to support quantitative analysis and comparative verification of cross-source multimodal remote sensing image registration results.

Images