A local energy response feature transformation multi-modal image matching method and system

Abstract

The application discloses a kind of local energy response feature transformation multi-modal image matching method and system, system includes image acquisition module, local energy response model establishment module, feature extraction module, feature description module, initial matching set construction module, remove false matching module;Method is: obtaining two multi-modal images to be matched;Establish local energy response model based on log-Gabor filter;On the basis of local energy response model, combined with joint bilateral filter, feature points in two multi-modal images are extracted using FAST algorithm;Log polar coordinate feature descriptor of feature point is established using three-dimensional Gaussian kernel and maximum index strategy;According to the similarity of feature descriptor in two images, establish a group of initial matching set;Affine transformation consistency index is obtained by fast sample consistency algorithm, and the final image matching result is obtained.The scheme of the application can cope with the interference of noise and local nonlinear deformation in multi-modal image, and has rotation invariance.

Description

Technical Field

[0001] This invention relates to the field of computer vision image processing technology, and in particular to a multimodal image matching method and system based on local energy response feature transformation. Background Technology

[0002] Image matching is a fundamental and crucial problem in computer vision, aiming to extract reliable matching relationships from two or more images with identical or similar content and structure. Multimodal image matching is an important branch of image matching, and also a challenging field. It aims to construct correspondences between multimodal images, which typically originate from different sensors or imaging conditions and exhibit significant modal appearance differences. Excellent multimodal image matching methods can unify key information from different images and achieve mutual compensation, improving the reliability of data processing and applications. Therefore, multimodal image matching plays a vital role in change detection, image fusion, medical research, and land use analysis.

[0003] However, due to the different imaging mechanisms, there are significant nonlinear intensity differences between multimodal images, which increases the difficulty of image matching and poses a serious challenge to traditional gradient-based and intensity-based image matching methods. There are three main reasons for the nonlinear intensity differences: (1) Differences in the imaging characteristics of the sensors themselves. Due to the imaging characteristics of different sensors, even image information obtained from the same object or scene will have different expressions. For example, in medical imaging, computer tomography (CT) images contain more structural information, while single-photon emission computed tomography (SPECT) images contain more functional information. (2) Radiative transfer errors caused by the atmosphere. This error is more common in multi-temporal remote sensing images, such as remote sensing images captured in different seasons or weather. (3) Appearance differences caused by differences in lighting conditions, imaging time, spectrum and resolution, such as cross-time and cross-spectral images commonly seen in computer vision.

[0004] In the field of multimodal image matching, many application scenarios require manual intervention, which is extremely time-consuming for processing large amounts of image data and inevitably introduces subjective matching errors. Existing gradient- and intensity-based image matching methods are sensitive to differences in nonlinear intensity, making it difficult to extract feature points with high repeatability and accuracy, and to establish precise matches. To address this issue, many methods have emerged to reduce inherent modal differences and local distortions in multimodal images. However, the nonlinear intensity differences vary across different types of multimodal images, resulting in low universality and difficulty in comprehensively addressing these issues. Furthermore, multimodal images also suffer from scale and rotation variations, local nonlinear geometric distortions, background noise, low resolution, and low texture, all of which significantly reduce the performance of existing multimodal image matching algorithms and fail to meet the needs of practical applications. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a multimodal image matching method and system based on local energy response feature transformation, wherein the method includes the following steps:

[0006] S1. Obtain the two multimodal images to be matched;

[0007] S2. Establish a local energy response model based on a log-Gabor filter;

[0008] S3. Based on the local energy response model, the FAST algorithm is used in conjunction with a joint bilateral filter to extract feature points from two multimodal images;

[0009] S4. Based on the local energy response model, a logarithmic polar coordinate feature descriptor for the feature points extracted in S3 is established using a three-dimensional Gaussian kernel and a maximum index strategy.

[0010] S5. Establish an initial matching set based on the similarity of feature descriptors;

[0011] S6. Obtain the affine transformation consistency index ξ of the matching points in the initial matching set using the fast sample consistency algorithm. i According to ξ i With ξ i The threshold δ is used to remove incorrect matches from the initial matching set, resulting in the final image matching result.

[0012] This invention also proposes a multimodal image matching system based on local energy response feature transformation, used to implement the steps of the proposed multimodal image matching method based on local energy response feature transformation, including the following modules:

[0013] The image acquisition module is used to acquire two multimodal images to be matched;

[0014] A local energy response model building module is used as the basis for feature extraction and description.

[0015] The feature extraction module is used to extract feature points with high repeatability and robustness from two multimodal images;

[0016] The feature description module is used to construct a fully robust and rotation-invariant descriptor for the extracted feature points in order to facilitate the establishment of initial matching.

[0017] The initial matching set construction module is used to construct an initial matching set based on the similarity of feature descriptors in two multimodal images;

[0018] The error matching removal module is used to accurately remove erroneous matches in the initial matching and establish more accurate matching relationships.

[0019] The beneficial effects of the technical solution provided by this invention are:

[0020] This invention proposes a multimodal image matching method based on local energy response feature transformation. It transforms multimodal images into intermediate modes with more common feature information, reducing modal differences and increasing similarity, and then performs feature extraction, description, and matching based on this transformation. First, a local energy response model is established using a multi-scale, multi-directional two-dimensional log-Gabor filter to obtain local energy response functions (energy) in multiple o-directions. o In feature extraction, energy is measured in all directions. o The total energy response (ET) is obtained through summation, denoising, weighting, and normalization. Guided by ET, a joint bilateral filter is used to filter the multimodal image, and the difference between this filter and ET yields the intermediate modal image. Feature extraction is then performed using the FAST algorithm on the intermediate modal image. In the feature description, a three-dimensional Gaussian kernel is used to analyze the energy response in multiple directions. o Convolution is performed, and then the final feature descriptor map is obtained through a maximum indexing strategy. Log-polar coordinate descriptors are then built based on this map. Initial matching is then established based on the similarity of the descriptors. Finally, a fast sample consensus algorithm is used to remove incorrect matches and build a more accurate set of correct matches.

[0021] The technical solution of this invention can address the differences in nonlinear intensity between multimodal images, improving the accuracy and quantity of feature point extraction; it can address the interference of noise and local nonlinear deformation in multimodal images, improving matching accuracy; and it has rotation invariance, enabling it to process images that undergo rotational changes. Attached Figure Description

[0022] Figure 1This is a flowchart of a multimodal image matching method based on local energy response feature transformation according to an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of feature extraction according to an embodiment of the present invention; wherein, (a) is the input image pair, and (b) and (c) are the total local energy response map and feature extraction map I of the remote sensing cross-seasonal image pair, respectively. out (d) shows the feature detection results on the input image, and (e) shows the results on the input image. out The feature detection results are shown above, where “·” represents unmatched outliers and “*” represents successfully matched points.

[0024] Figure 3 This is a schematic diagram of a test experiment using a three-dimensional Gaussian kernel-like descriptor to process Gaussian noise in an embodiment of the present invention; where (a) is an input image pair, (b) is an image with added Gaussian noise of mean u=0 and variance v=23, and (c) is the matching result of images with added Gaussian noise of different variances. The vertical axis of the upper figure is NCM, the vertical axis of the lower figure is CMR, column A is the result of using a three-dimensional Gaussian kernel-like descriptor, and column B is the result of not using a Gaussian kernel-like descriptor.

[0025] Figure 4 This is a schematic diagram of the manifold vector field and logarithmic polar coordinate grid of the feature description map in an embodiment of the present invention; wherein, (a) is a multimodal image pair, (b) is the feature description map of (a), (c) is the manifold vector field of (b), and (d) is a circular logarithmic polar coordinate grid containing 36 position bins, each position bin containing a grid with N... o Histogram of the distribution of each column;

[0026] Figure 5 This is a schematic diagram illustrating the sensitivity of the feature description map to rotation angle in an embodiment of the present invention; wherein, (a) and (b) are two original multimodal images, (c) and (d) are feature description maps of (a) and (b) respectively, (g) is the feature description map of (b) after being rotated 90° counterclockwise, (e) is the difference map between (c) and (d), (h) is the difference map between (c) and (g), (f) is the matching result based on (c) and (d), and (i) is the matching result based on (c) and (g);

[0027] Figure 6 This is a schematic diagram of the matching results and difference maps under different angle errors [0°, 5°, 10°, 15°, 20°, 25°] in an embodiment of the present invention; the left side of each group of images is the matching result, and the right side is the difference map with a feature description map of 50° rotation angle;

[0028] Figure 7 These are typical multimodal images and their matching results for all categories in the embodiments of the present invention;

[0029] Figure 8 The qualitative results of multimodal image matching in the embodiments of the present invention are shown. The two ends connected by the straight line are the matching points. Among them, (a) and (b) are map and optical image pairs and optical cross-time image pairs from remote sensing datasets, respectively, which have strong nonlinear intensity differences and speckle noise. (c) and (d) are visible light and infrared image pairs and visible light cross-season image pairs from computer vision datasets, respectively. Their lighting conditions are poor and the scenes are complex. (e) and (f) are PD-T2 image pairs and retinal image pairs from medical datasets, respectively. They have large imaging differences.

[0030] Figure 9 This is a schematic diagram of the quantitative results of image matching and the angle robustness test in an embodiment of the present invention; where the abscissas of (a)-(f) are different types of image pairs, (a)-(c) are the quantitative matching results of images without rotation transformation, (d)-(f) are the quantitative matching results of images rotated by 50°, (g)-(i) are different rotation angles, and (g)-(i) are the test results of angle robustness.

[0031] Figure 10 This is a structural diagram of a multimodal image matching system based on local energy response feature transformation in an embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0033] refer to Figure 1 This embodiment provides a multimodal image matching method based on local energy response feature transformation, including the following steps:

[0034] S1. Obtain the two multimodal images to be matched.

[0035] Two multimodal images to be matched must be of the same size and contain the same or similar objects or scenes to be eligible for matching. This invention can handle a wide range of multimodal image categories. Taking the dataset used in this embodiment as an example, the method of this invention can be applied to:

[0036] (1) Remote sensing multimodal image dataset: This dataset contains seven types of multimodal image pairs, namely UAV cross-seasonal image pairs, cross-day and night image pairs, depth map and optical image pairs, infrared image and optical image pairs, map and optical image pairs, optical cross-time image pairs, and SAR (Synthetic Aperture Radar) and optical image pairs. These remote sensing multimodal images come from different imaging devices, imaging times, and different categories, and also contain strong speckle noise;

[0037] (2) Computer Vision Multimodal Image Dataset: This dataset contains four types of image pairs: visible light and infrared image pairs, visible light and near-infrared image pairs, visible light cross-seasonal image pairs, and cross-day and night image pairs. These images suffer from problems such as large geometric deformation, low quality, background noise, occlusion, abnormal lighting, and low texture.

[0038] (3) Medical Multimodal Image Dataset: This dataset contains six types of image pairs: MRI (Magnetic Resonance Imaging) and PET (Positron Emission Tomography) image pairs, SPECT and CT image pairs, MRI T1 and T2 image pairs, PD (Proton Density) and T2 image pairs, PD and T1 image pairs, and retinal image pairs. It has the richest multimodal image content. Different modalities of medical images contain different information (structural or functional information) and the differences are significant, so matching between them is of great importance.

[0039] S2. Establish a local energy response model based on a log-Gabor filter.

[0040] The biggest challenge in multimodal image matching is the difference in nonlinear intensity between images. To overcome this problem, it is necessary to find significant common features between images and establish matching relationships. Extensive research on multimodal image matching shows that the most common common features between images of different modalities are edges and stripes. Figure 2 The experimental results in (e) show that correctly matched feature points are clearly concentrated on the edges and bars of the image. The log-Gabor filter can detect edges and bars in the image because step edges and bars have specific local phase characteristics, which can be detected using orthogonal filters. Since the image is a two-dimensional signal with a wide frequency range, a multi-scale, multi-directional two-dimensional log-Gabor filter is used to construct the local energy response model.

[0041] Step S2 specifically includes:

[0042] S21. Construct a two-dimensional log-Gabor filter (LGF). The frequency domain formula for the LGF is:

[0043]

[0044] Where f is the frequency of the log-Gabor filter, f0 is the center frequency of the log-Gabor filter, and θ is the direction angle of the log-Gabor filter. so σ is the principal direction angle of the log-Gabor filter. f Let σ be the frequency bandwidth of the log-Gabor filter. θ denoted as angular bandwidth of the log-Gabor filter, s represents the wavelet scale of the log-Gabor filter, and o represents the direction of the log-Gabor filter.

[0045] S22. The spatial domain expression of the LGF can be obtained by performing an inverse Fourier transform on the LGF(f, θ, s, o) in the frequency domain. The spatial domain expression of the LGF consists of an even-symmetric filter and an odd-symmetric filter, and its formula is as follows:

[0046] LGF(x,y,s,o)=LGF even (x,y,s,o)+j·LGF odd (x, y, s, o) (2)

[0047] LGF even (x,y,s,o) is the real part of the LGF at scale s and direction o. LGF odd (x,y,s,o) is the imaginary part of the LGF at scale s and direction o, which also represent even-symmetric and odd-symmetric log-Gabor wavelets, respectively. x is the abscissa in the spatial domain, y is the ordinate in the spatial domain, s represents the wavelet scale of the log-Gabor filter, and o represents the direction of the log-Gabor filter.

[0048] S23. Wavelets can detect discontinuous local features without intrinsic scale because these features can be reflected in the wavelet coefficient values. Input image I in (x, y) are convolved with the even-symmetric and odd-symmetric wavelets of the spatial domain LGF, respectively, to obtain the wavelet response component E at a given scale s and orientation o. so (x,y) and O so (x,y):

[0049]

[0050] S24. Since the local energy response model is only meaningful over a wide frequency range, it is necessary to calculate E at all scales in each direction. so (x, y) and O soThe sum of (x, y) gives the total response in each direction. and To achieve a relatively uniform spectral coverage of the signal. By calculating... and The square root of this value yields the local energy response in direction o:

[0051]

[0052] S25, Energy o (x,y) represents the local energy response of a point (x,y) in the image along direction o. To detect the features of feature points in various directions, the directions of the Local Energy Response (LGF) need to be uniformly distributed across the frequency plane. The local energy response vector is Energy. o It is fundamental to detecting and describing local features of an image. However, it also relates to Energy. o Adding the results along the direction o to obtain the total local energy response is inappropriate for direct use in feature detection. Therefore, it is necessary to perform energy... o The following steps should be taken:

[0053] (1) Normalize the local energy responses to avoid inconsistencies in their magnitude standards. Specifically, this involves normalizing the energy... o The sum divided by (Σ s ∑ o Am s,o (x,y)+ε), where ∑ s ∑ o Am s,o (x,y) is the sum of the amplitudes of the wavelet response components in all directions and at all scales, and ε is a small constant to avoid the sum of amplitudes being too small.

[0054] (2) Introduce a noise compensation term T0 and remove Energy. o The stray response of noise in (x,y) is shown, and noise compensation is performed independently in each direction.

[0055] (3) Add a weighting function W to the extended frequency distribution. o (x,y) is used to enhance feature localization, especially for smoothed features. W o The value of (x,y) is related to the bandwidth; reducing the weight in areas with narrow bandwidth can reduce spurious responses. After the above processing, the overall local energy response can be obtained:

[0056]

[0057] The amplitudes of the wavelet response components on s and o are:

[0058]

[0059] Where ε is a small constant to prevent the denominator from being 0, and |·| ensures that the output is non-negative.

[0060] S3. Based on the local energy response model, the FAST algorithm is used in conjunction with a joint bilateral filter to extract feature points from the two images. Through step S2, a local energy response model ET(x,y) for multimodal images is established, which contains rich common features of multimodal images, namely corner and edge information. However, when the image is disturbed by severe noise or outliers, the information carried by ET(x,y) will be destroyed, thereby reducing the accuracy and repeatability of feature point extraction.

[0061] S31, bilateral filters and their variations are commonly used and well-known tools for image edge preservation and smoothing, and have been widely applied in computer graphics and computer vision. Because joint bilateral filters offer good robustness against noise and outlier interference, they are used to filter input images. Considering that bilateral filters tend to filter out most fine textures while retaining only the coarse outlines of the image, ET(x,y) is used as a guide to preserve image details with minimal computational cost.

[0062] The information contained in ET(x,y) is provided in the form of an image, called a guided graph. Using this as the guided weight, the formula for the joint bilateral filter can be obtained as follows:

[0063]

[0064] Where I in (i) and JBF(j) represent the input and output images of the joint bilateral filter, respectively, where j and i represent the indices of the pixel coordinates, N(j) is the set of neighboring pixels of pixel j in the image spatial domain, and W j For the normalized operands, σ s and σ r The spatial and structural weights of the joint bilateral filter are controlled separately. The structural weights preserve the fine texture of the image and restore its structure as much as possible, while the spatial weights are used to remove noise. ET(j) is the total local energy response at pixel j, and ET(i) is the total local energy response at pixel i. ||ji|| 2 Let ||ET(j) - ET(i)|| be the distance between pixels j and i. 2 G represents the difference between ET(j) and ET(i). d (ij) is the spatial distance function, G r (ij) is the structure guidance function and the spatial distance function G. d Weights are set based on the distance between pixels, and the structure guidance function G... rThe weights are set according to the range of ET(x,y), and they are defined as follows:

[0065]

[0066] S32. Finally, take the difference between JBF(j) and ET(x,y) as plot I. out To add more detailed textures:

[0067] I out =JBF(j)-ET(j) (9)

[0068] Use I out As the final feature extraction map, the FAST feature point extraction algorithm is used in I out Feature points are extracted from the top. Figure 2 This is a schematic diagram of feature extraction. By comparing (d) and (e), it can be found that traditional gradient-based feature detectors (such as FAST) cannot directly handle the nonlinear intensity differences in the original multimodal image, resulting in low repeatability and matching rate of detected feature points. However, based on I... out The detected feature points are highly robust to differences in nonlinear intensity and can well reflect the common features between two images.

[0069] S4. Based on the local energy response model, a logarithmic polar coordinate feature descriptor is established using a three-dimensional Gaussian kernel and a maximum indexing strategy. Specifically:

[0070] S41. After extracting the feature points, it is necessary to construct feature point descriptors to distinguish each feature. The local energy response (Energy) in formula (4) o (x,y) contains many common features of multimodal images, making it highly suitable for feature description. This invention addresses the issue of N... o Energy in each direction o (x,y) employs a maximum indexing strategy to obtain a feature descriptor map that preserves details. Considering the sensitivity of local energy response to noise, such as multiplicative noise between SAR and optical images, a noise suppression strategy is designed to improve the robustness of the descriptor.

[0071] If we take the Energy in each direction o If (x, y) is taken as an energy response layer, then energy response layers in all directions can form a sequence. That is, arranging the energy response layers along the dimension of direction o yields a cube containing three-dimensional information. Subsequently, a three-dimensional Gaussian-like kernel is used in this cube to suppress noise. The three-dimensional Gaussian-like kernel consists of a two-dimensional Gaussian kernel on the x and y planes and a kernel d in the direction . o =[1,8,1] T Composition. The processing of a three-dimensional Gaussian kernel is as follows:

[0072]

[0073] in, Let T be a two-dimensional Gaussian kernel in the x and y planes. o This is the result of three-dimensional Gaussian kernel processing. The kernel is a two-dimensional Gaussian convolutional kernel for the energy response layer in the direction o. It reduces the energy contribution far from the center of the region, thereby suppressing noise. Let d be a two-dimensional Gaussian kernel in the x and y planes. o It is a convolution kernel in the orientation dimension. This convolution operation smooths the gradient in dimension o, which can reduce orientation distortion caused by local nonlinear geometric deformation.

[0074] Next, normalize T on dimension o. o '(x,y):

[0075]

[0076] Where ε is a very small constant, N o Represents Energy o The direction number of (x,y).

[0077] like Figure 3 As shown, the effect of a 3D Gaussian kernel can be tested by adding Gaussian noise to the image. The feature vector processed by the 3D Gaussian kernel can improve the robustness of the descriptor. Compared with the unprocessed version, the experimental results show a higher number of correct matches (NCM) and a higher correct match ratio (CMR).

[0078] S42. After processing with a three-dimensional Gaussian kernel, the energy response layers are rearranged again to obtain a multi-channel information mapping. Next, use the maximum index strategy from Each pixel is assigned a dominant orientation. The basic operation is to determine the dominant orientation. The maximum value T in the directional dimension max (x i ,y i ) and the corresponding direction o max ,Right now: Finally, use the o of all pixels. max The resulting maximally indexed graph serves as the final feature description graph. At the physical level, o max It is the direction in which the frequency domain energy of each pixel is most significant. For example... Figure 4As shown in (c), the feature description map is actually a manifold vector field composed of the energy salient directions of all pixels. It can be used to construct descriptors to avoid the effects of nonlinear intensity differences and local nonlinear deformations.

[0079] S43. Construct descriptors based on the aforementioned feature description maps. When the image undergoes affine transformation or local nonlinear deformation, a logarithmic polar coordinate structure is more robust than a traditional square patch structure descriptor. In this embodiment, circular patches centered on feature points are selected and divided into a logarithmic polar coordinate grid consisting of 4 radial bins and 9 angular bins, resulting in a total of 36 position bins. Figure 4 (d) shows a logarithmic polar grid with 36 position bins. A grid with N is constructed for each position bin. o The distribution histogram of each column can ultimately yield a size of 4×9×N. o A descriptor is generated that connects the distribution histograms of all location bins. Finally, the feature descriptor vector is normalized to achieve brightness invariance.

[0080] Achieve rotation invariance for feature descriptors:

[0081] In the feature description, the feature description map is constructed using a local energy response model based on LGF convolution, which is closely related to the angular component of LGF (the second term of Equation (1)). If the starting angle θ of the angular component... so Different features will result in different feature descriptor maps. Figure 5 In the difference plot (h), we can see that the feature description plots (c) and (g) are very different because their θ so There is a 90° difference. Therefore, the matching result (i) based on (c) and (g) is a failure. As can be seen from the difference map (e), the difference between the feature description maps (c) and (d) is small, so the matching result (f) based on them is good. Therefore, in order to achieve rotation invariance, a method based on the principal direction of the feature points is designed to reduce the difference in the feature description maps caused by rotation.

[0082] The basic steps involve using feature extraction map I. out The principal direction of gradient change for each feature point is extracted, and then used as the principal direction to rotate the descriptor patches. First, a multidimensional filtering operator H = [-1, 0, 1] is applied to I... out Filter the image and calculate its vertical and horizontal gradient maps:

[0083]

[0084] Among them, G x (x,y), G y(x,y) and P(x,y) represent the horizontal gradient, vertical gradient, and pixel value of the feature description map at position (x,y), respectively. The pixel gradient amplitude G(x,y) and gradient direction angle Angle(x,y) at (x,y) are defined as follows:

[0085]

[0086] Draw a circular patch with radius r centered on the feature point, and divide it into n equal angular intervals. Then, calculate the sum of pixel gradient amplitudes in each angular interval. Next, calculate a histogram with the n angles as the x-axis and the sum of gradient amplitudes in each angular interval as the y-axis. Select the direction with the largest y-axis (or the extreme value direction) as the principal direction of the feature point. After selecting the principal direction, rotate the feature descriptor patch according to the principal direction, and then perform matching to achieve rotation invariance.

[0087] However, experiments revealed that matching rotating images using the main direction method yielded relatively coarse results, with most matching points having errors exceeding 5 pixels. Therefore, directly using this as the final result is not feasible. However, a small number of high-precision matching points can be obtained by adjusting the algorithm parameters, and these can be used to estimate the total rotation angle θ. Finally, θ is used as the θ of the LGF (Low-Frequency Function). so This reduces the difference in feature descriptors between two images due to angular differences, achieves rotation invariance of feature descriptors, and improves the number and accuracy of matches.

[0088] The rotation angle error estimated using the principal direction method is within 10°. To explore what angle error range can meet the requirements for matching accuracy and quantity, test experiments were conducted on representative multimodal images, with different angle differences set in each experiment. Feature descriptive maps were then obtained and matched. The results are as follows: Figure 6 As shown in Table 1, which presents the NCM and CMR for different angular errors, the matching performance decreases with increasing error, but the results are acceptable when the error is within 10°. When the error is greater than 10°, both NCM and CMR decrease significantly, and the differences in the feature descriptors are obvious.

[0089] Table 1

[0090]

[0091]

[0092] S5. Establish an initial matching set based on the similarity of feature descriptors in the two images. Through steps S3 and S4, feature point sets and feature descriptors for each feature point of the two multimodal images have been obtained. Based on this, a one-to-one correspondence between the two feature point sets can be established using nearest neighbor matching technology and Euclidean distance similarity measurement, thus obtaining the initial matching set.

[0093] S6. Obtain the affine transformation consistency index ξ of the matching points in the initial matching set using the fast sample consistency algorithm. i According to ξ i With ξ i The threshold δ is used to remove erroneous matches from the initial matching set, resulting in the final image matching result. Specifically:

[0094] The Fast Sample Consistency Algorithm is used to remove erroneous matches and establish the affine transformation matrix H between the two multimodal images to be matched. An affine transformation consistency index ξ is then established using the matching points Xi and Yi from the initial matching set. i :

[0095] ξ i =Y i -H*X i (14)

[0096] When ξ i The i-th match is considered a correct match if it satisfies the following formula:

[0097] ξ i <δ (15)

[0098] Where δ is the threshold, the above processing can effectively remove erroneous matches in the initial matching set and establish high-precision image matching results.

[0099] In this embodiment, to test the performance and efficiency of the method, qualitative and quantitative comparative experiments were conducted on three types of multimodal image datasets, totaling 164 pairs of multimodal images. These three types of multimodal image datasets are: remote sensing multimodal image dataset, computer vision multimodal image dataset, and medical multimodal image dataset. Typical images for each type of multimodal image pair are shown below. Figure 7 As shown.

[0100] All image pairs in the dataset were manually labeled with 15 to 20 matching landmarks, which can be used to evaluate the accuracy of registration. Furthermore, the dataset authors provided an affine transformation matrix H as the ground truth. The matching points obtained in this invention can be transformed using the affine transformation formula, and the residuals between the points and the ground truth can be calculated. Matches with residuals less than 5 pixels are considered true matches to test and evaluate the performance of this invention.

[0101] (1) Parameter setting experiment

[0102] The LGF used in this invention has two hyperparameters: the number of directions N. s and scale number N o N s and N o A larger value means the feature descriptor map contains more details, but also higher computational complexity. This invention uses circular patches to construct the descriptor, where r represents the radius of the circular patch. The size of the radius r determines how much information the descriptor contains. If it is too small, the information content will be insufficient. If it is too large, due to local nonlinear geometric distortions, it will inevitably contain too much error information. To obtain the best results, this invention needs to select an optimal N. s N o Therefore, three experiments were designed for these three parameters: NCM, CMR, and runtime. The experiments used a hybrid dataset consisting of 90 pairs of multimodal images, randomly selected from the three datasets mentioned above. The evaluation metrics were NCM, CMR, and runtime.

[0103] Tables 2, 3, and 4 present the results of the three experiments, with Table 2 showing the results at a fixed N. s =4, r=40, parameter N o Matching results at different times; Table 3 shows the results with N fixed. o =6, r=40, parameter N s Matching results at different times; Table 4 shows the results with N fixed. s =4, N o =6, matching results when parameter r is different.

[0104] Table 2

[0105]

[0106] In Table 2, as N o With the increase of N, both NCM and CMR improved, but the runtime also increased significantly. o When the value is greater than 6, increase N. o The improvement on NCM and CMR is not significant. Therefore, in order to balance the performance and efficiency of this invention, N is set... o =6.

[0107] Table 3

[0108]

[0109] In Table 3, when N s The performance of this invention is best when N = 4, so N is set to 4. s =4. It is known that r determines the amount of information.

[0110] Table 4

[0111]

[0112] In Table 4, NCM decreases with increasing r, while CMR and runtime increase with increasing r. Considering all three evaluation metrics, setting r = 40 is optimal. Based on the above experimental results, the optimal parameter settings can be obtained: N s =4,N o =6, r=40.

[0113] (2) Feature Extractor Testing Experiment

[0114] To test the performance of the feature extractor in this invention, it was abbreviated as LEFT (Local Energy-response Feature Transformation) in the experiment. LEFT was compared with the feature extractor modules of the original Accelerated Segment Test Feature Detector (FAST), the Co-occurrence Filtering Space Matching (CoFSM), the Position, Scale, and Orientation-Scale-Invariant Feature Transform (PSO-SIFT), the KAZE-Synthetic Aperture Rada (KAZE-SAR), and the Radiation-variation Insensitive Feature Transform (RIFT) algorithms. The feature extractor has four evaluation metrics: the number of matches N. c Repeatability (Rep), Matchability Ratio (N) c The repeatability stability ratio (Ratio(Rep)) and the number of matches N. c N represents the number of correctly matched feature points extracted. Repeatability Rep is N. c The ratio of the average number of feature points detected in the two matching images I1 and I2:

[0115]

[0116] Where, x i and y iLet N1 and N2 be the homogeneous coordinates of the i-th feature point in I1 and I2, respectively; H is the affine transformation matrix (Ground Truth); N1 and N2 are the number of feature points in I1 and I2, respectively; and N0 is the number of matchable feature points. Indicates satisfaction The number of matches.

[0117] Ratio(N c Ratio(Rep) and Ratio(Rep) are used to measure the stability of the feature extractor. They are defined as follows:

[0118]

[0119] Where N im Ratio(N) represents the total number of image pairs in the test dataset. c >100) represents N c The proportion of images with a repeatability Rep greater than 100; Ratio(Rep>10%) represents the proportion of images with a repeatability Rep greater than 10%.

[0120] The nonlinear intensity differences in multimodal images can lead to many feature points extracted by the extractor being insufficiently significant, failing to construct effective descriptors, and easily being discarded. Therefore, the repeatability of the feature extractor is very low, indicating that extracting effective feature points from multimodal images is a challenging task. The feature extractor was tested using the mixed dataset from the parameter setting experiment. The results of all extractors are shown in Table 5. Table 5 presents the test results of the feature extractors.

[0121] Table 5

[0122]

[0123]

[0124] Both LEFT and RIFT have good overall performance, but Rep and Ratio(N) have lower performance. c The higher N and Ratio(Rep) indicate that LEFT's feature extractor is more stable. c Slightly smaller than RIFT, but significantly better than other feature extractors. As shown in Table 5, directly using FAST to detect feature points in the original multimodal image results in poor overall performance due to the large modal differences. Building upon FAST, LEFT combines a local energy response model and a joint bilateral filter to transform the original image into an intermediate modal image. The intermediate modal image not only effectively removes noise but also retains sufficient common information, improving the similarity between images. Therefore, LEFT's detector exhibits the best overall performance.

[0125] (3) Feature matching test experiment

[0126] To test the feature matching performance of LEFT, it was compared with CoFSM, PSO-SIFT, KAZE-SAR and RIFT, and NCM, CMR and runtime were used as evaluation metrics.

[0127] Qualitative Results: Qualitative experiments were conducted on two typical image pairs from each class of multimodal image datasets, one of which underwent a rotational transformation. The results are as follows: Figure 8 As shown, the rotation degree of (b), (d), and (f) is 50°.

[0128] The PSO-SIFT and KAZE-SAR algorithms perform poorly in matching visible light and infrared image pairs and visible light cross-seasonal image pairs, with near-zero NCM and CMR, due to the low texture detail and quality of these images. PSO-SIFT has the smallest NCM, particularly in optical cross-time image pairs, visible light and infrared image pairs, and optical cross-seasonal image pairs. PSO-SIFT uses the extreme points on the differences of the Gaussian pyramids of the images as feature points, but this method struggles to find a large number of highly repetitive keypoints in multimodal images. The KAZE-SAR algorithm uses the KAZE detector to improve the accuracy of feature point localization and achieves good results on map and optical image pairs, PD-T2 image pairs, and retinal image pairs. However, in... Figure 8 In (c) and (d), its (NCM,CMR) are (1,8.3%) and (0,0) respectively, indicating that the modeling method for discrete stochastic second-order nonlinear partial differential equations still cannot effectively handle differences in nonlinear intensity. CoFSM and RIFT algorithms achieved good matching results in all image pairs, but still lag behind LEFT.

[0129] Quantitative results: Quantitative experiments were conducted on three types of multimodal datasets, including 16 types and 164 pairs of multimodal images. Figure 9Line graphs show the NCM, CMR, and runtime for each algorithm. The vertical axis represents the type of multimodal image pair, from left to right: SAR-optical (SO), Remote sensing Cross-Season (RCS), Remote Sensing Day-Night (RDN), Depth-Optical (DO), Infare-Optical (IO), Map-Optical (MO), Optical-Optical (OO), Computervision Cross-Season (CCS), and Computervision Day-Night (CCS). The dataset includes Day-Night (CDN), Visible-Near-infrared (VN), Visible-Infare (VI), MRI-PET (MP), PD-T1 (PT1), PD-T2 (PT2), Retinal (Ret), SPECT-CT (SC), and MRI T1-T2 (T1T2) image pairs. Image pairs 1-7 are from a remote sensing multimodal image dataset, pairs 8-11 from a computer vision multimodal image dataset, and pairs 12-17 from a medical multimodal image dataset. Figure 9 In (a), CoFSM exhibits high NCM in some remote sensing multimodal image pairs, particularly in RDN, IO, and MO, with average NCMs of 362.8, 826.3, and 426.3, respectively. CoFSM effectively increases the number of feature points by constructing the image scale space using the co-occurrence filter algorithm and extracting feature points in the new scale space using the optimized image gradient. However, in VN, the NCM of CoFSM decreases significantly, mainly due to the greater complexity and disorder of information in VN. The NCMs of CoFSM for MP and SC are 1.8 and 8.7, respectively, and the CMRs are 17.43% and 7.151%, respectively, because the information contained in MP and SC differs greatly. Furthermore, CoFSM generally has a high runtime, especially in CDN and VN, where the complex and discontinuous texture information leads to computational complexity in constructing the scale space of the multi-layer co-occurrence filter.

[0130] KAZE-SAR performs best in VN, with NCM = 1035 and CMR = 99.86%. It also performs well on PD-T2, retinal, MO, and OO image pairs. This is because KAZE-SAR uses discrete stochastic second-order nonlinear partial differential equations to simulate the edge structure of the image and uses nonlinear diffusion filtering to minimize noise. However, the CMR values ​​show that KAZE-SAR has poor stability and versatility. If the image appearance varies greatly, or if the image structure information is scarce and unclear, KAZE-SAR's performance degrades; its NCM and CMR are close to zero in IO, DN, VI, MP, and SC. PSO-SIFT outperforms KAZE-SAR overall. PSO-SIFT mainly utilizes the Sobel algorithm to redefine the image gradient to overcome differences in nonlinear intensity. It achieves good results in IO, OO, VN, and PDT2, with NCM and CMR greater than 100% and 50%, respectively. However, if noise and local nonlinear deformation are severe, it will lead to a decrease in the performance of the algorithm or even failure. For example, its NCM in MP and SC is close to zero.

[0131] RIFT successfully matched all images and achieved the highest NCM in RCS. Furthermore, RIFT outperformed CoFSM in CMR across SO, RCS, DO, OO, and all medical images. RIFT detects feature points through phase consistency and constructs descriptors based on a maximum index map designed from LGF convolutional image sequences. RIFT demonstrates good versatility for multimodal image matching and exhibits better robustness to nonlinear intensity differences than traditional gradient-based methods. Figure 9 It can be seen that RIFT and CoFSM outperform PSO-SIFT and KAZE-SAR because both propose countermeasures against differences in nonlinear intensity.

[0132] Table 6 shows the average NCR and NCM of LEFT, CoFSM, and RIFT for all image pairs.

[0133] Table 6

[0134]

[0135] The results show that LEFT has a significant advantage over CoFSM and RIFT in CMR. Because LEFT extracts and describes features based on a local energy response model, it largely preserves common information and details, and employs a joint bilateral filter and a 3D Gaussian-like kernel to suppress noise and local nonlinear geometric distortions. In most image pairs, LEFT outperforms RIFT overall. Although CoFSM has higher NCM in some remote sensing image pairs and VN image pairs, its accuracy is lower than LEFT, and its generalization ability is weaker. Its performance deteriorates significantly, especially for medical image pairs.

[0136] (4) Test of rotation invariance

[0137] LEFT achieves rotation invariance and its performance has been compared with other algorithms.

[0138] Comparative experiment: Figure 8 Tables (b), (d), and (f) show the qualitative matching results for rotated multimodal images, with LEFT showing the best performance. Table 7 shows the NCM and CMR for CoFSM, RIFT, and LEFT across all 50° rotated image pairs. Figure 9 (d)-(f) and Table 7 show the quantitative results of matching after rotating all image pairs by 50°.

[0139] Table 7

[0140]

[0141] exist Figure 9 In (d)-(f), all algorithms show varying degrees of decrease in NCM and CMR. CoFSM maintains its numerical advantage in NCM across RCS, DO, MO, OO, and CDN, but LEFT performs better for the remaining image pairs. In contrast, RIFT's performance is significantly reduced because its angle traversal strategy can only handle six uniformly distributed angles within the range [0°, 180°]. Therefore, when the image is rotated by 50°, the closest angle it can traverse is 30°, still resulting in a 20° error compared to 50°, which leads to a decrease in RIFT's matching accuracy and number. Furthermore, RIFT can only handle images with rotation angles within the range [0°, 180°]; performance deteriorates further if the angle exceeds this range. The CMR of the KAZE-SAR and PRO-SIFT algorithms decreases significantly in rotated images, resulting in worse performance compared to the other three algorithms. LEFT shows little change in CMR and NCM, still maintaining a clear advantage. CoFSM remains very time-consuming, but LEFT has almost the same runtime as RIFT for all image pairs except those in CDN. Both qualitative and quantitative results show that LEFT is robust to rotation invariance and outperforms other algorithms.

[0142] Angular Robustness Test: To test the robustness of LEFT to different rotation angles, 24 rotation angles were uniformly selected from [0°, 360°] at 15° intervals, resulting in θ = [0°, 15°, 30°, ..., 360°]. LEFT was used to match image pairs rotated by θ, and the mean NCM, CMR, and runtime were recorded for each angle. Results are as follows: Figure 9 As shown in (g)-(i), the horizontal axis represents the rotation angle, and the vertical axis represents the average NCM, average CMR, and average runtime, respectively. Although the NCM, CMR, and runtime are different for each angle, their values ​​vary only within a small range. The NCM and CMR for all angles are greater than 50% and 80%, respectively, and the runtime is less than 25 seconds, indicating that LEFT is robust to image rotation in the range of [0°, 360°].

[0143] refer to Figure 10 In some embodiments, a multimodal image matching system based on local energy response feature transformation is also provided, which implements the steps of a multimodal image matching method based on local energy response feature transformation, including the following modules:

[0144] Image acquisition module 1 is used to acquire two multimodal images to be matched;

[0145] Local energy response model building module 2 is used to detect feature points in two images using the SIFT algorithm and to build feature descriptors;

[0146] Initial matching set construction module 3 is used as the basis for feature extraction and description;

[0147] Feature extraction module 4 is used to extract feature points with high repeatability and robustness from two multimodal images;

[0148] Feature description module 5 is used to construct a fully robust and rotation-invariant descriptor for the extracted feature points in order to facilitate the establishment of initial matching;

[0149] Initial matching set construction module 6 is used to construct an initial matching set based on the similarity of feature descriptors in two multimodal images;

[0150] The erroneous match removal module 7 is used to accurately remove erroneous matches present in the initial match and establish a more accurate matching relationship;

[0151] This invention addresses various challenges in multimodal image matching by establishing a robust multimodal image matching method based on local energy response feature transformation. The fundamental principle is to transform multimodal images into intermediate modes with more shared feature information, reducing modal differences and increasing similarity. Feature extraction, description, and matching are then performed based on this transformation. First, a local energy response model is established using a multi-scale, multi-directional two-dimensional log-Gabor filter, yielding local energy response functions (energy) in multiple o-directions. o In feature extraction, energy is measured in all directions. o The total energy response (ET) is obtained through summation, denoising, weighting, and normalization. Guided by ET, a joint bilateral filter is used to filter the multimodal image, and the difference between this filter and ET yields the intermediate modal image. Feature extraction is then performed using the FAST algorithm on the intermediate modal image. In the feature description, a three-dimensional Gaussian kernel is used to analyze the energy response in multiple directions. o Convolution is performed, and then the final feature descriptor map is obtained through a maximum indexing strategy. Log-polar coordinate descriptors are then built based on this map. Initial matching is then established based on the similarity of the descriptors. Finally, a fast sample consensus algorithm is used to remove incorrect matches and build a more accurate set of correct matches.

[0152] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0153] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the unit claims listing several systems, these systems may be embodied by the same hardware item. The use of the terms "first," "second," and "third," etc., does not indicate any order and can be interpreted as identifiers.

[0154] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method of multi-modal image matching by local energy response feature transformation, characterized by, Includes the following steps: S1. Obtain the two multimodal images to be matched; S2. Establish a local energy response model based on a log-Gabor filter; S3. Based on the local energy response model, the FAST algorithm is used in conjunction with a joint bilateral filter to extract feature points from two multimodal images; S4. Based on the local energy response model, a logarithmic polar coordinate feature descriptor for the feature points extracted in S3 is established using a three-dimensional Gaussian kernel and a maximum index strategy. S5. Establish an initial matching set based on the similarity of feature descriptors; S6, obtaining the affine transformation consistency index of the matching points in the initial matching set through a fast sample consistency algorithm , according to and the threshold relationship from the initial matching set to remove the error matching, get the final image matching result; Step S2 is as follows: S21. Establish the frequency domain expression of the log-Gabor filter: （1） Where f is the frequency of the log-Gabor filter. The center frequency of the log-Gabor filter. Let θo be the direction angle of the log-Gabor filter, and θso be the principal direction angle of the log-Gabor filter. This represents the frequency bandwidth of the log-Gabor filter. This is the angular bandwidth of the log-Gabor filter. The wavelet scale representing the log-Gabor filter, This represents the direction of the log-Gabor filter; S22. Derive the spatial domain expression of the log-Gabor filter from its frequency domain formula: （2） in and These are the even-symmetric and odd-symmetric wavelets of the log-Gabor filter, respectively. In terms of formula structure... and is the real and imaginary parts of the log-Gabor filter in scale and direction, where x represents the horizontal coordinate in the spatial domain, y represents the vertical coordinate in the spatial domain, s represents the wavelet scale of the log-Gabor filter, and o represents the direction of the log-Gabor filter. S23, Input image Convolution with odd-symmetric wavelet yields a given scale and direction wavelet response components Input image Convolution with even-symmetric wavelet yields a given scale. and direction wavelet response components : （3） Where x is the abscissa in the spatial domain and y is the ordinate in the spatial domain; S24, using wavelet response components and Get direction Energy response on: （4） in, It is a point on the image Local energy response in the direction, It is a wavelet response component The sum at all scales It is a wavelet response component The sum across all scales; S25. Based on the energy response in the direction To obtain points on the image The total local energy response in all directions.

2. The multimodal image matching method based on local energy response feature transformation according to claim 1, characterized in that, The multimodal image types in step S1 include: remote sensing multimodal images, computer vision multimodal images, and medical multimodal images.

3. The multimodal image matching method based on local energy response feature transformation according to claim 1, characterized in that, Step S25 is as follows: Introducing a noise compensation term To remove Stray response to noise; Add a weighting function to the extended frequency distribution ; Normalized local energy response; After the above processing, the overall local energy response can be obtained: （5） （6） in, Represents wavelet response components and Amplitude in scale and direction, It is the sum of amplitudes in all directions and scales. It is a small constant.

4. The multimodal image matching method based on local energy response feature transformation according to claim 3, characterized in that, Step S3 is as follows: S31, Using Total Local Energy Response Guided joint bilateral filter on input image The filtered image is obtained by performing filtering processing. ; （7） in, and Indices representing pixel coordinates pixels in the image spatial domain The set of adjacent pixels, For normalized operands, ET( j (pixels) The total local energy response at the point, ET( i (pixels) The total local energy response at the point, || ji || 2 For pixels and The distance between them, ||ET( j )-ET( i )‖ 2 For ET ( j ) and ET ( i The differences between them; （8） It is a spatial distance function. This is the structure guiding function. and The spatial weights and structural weights of the joint bilateral filter are controlled separately; S32. Obtain the filtered image. With total local energy response Difference plot For the final feature extraction map: （9） Using the FAST feature point extraction algorithm Feature points are extracted from the top.

5. The multimodal image matching method based on local energy response feature transformation according to claim 4, characterized in that, Step S4 is as follows: S41, according to A three-dimensional Gaussian-like kernel is obtained, and the result of processing the three-dimensional Gaussian-like kernel is normalized: （10） （11） in, The energy response is processed by a three-dimensional Gaussian kernel. After normalization , The Gaussian convolution kernel for the energy response layer in the direction, It is a convolution kernel in the directional dimension. for and A two-dimensional Gaussian kernel on a plane. It is a small constant. This represents the number of all directions of the log-Gabor filter; S42, Energy Response Sorting along the directional dimension yields a three-dimensional multi-channel information mapping. Calculate using the maximum index strategy Maximum direction index corresponding to all pixels in Using all pixels The resulting maximum index map serves as the final feature description map. S43. Based on the feature description map, construct a descriptor for each extracted feature point. The feature descriptor uses a logarithmic polar coordinate structure and uses the principal direction of the feature point to achieve rotation invariance of the matching algorithm.

6. The multimodal image matching method based on local energy response feature transformation according to claim 5, characterized in that, In step S43, the rotation invariance of the matching algorithm is achieved using the principal direction of the feature points as follows: Using feature extraction maps Extract the principal direction of gradient change for each feature point, which serves as the principal direction of patch rotation for the descriptor, and calculate the vertical and horizontal gradient maps of the image: （12） in, , and These represent the feature description maps at different locations. The horizontal gradient, vertical gradient, and pixel value at that location. pixel gradient amplitude at and gradient direction angle Defined as: （13） Draw a circle with the feature point as the center and a radius of... The circular patches were divided into equal parts. For each angle interval, calculate the sum of the pixel gradient amplitudes in that interval. Using each angle as the horizontal axis and the sum of the gradient magnitudes in each angle interval as the vertical axis, a histogram is calculated. The direction with the largest vertical axis is selected as the principal direction of the feature point. The feature descriptor patches are rotated according to the principal direction and then matched to achieve rotation invariance.

7. The multimodal image matching method based on local energy response feature transformation according to claim 1, characterized in that, Step S5 is as follows: For each feature point, a descriptor is constructed, and a one-to-one initial match is established using nearest neighbor matching and Euclidean distance similarity metrics.

8. The multimodal image matching method based on local energy response feature transformation according to claim 1, characterized in that, Step S6 is as follows: The Fast Sample Consensus algorithm is used to remove false matches, and an affine transformation matrix is ​​established between the two multimodal images to be matched. Combined with the matching points in the initial matching set and Establish affine transformation consistency index : when Satisfying the following formula, the first The following matches were determined to be correct: in The threshold is used to remove incorrect matches that do not satisfy the above relationship.

9. A multimodal image matching system based on local energy response feature transformation, characterized in that, The steps for implementing the multimodal image matching method for local energy response feature transformation as described in any one of claims 1-8 include the following modules: The image acquisition module is used to acquire two multimodal images to be matched; A local energy response model building module is used as the basis for feature extraction and description. The feature extraction module is used to extract feature points with high repeatability and robustness from two multimodal images; The feature description module is used to construct a fully robust and rotation-invariant descriptor for the extracted feature points in order to facilitate the establishment of initial matching. The initial matching set construction module is used to construct an initial matching set based on the similarity of feature descriptors in two multimodal images; The error matching removal module is used to accurately remove erroneous matches in the initial matching and establish more accurate matching relationships.

Images