Highly adaptable multimodal image matching method based on abstract spatial structure features

Abstract

The present invention discloses a highly adaptable multimodal image matching method based on abstract spatial structural features. The method first uses training set images to train a regressor, calculates imaging feature vectors based on internal statistical information of the image, and uses the trained regressor to regress the optimal multiscale filtering parameter combination of the image, and performs multiscale filtering on the image; then, the image filtering results are used to extract the abstract structural feature points of the image in combination with moment analysis and principal component analysis; finally, according to the filtering results, the abstract structural features of the feature points are described in combination with a logarithmic polar coordinate window, and a high-precision matching of the image is achieved in combination with a coarse-to-fine matching strategy. The present invention uses a particle swarm algorithm to find the optimal match of an image pair, and then obtains the filtering parameters under the optimal match, which not only avoids manual adjustment of the filtering parameters and greatly reduces the workload, but also achieves higher matching accuracy and matching accuracy, and has more powerful generalization performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital image processing, and in particular relates to a highly adaptable multimodal image matching method based on abstract spatial structure features. Background Art

[0002] Multimodal image matching refers to the process of identifying the same-name image points between different modal images through matching algorithms, covering multiple application fields such as remote sensing, computer vision, medicine and art. In the process of multimodal image matching, due to different imaging sensors (such as optical, infrared, SAR), imaging conditions (such as day and night, cross-season) or data types (such as paintings, maps), in addition to solving geometric differences such as translation, rotation, and scaling between images, it is also necessary to deal with significant nonlinear radiation differences between images. Although traditional feature matching can effectively solve the geometric differences between images, it usually calculates the gradient of the image as a preprocessing step. In the presence of nonlinear radiation differences, the gradient size and direction of the same-name point area of ​​different modal images are often very different, which makes the traditional matching method usually fail when applied to multimodal images. Therefore, the nonlinear radiation difference between images has become the main bottleneck of multimodal image matching, and how to robustly weaken the nonlinear radiation difference between different modal images has become an important factor in determining the generalization and adaptability of multimodal image matching methods.

[0003] Traditional radiation correction methods often fail to effectively eliminate the nonlinear radiation differences of random cross-modal image combinations. Therefore, mainstream multimodal matching methods usually use high-pass filtering to preprocess images to reduce image noise and retain image texture edges, thereby weakening the negative impact of radiation differences on matching performance. However, the types of images potentially processed by multimodal image registration are extremely diverse, making it difficult for filters with single parameter configurations to maintain a robust filtering effect in different registration scenarios. Without debugging and manual intervention, simply using fixed empirical parameters to filter multimodal images may not produce good filtering effects. In this case, the image matching error of a specific modality combination may be very serious or even completely fail, resulting in very few matching points or obviously erroneous transformation estimates. Therefore, studying filtering strategies that do not require parameter adjustment and are highly adaptable can effectively improve the generalization ability of multimodal matching methods.

[0004] Due to the huge differences in imaging mechanisms, local or detailed features of images of different modalities are likely to be non-repeatable. For the feature points in these areas, simply matching based on similarity measures will result in a high error rate. To solve this problem, traditional matching methods usually extract feature points uniformly across the entire image, and then use methods such as RANSAC to remove gross errors. However, this processing method can easily produce too many false matches when the nonlinear radiation differences are large, which seriously challenges the robustness of the gross error removal algorithm and leads to matching failure. In order to avoid the above situation, it is necessary to pre-screen points in the feature point extraction stage to ensure that the points that are successfully matched account for as high a proportion as possible. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a highly adaptable multimodal image matching method based on abstract spatial structure features, so as to solve the problems of low generalization performance and low degree of automation in the multimodal image matching method.

[0006] In order to achieve the above object, the technical solution provided by the present invention is a highly adaptable multimodal image matching method based on abstract spatial structure features, comprising the following steps:

[0007] Step 1, find the filter parameters under the optimal matching of the training set image;

[0008] Step 2, using the filter parameters and internal statistical features of the image under the optimal match of the training set image found in step 1 to train the regressor;

[0009] Step 3, using the regressor trained in step 2 to infer the filter parameters under the optimal matching of the image, and using the optimal filter parameters to filter the image;

[0010] Step 4, extracting abstract structural feature points based on the image filtering results;

[0011] Step 5, based on the image filtering result, feature description is performed on the abstract structural feature points of the image;

[0012] Step 6: Use a coarse-to-fine matching strategy to achieve accurate matching of the image to the abstract structural feature points according to the abstract structural features of the image.

[0013] Moreover, in step 1, several pairs of images are selected from the public data set as training sets and test sets of the regressor, all training set images are filtered using a multi-scale filter group and then matched, the matched image pairs are manually measured for checkpoints, the manually measured checkpoints are used to calculate the geometric transformation model to evaluate the matching effect, and the particle swarm algorithm is used to find the best match for each image, thereby obtaining the corresponding filter parameters under the best match.

[0014] The objective function of particle swarm optimization is:

[0015]

[0016] in,

[0017]

[0018] G TM ={(X L ,X R )} (3)

[0019]

[0020] In the formula, score k It represents the score obtained by matching the image using the kth set of filter parameters. and represents the affine transformation model and internal accuracy calculated from the manually measured checkpoints, G TM is the set of all matching points, X L is the feature point on the left image, X R is the feature point on the right image, card(·) represents the count of all elements in the set, G CM is the set of all correct matching points, assuming that X L After passing the matrix After transformation and X R The distance does not exceed The point is the correct matching point, (X′ L ,X′ R ) indicates the correct matching point.

[0021] The multi-scale filter bank is constructed using Log-Gabor filter, which is defined as:

[0022]

[0023] In the formula, represents the Log-Gabor filter, f and θ are the polar coordinate representations of the frequencies in two directions in the two-dimensional frequency domain of the image after Fourier transform, σ f , σ θ is the Gaussian standard deviation of the transfer function of the Log-Gabor filter in two directions in the frequency domain, ω is the wavelet wavelength, is the filter direction.

[0024] Based on the Log-Gabor filter, a set of filters with equal scale and equal direction are constructed as follows:

[0025]

[0026] In the formula, LGFS(S,O,μ,σ f ) represents the filter bank, S and O represent the number of layers of the filter bank and the direction of each layer, s and o represent the index values ​​of the number of layers and directions, μ is the multiplier, ω is the wavelength of the wavelet, is the filter direction, σ f , σ θ is the Gaussian standard deviation of the transfer function of the Log-Gabor filter in both directions in the frequency domain.

[0027] Furthermore, in step 2, the internal statistical parameters of each image in the training set are calculated and used as the input of the regressor, and the regressor is trained with the filter parameters corresponding to the optimal match obtained in step 1 as the optimization target.

[0028] Moreover, in step 3, for each image to be matched in the test set, the internal statistical parameters of the image are calculated and input into the regressor trained in step 2 to regress the filter parameters under the optimal match of the image, and the filter parameters are used to perform multi-scale filtering on the image.

[0029] Moreover, in step 4, according to the classical moment analysis equation, the multi-scale moment of the image is constructed using the multi-scale filtering result of the image in step 3, and then the multi-scale moment of the image is subjected to principal component analysis to obtain the abstract structure feature map of the image, and the feature points on the abstract structure feature map are extracted using the feature point extraction algorithm; the moment analysis equation is defined as follows:

[0030]

[0031] in:

[0032]

[0033]

[0034]

[0035] Where M s It represents the moment calculated by filtering the image with the filter of scale s in the LGFS filter group, a s 、b s 、c s is the intermediate quantity of the Taylor expansion of the image, Represents the real part of the image filtered by the LGFS filter group. represents the direction of the filter, which is calculated by equation (6).

[0036] Moreover, in step 5, based on the filtering result obtained in step 3, the logarithmic polar coordinate window is used to extract the description information of the abstract structural features in step 4. The feature description information includes intensity and direction. For any pixel on the image I (x, y), the intensity A (x, y) and direction D (x, y) of the pixel are:

[0037]

[0038] D(x,y)=arctan(a(x,y),b(x,y)) (12)

[0039] in:

[0040]

[0041]

[0042] Where a(x,y) and b(x,y) are the sum of the real intensity components of the LGFS filtering results in the y and x directions, respectively. Represents the imaginary part of the image filtered by the LGFS filter group. represents the direction of the filter, which is calculated by equation (6).

[0043] The entire logarithmic polar coordinate window is divided into 30° sectors, and the entire window is divided into 25 sectors. The direction histograms of 12 directions are counted for each sector, and finally a 156-dimensional feature descriptor is obtained. When constructing the logarithmic polar coordinate descriptor, the direction histograms of the two centrally symmetric sectors are counted together.

[0044] Moreover, in step 6, the feature points are matched using the minimum Euclidean distance as the measurement. After the matching is completed, the error is eliminated, and the initial transformation model A between the two images is obtained from the correct matching points. o , after getting A o After that, the Euclidean distance d′ with position constraints is used as the measure for enhanced matching, that is:

[0045]

[0046]

[0047]

[0048] In the formula, and are the coordinates of the feature points on the left and right images respectively, and are the feature point descriptors on the left and right images respectively, d(·) is the Euclidean distance between feature descriptors, p1 and p2 are penalty coefficients, and σ is the standard deviation of the error distribution between the matching position and the true value.

[0049] Compared with the prior art, the present invention has the following advantages:

[0050] The present invention uses a particle swarm algorithm to find the optimal match of an image pair, and then obtains the filtering parameters under the optimal match. This not only avoids manual adjustment of the filtering parameters and greatly reduces the workload, but also achieves higher matching accuracy and matching correctness, and has more powerful generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a technical flow chart of an embodiment of the present invention.

[0052] Figure 2 It is a schematic diagram of calculating the abstract structure feature map of an image according to an embodiment of the present invention.

[0053] Figure 3 It is a schematic diagram of acquiring abstract structural feature points of an image according to an embodiment of the present invention.

[0054] Figure 4 It is a schematic diagram of describing the abstract structural features of an image according to an embodiment of the present invention.

[0055] Figure 5 It is a schematic diagram of matching results of an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention provides a highly adaptable multimodal image matching method based on abstract spatial structure features. The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0057] like Figure 1 As shown, the present invention provides a highly adaptable multimodal image matching method based on abstract spatial structure features, comprising the following steps:

[0058] Step 1: Find the filter parameters that best match the training set images.

[0059] Several pairs of images are selected from the public dataset as the training and test sets of the regressor. All the training set images are filtered using a multi-scale filter group and then matched. The matched image pairs are manually measured and checked. The geometric transformation model is calculated using the manually measured checkpoints to evaluate the matching effect. The particle swarm algorithm is used to find the best match for each image, and then the corresponding filter parameters S, O, μ and σ under the best match are obtained. f .

[0060] The objective function of particle swarm optimization is:

[0061]

[0062] in,

[0063]

[0064] G TM ={(X L ,X R )} (3)

[0065]

[0066] In the formula, score k It represents the score obtained by matching the image using the kth set of filter parameters. and represents the affine transformation model and internal accuracy calculated from the manually measured checkpoints, G TM is the set of all matching points, X L is the feature point on the left image, X R is the feature point on the right image, card(·) represents the count of all elements in the set, G CM is the set of all correct matching points, assuming that X L After passing the matrix After transformation and X R The distance does not exceed The point is the correct matching point, (X′ L ,X′ R ) indicates the correct matching point.

[0067] This embodiment uses Log-Gabor filter to construct a multi-scale filter bank. Other filters can also be used according to actual conditions. The definition of Log-Gabor filter is:

[0068]

[0069] In the formula, represents the Log-Gabor filter, f and θ are the polar coordinate representations of the frequencies in two directions in the two-dimensional frequency domain of the image after Fourier transform, σ f , σ θ is the Gaussian standard deviation of the transfer function of the Log-Gabor filter in two directions in the frequency domain, ω is the wavelet wavelength, is the filter direction.

[0070] Based on the Log-Gabor filter, a set of filters with equal scale and equal direction are constructed as follows:

[0071]

[0072] In the formula, LGFS(S,O,μ,σ f ) represents the filter bank, S and O represent the number of layers of the filter bank and the direction of each layer, s and o represent the index values ​​of the number of layers and directions, μ is the multiplier, ω is the wavelength of the wavelet, is the filter direction, σ f , σ θ is the Gaussian standard deviation of the transfer function of the Log-Gabor filter in both directions in the frequency domain.

[0073] Before image matching, filtering the affected pairs with matching is equivalent to preprocessing and enhancing the image. The effect of this preprocessing enhancement will affect the final matching effect. The quality of the preprocessing effect is highly correlated with the different parameter configurations of the filter. The particle swarm algorithm is used to select the best image according to the score. k Find the best match for each image, and thus obtain the corresponding filter parameters under the best match.

[0074] Step 2: Use the filter parameters and internal statistical features of the image under the optimal match of the training set image found in step 1 to train the regressor.

[0075] Calculate the internal statistical parameters of each image in the training set, use them as the input of the regressor, and train the regressor with the filter parameters corresponding to the optimal match obtained in step 1 as the optimization target. This embodiment uses a support vector regressor, and other regressors can also be used according to actual conditions; the image internal statistical parameters calculated in this embodiment are the BRISQUE features of the image, and other features can also be used according to actual conditions.

[0076] Step 3: Use the regressor trained in step 2 to infer the filter parameters under the optimal match of the image, and use the optimal filter parameters to filter the image.

[0077] For each image to be matched in the test set, calculate its internal statistical parameters and input them into the regressor trained in step 2 to regress the filter parameters under the optimal match of the image, and use the filter parameters to perform multi-scale filtering on the image.

[0078] Step 4: extract abstract structural feature points based on the image filtering results.

[0079] like Figure 2-4 As shown, according to the classical moment analysis equation, the multi-scale moment of the image is constructed using the multi-scale filtering result of the image in step 3, and then the multi-scale moment of the image is subjected to principal component analysis to obtain the abstract structure feature map of the image, and the feature points on the abstract structure feature map are extracted using the feature point extraction algorithm.

[0080] The moment analysis equation is defined as follows:

[0081]

[0082] in:

[0083]

[0084]

[0085]

[0086] Where M s It represents the moment calculated by filtering the image with the filter of scale s in the LGFS filter group, a s , b s 、c s is the intermediate quantity of the Taylor expansion of the image, It represents the real part of the image filtered by the LGFS filter group, o is the index value of the filter direction, represents the direction of the filter, which is calculated by equation (6).

[0087] The feature point extraction algorithm performed on the abstract structure feature map in this embodiment is the AGAST algorithm, and other feature point extraction algorithms may also be used according to actual conditions.

[0088] Step 5: Describe the abstract structural feature points of the image based on the image filtering results.

[0089] Based on the filtering results obtained in step 3, the logarithmic polar coordinate window is used to extract the descriptive information of the abstract structural features in step 4. The feature description information includes intensity and direction, which is specifically defined as follows: for any pixel on the image I(x,y), the intensity A(x,y) and direction D(x,y) of the pixel are:

[0090]

[0091] D(x,y)=arctan(a(x,y),b(x,y)) (12)

[0092] in:

[0093]

[0094]

[0095] Where a(x,y) and b(x,y) are the sum of the real intensity components of the LGFS filtering results in the y and x directions, respectively. It represents the imaginary part of the image filtered by the LGFS filter group, s and o represent the index values ​​of the layer number and direction respectively. represents the direction of the filter, which is calculated by equation (6).

[0096] The entire logarithmic polar coordinate window is divided into 30° sectors, and the entire window is divided into 25 sectors. The direction histograms of 12 directions are counted for each sector, and finally a 156-dimensional feature descriptor is obtained. When constructing the logarithmic polar coordinate descriptor, the direction histograms of the two centrally symmetric sectors are counted together.

[0097] Step 6: Use a coarse-to-fine matching strategy to achieve accurate matching of the image to the abstract structural feature points according to the abstract structural features of the image.

[0098] The feature points are matched using the minimum Euclidean distance as the measure. After the matching is completed, the error is eliminated and the initial transformation model A between the two images is obtained from the correct matching points. o , after getting A o After that, the Euclidean distance d′ with position constraints is used as the measure for enhanced matching, that is:

[0099]

[0100]

[0101]

[0102] In the formula, and are the coordinates of the feature points on the left and right images respectively, and are the feature point descriptors on the left and right images respectively, d(·) is the Euclidean distance between feature descriptors, p1 and p2 are penalty coefficients, and σ is the standard deviation of the error distribution between the matching position and the true value.

[0103] In specific implementation, the method proposed in the technical solution of the present invention can be implemented by technical personnel in this field using computer software technology to realize automatic operation process. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of the present invention and computer equipment running the corresponding computer program, should also be within the scope of protection of the present invention.

[0104] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific implementation cases described, or replace them with similar methods, without departing from the spirit of the present invention or exceeding the scope defined by the appended claims.

Claims

1. A highly adaptable multimodal image matching method based on abstract spatial structure features, characterized in that: The following steps are involved: Step 1, find the filter parameters under the optimal matching of the training set image; Step 2, using the filter parameters and internal statistical features of the image under the optimal match of the training set image found in step 1 to train the regressor; Step 3, using the regressor trained in step 2 to infer the filter parameters under the optimal matching of the image, and using the optimal filter parameters to filter the image; Step 4, extracting abstract structural feature points based on the image filtering results; According to the classical moment analysis equation, the multi-scale moment of the image is constructed using the multi-scale filtering result of the image in step 3, and then the multi-scale moment of the image is subjected to principal component analysis to obtain the abstract structure feature map of the image, and the feature points on the abstract structure feature map are extracted using the feature point extraction algorithm; Step 5, based on the image filtering result, feature description is performed on the abstract structural feature points of the image; Step 6: Use a coarse-to-fine matching strategy to achieve accurate matching of the image to the abstract structural feature points according to the abstract structural features of the image.

2. The highly adaptable multimodal image matching method based on abstract spatial structure features according to claim 1, characterized in that: In step 1, several pairs of images are selected from the public dataset as the training set and test set of the regressor. All the training set images are filtered using a multi-scale filter group and then matched. The matched image pairs are manually measured and checked. The geometric transformation model is calculated using the manually measured checkpoints to evaluate the matching effect. The particle swarm algorithm is used to find the best match for each image, and then the corresponding filter parameters under the best match are obtained.

3. The highly adaptable multimodal image matching method based on abstract spatial structure features as claimed in claim 2, characterized in that: The objective function of particle swarm optimization in step 1 is: in, G TM ={(X L ,X R )}(3) In the formula, score k It represents the score obtained by matching the image using the kth set of filter parameters. and represents the affine transformation model and internal accuracy calculated from the manually measured checkpoints, G TM is the set of all matching points, X L is the feature point on the left image, X R is the feature point on the right image, card(·) represents the count of all elements in the set, G CM is the set of all correct matching points, assuming that X L After passing the matrix After transformation and X R The distance does not exceed The point is the correct matching point, (X′ L ,X′ R ) indicates the correct matching point.

4. The highly adaptable multimodal image matching method based on abstract spatial structure features as claimed in claim 3, characterized in that: In step 1, a multi-scale filter bank is constructed using Log-Gabor filter, which is defined as: In the formula, represents the Log-Gabor filter, f and θ are the polar coordinate representations of the frequencies in two directions in the two-dimensional frequency domain of the image after Fourier transform, σ f , σ θ is the Gaussian standard deviation of the transfer function of the Log-Gabor filter in two directions in the frequency domain, ω is the wavelet wavelength, is the filter direction.

5. The highly adaptable multimodal image matching method based on abstract spatial structure features as claimed in claim 4, characterized in that: In step 1, a set of filters with equal scale and equal direction are constructed based on Log-Gabor filter: In the formula, LGFS(S,O,μ,σ f ) represents the filter bank, S and O represent the number of layers of the filter bank and the direction of each layer, s and o represent the index values ​​of the number of layers and directions, μ is the multiplier, ω is the wavelength of the wavelet, is the filter direction, σ f , σ θ is the Gaussian standard deviation of the transfer function of the Log-Gabor filter in both directions in the frequency domain.

6. The highly adaptable multimodal image matching method based on abstract spatial structure features according to claim 1, characterized in that: In step 2, the internal statistical parameters of each image in the training set are calculated and used as the input of the regressor, and the regressor is trained with the filter parameters corresponding to the optimal match obtained in step 1 as the optimization target.

7. The highly adaptable multimodal image matching method based on abstract spatial structure features according to claim 1, characterized in that: In step 3, for each image to be matched in the test set, the internal statistical parameters of the image are calculated and input into the regressor trained in step 2 to regress the filter parameters under the optimal match of the image, and use the filter parameters to perform multi-scale filtering on the image.

8. The highly adaptable multimodal image matching method based on abstract spatial structure features as claimed in claim 1, characterized in that: The moment analysis equation in step 4 is defined as follows: in: Where M s It represents the moment calculated by filtering the image with the filter of scale s in the LGFS filter group, a s 、b s 、c s is the intermediate quantity of the Taylor expansion of the image, Represents the real part of the image filtered by the LGFS filter group. represents the direction of the filter, which is calculated by equation (6).

9. The highly adaptable multimodal image matching method based on abstract spatial structure features as claimed in claim 1, characterized in that: In step 5, based on the filtering result obtained in step 3, the logarithmic polar coordinate window is used to extract the description information of the abstract structural features in step 4. The feature description information includes intensity and direction. For any pixel on the image I(x,y), the intensity A(x,y) and direction D(x,y) of the pixel are: D(x,y)=arctan(a(x,y),b(x,y)) (12) in: Where a(x,y) and b(x,y) are the sum of the real intensity components of the LGFS filtering results in the y and x directions, respectively. Represents the imaginary part of the image filtered by the LGFS filter group. represents the direction of the filter, which is calculated by equation (6); The entire log-polar coordinate window is divided into 30° sectors. The entire window is divided into 25 sectors. The direction histograms of 12 directions are counted for each sector, and finally a 156-dimensional feature descriptor is obtained. When constructing the log-polar coordinate descriptor, the direction histograms of the two centrally symmetric sectors are counted together.

10. The highly adaptable multimodal image matching method based on abstract spatial structure features according to claim 1, characterized in that: In step 6, the feature points are matched using the minimum Euclidean distance as the measure. After the matching is completed, the error is eliminated and the initial transformation model A between the two images is obtained from the correct matching points. o , after getting A o After that, the Euclidean distance d′ with position constraints is used as the measure for enhanced matching, that is: In the formula, and are the coordinates of the feature points on the left and right images respectively, and are the feature point descriptors on the left and right images respectively, d(·) is the Euclidean distance between feature descriptors, p1 and p2 are penalty coefficients, and σ is the standard deviation of the error distribution between the matching position and the true value.

Images