Multimodal remote sensing image registration method based on domain self-adaption and optimizer set algorithm

Abstract

The invention discloses a multi-modal remote sensing image registration method based on domain self-adaption and an optimizer set algorithm, and relates to the field of image processing, and the method comprises the steps: respectively extracting the feature vectors of a reference image and an image to be registered, carrying out the feature matching of the feature vectors according to the class label containing conditions of the feature vectors, and obtaining a registration result; obtaining a feature matching vector based on a matching result; estimating transformation parameters corresponding to the affine transformation model by using a random sampling consistency algorithm, and optimizing the affine transformation model according to an estimation result and similarity measurement; and performing spatial transformation on a to-be-registered remote sensing image by using the optimized affine transformation model to realize spatial alignment between the reference image and the to-be-registered image. According to the method, firstly, the feature points of the multi-modal remote sensing image are extracted, the difference of feature vectors is reduced by using a domain adaptive method, the precision and robustness of feature matching are improved, transformation parameters are optimized through an optimizer set algorithm, and search is prevented from falling into local optimum.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm. Background Art

[0002] With the continuous emergence of new sensors, the remote sensing field has acquired a large number of multimodal images (such as visible light and near-infrared images) with different phases, different spatial resolutions and different spectral properties. These multimodal images reflect the different characteristics of ground objects and can provide complementary information for surface monitoring, but at the same time they also bring major challenges to image analysis. Image registration is one of the basic tasks of multimodal remote sensing image analysis. It is used to geometrically calibrate two or more images with overlapping areas, which has an important impact on subsequent applications. With the rapid development of remote sensing technology, multimodal remote sensing image registration has become a research hotspot.

[0003] Remote sensing image registration methods are primarily categorized into two types: feature-based and region-based. Feature-based registration methods extract feature information from images, such as points, lines, and surfaces, to perform registration. Feature point matching is the most widely used method, and common feature point extraction algorithms include SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), and KAZE. Due to the influence of nonlinear radiation differences and geometric distortion between multimodal remote sensing images, the distribution of feature vectors in remote sensing images of different modalities varies significantly, resulting in a small number of correctly matched point pairs and a high mismatch rate. Furthermore, due to the significant nonlinear grayscale differences between multimodal remote sensing images, region-based registration methods are prone to falling into local optima when searching for optimal transformation parameters, severely impacting the accuracy of image registration.

[0004] Currently, no effective solutions have been proposed for the problems in related technologies. Summary of the Invention

[0005] In response to the problems in the related art, the present invention proposes a multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm to overcome the above-mentioned technical problems existing in the existing related art.

[0006] To this end, the specific technical solutions adopted in the present invention are as follows:

[0007] A multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm includes:

[0008] Extract the feature vectors of the reference image and the image to be registered respectively, perform feature matching on the feature vectors according to the category label inclusion status of the feature vectors, and obtain the feature matching vector based on the matching results;

[0009] The transformation parameters corresponding to the affine transformation model are estimated using a random sampling consensus algorithm, and the affine transformation model is optimized based on the estimation results and the similarity measure;

[0010] The optimized affine transformation model is used to perform spatial transformation on the remote sensing image to be registered, so as to achieve spatial alignment between the reference image and the image to be registered.

[0011] Preferably, feature vectors of the reference image and the image to be registered are extracted respectively, feature matching is performed on the feature vectors according to the category label inclusion status of the feature vectors, and a feature matching vector is obtained based on the matching results, including:

[0012] According to the category label inclusion status of the feature vector, the domain adaptation method is used to optimize the difference status between the feature vectors, and the feature vectors are mapped to the reproducing kernel Hilbert space to quantify the mean difference;

[0013] A distance threshold is set and combined with the nearest neighbor method, feature matching is performed on the reference image and the image to be registered, and the matching result is used as the feature matching vector.

[0014] Preferably, according to the category label inclusion status of the feature vectors, a domain adaptation method is used to optimize the difference state between the feature vectors, and the feature vectors are mapped to the reproducing kernel Hilbert space to quantize the mean difference, including:

[0015] According to the judgment requirements, the category label inclusion status of the feature vector is judged, and when the feature vector does not contain the category, the edge distribution of the reference image and the image to be registered is judged respectively, and the feature vector is mapped to the reproducing kernel Hilbert space;

[0016] According to the mapping results, the maximum mean difference distance between the edge distributions of the reference image and the image to be registered is obtained, and the transformation matrix is ​​calculated to minimize the maximum mean difference distance between the edge distributions;

[0017] When the feature vector contains a category, the balanced distribution adaptive algorithm is used to calculate the optimal mapping function with the goal of minimizing the weighted distribution distance, and the feature vector is mapped to the reproducing kernel Hilbert space based on the calculation result;

[0018] According to the mapping results, the maximum mean difference distance between the conditional distribution of the reference image and the image to be registered is obtained, and the transformation matrix is ​​calculated to minimize the weighted maximum mean difference distance between the marginal distribution and the conditional distribution.

[0019] Preferably, the calculation formula for the maximum mean difference distance between the edge distributions of the reference image and the image to be registered is:

[0020]

[0021] Where, M(X R , X S ) represents the reference image edge distribution X R and the edge distribution X of the image to be registered S The maximum mean difference distance between them, n1 represents the edge distribution X of the reference image R The number of samples in represents the mapping function, x Ri represents the i-th eigenvector of the reference image, and n2 represents the edge distribution X of the image to be registered. S The number of samples in x Si represents the i-th eigenvector of the image to be registered, represents the reproducing kernel Hilbert space normal form.

[0022] Preferably, the calculation formula for the maximum mean difference distance between the conditional distribution of the reference image and the image to be registered is:

[0023]

[0024] Where, Dis β represents the maximum mean difference distance between the conditional distribution of the reference image and the image to be registered, C represents the number of categories, represents the feature vector belonging to category c in the reference image, represents the feature vector of category c in the image to be registered, x i represents the i-th feature vector in the feature vector of category c in the reference image, x j represents the jth eigenvector in the eigenvectors of category c in the image to be registered, represents the number of feature vectors belonging to category c in the reference image, Indicates the number of feature vectors belonging to category c in the image to be registered.

[0025] Preferably, the transformation parameters corresponding to the affine transformation model are estimated using a random sampling consistency algorithm, and the affine transformation model is optimized according to the estimation result and the similarity measure, including:

[0026] Randomly sampling some feature matching vectors to estimate the affine transformation model parameters, and testing the feature matching vectors using the affine transformation model parameters according to the estimation results;

[0027] Generate the transformation parameters of the affine transformation model based on the test results, integrate the optimizer set, and use the transformation parameters to initialize the optimizer set population;

[0028] The transformation parameters of the affine transformation model are optimized according to the population initialization results and the optimizer ensemble algorithm, and the majority voting method is used based on the optimization results to determine the highest-ranked solution as the final optimization result.

[0029] Preferably, generating transformation parameters of the affine transformation model based on the test results, integrating the optimizer set, and initializing the optimizer set population using the transformation parameters includes:

[0030] According to the test results, the feature matching vectors that satisfy the affine transformation model are selected as the inliers, and the feature matching vectors that do not satisfy the affine transformation model are selected as the outliers;

[0031] By repeating random sampling and estimating model parameters in an iterative manner, the affine transformation model with the largest number of inliers is selected as the final estimation result according to the iterative repetition results to obtain the estimated transformation parameters;

[0032] Several optimization algorithms are selected to integrate the optimizer set, and the population of the optimizer set is initialized using the estimated transformation parameters. A learning table is generated to perform exchange iterative processing, and the optimizer set algorithm is determined based on the processing results.

[0033] Preferably, optimizing the transformation parameters of the affine transformation model according to the population initialization result and the optimizer ensemble algorithm, and determining the highest-ranked solution as the final optimization result by majority voting based on the optimization result includes:

[0034] Calculate the entropy and joint entropy between the reference image and the image to be registered respectively, and use the entropy and joint entropy as the basis to analyze the mutual information between the reference image and the image to be registered to determine the degree of similarity;

[0035] The mutual information is used as the fitness value, and the optimizer set algorithm is used to maximize the mutual information between the reference image and the image to be registered, and the optimal transformation parameters in the affine transformation model are searched;

[0036] Based on the search results, the voting method is used to determine the solution with the highest ranking as the final optimization result, and the affine transformation model is determined according to the final optimization result.

[0037] Preferably, the calculation formula of mutual information is:

[0038]

[0039] Where I(A, B) represents the mutual information between the reference image A and the image to be registered B, P A (a) With P B(b) indicates that the reference image A and the image to be registered B are completely independent of each other in probability density distribution, P AB (a, b) represents the joint probability density distribution between the reference image A and the image to be registered B.

[0040] Preferably, the mutual information is used as the fitness value, and the mutual information between the reference image and the image to be registered is maximized using the optimizer set algorithm. Searching for the optimal transformation parameters in the affine transformation model includes:

[0041] The mutual information is used as the fitness value, and the number and frequency of exchange iterations in the maximization process are determined. At the same time, the search space dimension for searching the optimal transformation parameters of the affine transformation model is set.

[0042] Using the search mechanism of each optimizer in the optimizer ensemble algorithm, individuals are randomly selected from the learning table until the number of exchange iterations is reached, and the fitness value of the individual is calculated during the iteration process;

[0043] According to the optimal fitness value of each optimizer and its corresponding position, the global optimal fitness value and its corresponding position are updated, and the optimal transformation parameters in the affine transformation model are output.

[0044] The beneficial effects of the present invention are:

[0045] 1. The present invention first extracts the feature points of multimodal remote sensing images and uses a domain adaptation method to reduce the difference in feature vectors. The feature vectors of the reference image and the image to be registered are mapped to the reproducing kernel Hilbert space, minimizing the maximum mean difference between the marginal distribution and the conditional distribution. At the same time, the nearest neighbor method is used to match the feature vectors of the reference image and the image to be registered to obtain the same-name points of the two images. Finally, the values ​​of the transformation parameters of the affine transformation model are calculated, and the images are spatially aligned. When the distribution of the feature vectors of the reference image and the image to be registered are similar, the feature vectors are matched again, which can increase the number of correctly matched point pairs and reduce the false matching rate, thereby improving the accuracy and robustness of feature matching.

[0046] 2. In order to further improve the accuracy of registration, the present invention uses an optimizer set algorithm to search for the optimal transformation parameters. In the optimizer set, the population of all optimizers is used as a learning table. Each optimizer exchanges individuals with the learning table to achieve information exchange and sharing, and uses a bootstrap method to randomly extract individuals from the learning table. The number of individuals exchanged between the optimizer and the learning table is adaptively adjusted according to the fitness value. The optimization algorithm is used as the optimizer, and multiple optimizers are integrated to form a strong optimizer, thereby giving full play to the search advantages of multiple optimization algorithms, improving the global search capability, and avoiding the search from falling into the local optimum. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 is a flow chart of a multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm according to an embodiment of the present invention;

[0049] Figure 2 is an overall implementation flow chart of a multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm according to an embodiment of the present invention;

[0050] Figure 3 is a flowchart of feature matching in a multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm according to an embodiment of the present invention;

[0051] Figure 4 3 is an effect diagram of feature matching without TCA processing in a multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm according to an embodiment of the present invention;

[0052] Figure 5 3. This is an effect diagram of feature matching after TCA processing in a multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm according to an embodiment of the present invention;

[0053] Figure 6 is an integration strategy diagram of multiple optimizers in a multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm according to an embodiment of the present invention;

[0054] Figure 7 This is a flowchart of an image registration method based on an optimizer ensemble algorithm in a multimodal remote sensing image registration method based on domain adaptation and an optimizer ensemble algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention. They are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. By referring to these contents, ordinary technicians in this field should be able to understand other possible implementation methods and the advantages of the present invention.

[0056] According to an embodiment of the present invention, a multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm is provided.

[0057] The present invention will now be further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, according to an embodiment of the present invention, a multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm includes:

[0058] Step S1: extract feature vectors of the reference image and the image to be registered respectively, perform feature matching on the feature vectors according to the category label inclusion status of the feature vectors, and obtain a feature matching vector based on the matching results.

[0059] In one embodiment, feature vectors of the reference image and the image to be registered are extracted respectively, and feature matching is performed on the feature vectors based on the category label inclusion status of the feature vectors. The feature matching vector obtained based on the matching results includes:

[0060] According to the category label inclusion status of the feature vector, the domain adaptation method is used to optimize the difference status between the feature vectors, and the feature vectors are mapped to the reproducing kernel Hilbert space to quantify the mean difference, minimizing the maximum mean difference between the marginal distribution and the conditional distribution;

[0061] A distance threshold is set and combined with the nearest neighbor method, feature matching is performed on the reference image and the image to be registered, and the matching result is used as the feature matching vector.

[0062] Specifically, when optimizing the difference state between feature vectors by using a domain adaptation method according to the category label inclusion status of the feature vectors, and mapping the feature vectors to the reproducing kernel Hilbert space to quantify the mean difference, the category label inclusion status of the feature vectors can be judged according to the judgment requirements, and when the feature vector does not include the category, the edge distribution of the reference image and the image to be registered are judged respectively, and the feature vector is mapped to the reproducing kernel Hilbert space; according to the mapping result, the maximum mean difference distance between the edge distribution of the reference image and the image to be registered is obtained, the transformation matrix is ​​calculated, and the maximum mean difference distance between the edge distributions is minimized; when the feature vector includes the category, the optimal mapping function is calculated by using a balanced distribution adaptive algorithm with the goal of minimizing the weighted distribution distance, and the feature vector is mapped to the reproducing kernel Hilbert space according to the calculation result; according to the mapping result, the maximum mean difference distance between the conditional distribution of the reference image and the image to be registered is obtained, the transformation matrix is ​​calculated, and the weighted maximum mean difference distance between the edge distribution and the conditional distribution is minimized.

[0063] It should be explained that this embodiment collects and organizes global public data sets and multimodal remote sensing images of actual survey areas, including visible light images, SAR images, LiDAR depth maps, near-infrared images and maps. In order to better extract the same-name feature points of multimodal remote sensing images, this embodiment uses the KAZE algorithm to extract feature points.

[0064] In the process of extracting feature vectors, the KAZE algorithm is used to extract feature vectors of multimodal remote sensing images. The KAZE algorithm is a feature detection algorithm based on scale space, which can detect key points in the image and their descriptors. It extracts the feature information of the image by performing multiple iterative diffusion filtering on the image. The KAZE algorithm mainly includes the following steps: construction of nonlinear scale space; detection and precise positioning of feature points; determination of the main direction of feature points; and generation of feature descriptors.

[0065] When the feature vector has no category label, the Transfer Component Analysis (TCA) algorithm is used to process the feature vector and then perform feature point matching. Specifically, the TCA algorithm maps the feature vectors of the reference image and the image to be registered into the reproducing kernel Hilbert space, minimizing the maximum mean difference between the two sets of feature vectors and making the edge distributions of the two sets of feature vectors as similar as possible.

[0066] Assume that the marginal distributions of the feature vectors of the reference image and the image to be registered are X R and X S , then X R and X S The MMD (Maximum Mean Discrepancy) distance M(X R , X S )for:

[0067]

[0068] Where, M(X R , X S ) represents the reference image edge distribution X R and the edge distribution X of the image to be registered S The maximum mean difference distance between them, n1 represents the edge distribution X of the reference image R The number of samples in Represents the mapping function, which is used to map the original eigenvector to the reproducing kernel Hilbert space, x Ri represents the i-th eigenvector of the reference image, and n2 represents the edge distribution X of the image to be registered. S The number of samples in x Si represents the i-th eigenvector of the image to be registered, represents the reproducing kernel Hilbert space normal form.

[0069] When the feature vector has no category label, this embodiment uses the Transfer Component Analysis (TCA) algorithm to search for the optimal transformation matrix to minimize X R and X SMMD distance of the feature vectors X R and X S so that the marginal distributions of the feature vectors X

[0070] When the feature vectors have class labels, the feature vectors are processed by a Balanced Distribution Adaptation (BDA) algorithm, and then feature point matching is performed. Specifically, the feature vectors of the reference image and the image to be registered are mapped to a reproducing kernel Hilbert space, the maximum mean discrepancy between the two sets of feature vectors is minimized, and the marginal distribution and the conditional distribution of the two sets of feature vectors are made as similar as possible.

[0071] When the feature vectors of the reference image and the image to be registered have class labels, the MMD distance Disβ of the feature vector inter-class conditional distribution is:

[0072]

[0073] In the formula, Disβ represents the maximum mean discrepancy distance between the conditional distributions of the reference image and the image to be registered, C represents the number of classes, x c represents the feature vector belonging to the class c in the reference image, x c represents the feature vector belonging to the class c in the image to be registered, x i x c represents the i-th feature vector among the feature vectors belonging to the class c in the reference image, x j x c represents the j-th feature vector among the feature vectors belonging to the class c in the image to be registered, x x c represents the number of feature vectors belonging to the class c in the reference image, x c represents the number of feature vectors belonging to the class c in the image to be registered.

[0074] In multi-modal remote sensing image registration, the marginal distribution and the conditional distribution have different importance for different modal images. The Balanced Distribution Adaptation (BDA) algorithm can dynamically adjust the importance of the marginal distribution and the conditional distribution according to a specific data field through a balance factor μ. In the BDA algorithm, the MMD distance Dis of the marginal distribution and the conditional distribution is:

[0075] Dis = (1-μ) Dis α + μ Dis β ;

[0076] where μ ∈ [0, 1] represents the balance factor, Dis αThe MMD distance represents the marginal distribution of feature vectors. This embodiment statistically analyzes feature matching data from a large number of multimodal remote sensing images, assigns different balance factors to remote sensing images of different modalities, and uses the BDA algorithm to calculate the optimal mapping function with the goal of minimizing the weighted distribution distance Dis when the feature vectors have category labels. The feature vectors are mapped to the reproducing kernel Hilbert space, and feature matching is performed after minimizing the maximum mean difference between the two sets of feature vectors.

[0077] At the same time, the nearest neighbor method is used to match the feature vectors of the reference image and the image to be registered. Specifically, a distance threshold is set in the nearest neighbor method. When the distance between the two feature vectors is less than the threshold, the two are considered to be matched. The distance can be Euclidean distance, Manhattan distance, Chebyshev distance, etc. In this embodiment, Euclidean distance is selected.

[0078] When the Euclidean distance between two feature vectors is less than a threshold value σ, this embodiment considers that the two feature vectors are successfully matched and their corresponding feature points are points of the same name. In this embodiment, the value of σ is set to 1.

[0079] Step S2: using a random sampling consistency algorithm to estimate the transformation parameters corresponding to the affine transformation model, and optimizing the affine transformation model according to the estimation result and the similarity measure.

[0080] In one embodiment, estimating transformation parameters corresponding to the affine transformation model using a random sampling consensus algorithm, and optimizing the affine transformation model based on the estimation result and the similarity measure includes:

[0081] Randomly sampling some feature matching vectors to estimate the affine transformation model parameters, and testing the feature matching vectors using the affine transformation model parameters according to the estimation results;

[0082] Generate the transformation parameters of the affine transformation model based on the test results, integrate the optimizer set, and use the transformation parameters to initialize the optimizer set population;

[0083] The transformation parameters of the affine transformation model are optimized according to the population initialization results and the optimizer ensemble algorithm, and the majority voting method is used based on the optimization results to determine the highest-ranked solution as the final optimization result.

[0084] Specifically, when generating the transformation parameters of the affine transformation model based on the test results, integrating the optimizer set, and initializing the optimizer set population using the transformation parameters, the feature matching vectors that satisfy the affine transformation model can be selected as internal points according to the test results, and the feature matching vectors that do not satisfy the affine transformation model can be selected as external points; the random sampling and estimation model parameter operations are repeated in an iterative manner, and the affine transformation model with the largest number of internal points is selected as the final estimation result according to the iterative repetition results to obtain the estimated transformation parameters; several optimization algorithms are selected to integrate the optimizer set, and the optimizer set is initialized using the estimated transformation parameters, a learning table is generated to perform exchange iterative processing, and the optimizer set algorithm is determined according to the processing results.

[0085] It should be explained that, based on the matched feature points, the transformation parameters of the affine transformation model are estimated using the random sampling consensus algorithm (RANSAC). The basic idea of ​​the RANSAC algorithm to estimate the model transformation parameters is to estimate the model parameters by randomly sampling a small part of the data, and use this model to test all the data. The data points that meet the model are regarded as inliers, and the data points that do not meet the model are regarded as outliers. The model parameters are continuously randomly sampled and estimated in an iterative manner, and finally the model with the largest number of inliers is obtained as the final estimation result.

[0086] At the same time, the transformation parameters estimated by the RANSAC algorithm are used to initialize the population of the optimizer ensemble algorithm, and the values ​​of all population individuals are set to the transformation parameters estimated by the RANSAC algorithm. As a preferred method, the optimizer ensemble algorithm integrates three optimization algorithms, namely differential evolution algorithm, particle swarm optimization algorithm, and gravitational search algorithm.

[0087] This embodiment uses an affine transformation model commonly used in the field of remote sensing image registration to spatially align the reference image and the image to be registered. Based on the matched feature points, the RANSAC algorithm is used to estimate the transformation parameters of the affine transformation model. The formula for affine transformation of a point (x, y) in a multimodal remote sensing image to a point (x′, y′) is:

[0088]

[0089] This embodiment uses the RANSAC algorithm to estimate the six transformation parameters a in the affine transformation model. 11 、a 12 , a 13 , a 21 , a 22 , a 23 The value of .

[0090] Inspired by ensemble learning, an optimizer ensemble algorithm is proposed in this embodiment, which takes optimization algorithm as optimizer, integrates multiple optimizers to form a strong optimizer, and is used to optimize the transformation parameters of multi-modal remote sensing images. In the exchange iteration times, each optimizer exchanges individuals with the learning table to realize the exchange and sharing of information. In other iteration times, each optimizer independently runs according to its own search mechanism to make full use of the search mechanism of different optimizers. Individuals are randomly extracted from the learning table by bootstrap method, and the number of individuals exchanged between the optimizer and the learning table is adaptively adjusted according to the value of fitness. The optimizer ensemble algorithm proposed in this embodiment can integrate any kind of population-based intelligent optimization algorithm, take advantage of the search of multiple optimizers, and prevent the optimization of transformation parameters from falling into local optimum. In this embodiment, the optimizer ensemble algorithm integrates three optimization algorithms, which are selected as differential evolution algorithm, particle swarm optimization algorithm and universal gravitation search algorithm. The transformation parameters estimated by RANSAC algorithm are used to initialize the population of the three optimization algorithms.

[0091] Specifically, when the transformation parameters of the affine transformation model are optimized according to the population initialization result and the optimizer ensemble algorithm, and the highest ranked solution is determined as the final optimization result based on the optimization result by using the majority voting method, the entropy and joint entropy between the reference image and the image to be registered can be calculated respectively, and the mutual information between the reference image and the image to be registered is analyzed based on the entropy and joint entropy as the reference for the similarity between the reference image and the image to be registered; the mutual information is used as the fitness value, and the optimizer ensemble algorithm is used to maximize the mutual information between the reference image and the image to be registered to search for the optimal transformation parameters in the affine transformation model; the highest ranked solution is determined as the final optimization result based on the search result by using the voting method, and the affine transformation model is determined according to the final optimization result.

[0092] Among them, the mutual information is used as the fitness value, and the optimizer ensemble algorithm is used to maximize the mutual information between the reference image and the image to be registered to search for the optimal transformation parameters in the affine transformation model, which includes: taking the mutual information as the fitness value, and determining the exchange iteration times and frequency in the maximization process, and setting the search space dimension of the optimal transformation parameters in the affine transformation model; individuals are randomly extracted from the learning table by using the search mechanism of each optimizer in the optimizer ensemble algorithm until the exchange iteration times are reached; the fitness value of the global optimum and its corresponding position are updated according to the optimal fitness value of each optimizer and its corresponding position, and the optimal transformation parameters in the affine transformation model are output.

[0093] It needs to be explained that the mutual information is used as the similarity measure, and the optimizer ensemble algorithm is used to optimize the transformation parameters of the affine transformation model. The six transformation parameters a 11 , a 12 , a 13 , a21 , a 22 , a 23 As a population of optimizer ensemble algorithms, the transformation parameters of the affine transformation model are optimized by iteratively searching for the maximum mutual information.

[0094] In the optimizer ensemble algorithm, the optimization algorithm serves as the optimizer, integrating multiple optimizers to form a strong optimizer. During the exchange iterations, each optimizer exchanges individuals with the learning table to communicate and share information. During the remaining iterations, each optimizer operates independently based on its own search mechanism, fully leveraging the search mechanisms of different optimizers. Individuals are randomly drawn from the learning table using a bootstrap method, and the number of individuals exchanged between the optimizer and the learning table is adaptively adjusted based on the fitness value. For detailed procedures, refer to Algorithm 1 in this embodiment.

[0095] Since mutual information can better adapt to the grayscale changes between multimodal remote sensing images, this embodiment uses mutual information as a similarity measure. Mutual information is a measure of the statistical correlation between two random variables, which reflects the degree of association between two objects. The calculation formula of mutual information is as follows:

[0096] I(A,B)=H(A)+H(B)-H(A,B);

[0097] Where H(A) and H(B) represent the entropy of images A and B respectively, and H(A, B) is the joint entropy between the two images. Then:

[0098]

[0099] Where a∈A, b∈B, I(A, B) represents the mutual information between the reference image A and the image to be registered B, and P A (a) With P B (b) indicates that the reference image A and the image to be registered B are completely independent of each other in probability density distribution, P AB (a, b) represents the joint probability density distribution between the reference image A and the image to be registered B. The mutual information represents the common information contained in the two images. The larger the mutual information value between the reference image and the image to be registered, the higher the accuracy of multimodal image registration.

[0100] When the optimizer ensemble algorithm is used to register multimodal remote sensing images, the mutual information between images in each iteration is the fitness value. The optimizer ensemble algorithm is used to maximize the mutual information between images, thereby searching for the optimal transformation parameter a in the affine transformation model. 11 、a 12 , a 13 , a 21 , a 22 , a 23 The value of .

[0101] In the optimizer ensemble algorithm, a learning table is formed from the population of all optimizers, used to share information between optimizers. The maximum number of iterations is divided into multiple exchange iterations. Each optimizer exchanges individuals with the learning table during the exchange iterations and operates independently during the remaining iterations, randomly selecting exchange individuals from the learning table through bootstrap sampling. To balance the number of exchanged individuals and retained individuals, the number of exchanged individuals for each optimizer is allocated based on its fitness value. The highest-ranked solution is determined by majority voting as the final optimization result.

[0102] Step S3: Using the optimized affine transformation model to perform spatial transformation on the remote sensing image to be registered, so as to achieve spatial alignment between the reference image and the image to be registered.

[0103] In order to facilitate understanding of the above technical solutions of the present invention, the working principle or operation mode of the present invention in actual process is described in detail below.

[0104] In practical applications, such as Figure 2 As shown in the figure, a domain-adaptive multimodal remote sensing image feature matching algorithm is used to match multimodal remote sensing images, and the values ​​of transformation parameters in the affine transformation model are estimated by the RANSAC algorithm. The transformation parameters estimated by the RANSAC algorithm are used as the initial population, and the globally optimal transformation parameters are searched through the optimizer set algorithm to further improve the accuracy and robustness of image registration. Finally, the optimized transformation parameters are used to register multimodal remote sensing images.

[0105] like Figure 3 As shown, this embodiment first uses the KAZE algorithm to extract feature points, and then determines whether the feature vector has a category label. When the feature vector does not have a category label, the feature vector is processed using the marginal distribution adaptive method, and the feature vectors of the reference image and the image to be registered are mapped to the reproducing kernel Hilbert space, minimizing the maximum mean difference between the two sets of feature vectors to make the marginal distribution as similar as possible.

[0106] When the feature vector has a category label, the balanced distribution adaptive method is used to process the feature vector, and the feature vectors of the reference image and the image to be registered are mapped to the reproducing kernel Hilbert space, minimizing the maximum mean difference between the two sets of feature vectors and making the marginal distribution and the conditional distribution as similar as possible. At this time, the distribution of the feature vectors of the multimodal remote sensing images is similar. Matching the feature vectors of the two images can increase the correct matching point pairs and reduce the false matching rate.

[0107] Specifically, if Figure 4 and Figure 5As shown in the figure, the matching effect of multimodal remote sensing image feature vectors before and after processing using the TCA algorithm is listed. The figure shows two different modal images, visible light image and SAR image. Figure 4 and Figure 5 It can be seen that after using the TCA algorithm to perform edge distribution adaptive processing on the feature vector, the number of correctly matched point pairs is significantly increased, the false matching rate is reduced, and the performance of feature matching is improved.

[0108] like Figure 6 As shown in the figure, the optimizer ensemble algorithm draws on the idea of ​​ensemble learning to integrate multiple optimizers. The learning table consists of the populations of all optimizers. In the exchange iterations, each optimizer exchanges individuals with the learning table to achieve information exchange and sharing. In other iterations, each optimizer runs independently according to its own search mechanism, which can reduce the computational cost, make full use of the search mechanisms of different optimizers, and simplify the integration process. The new population of each optimizer consists of a part of the individuals in the current population and a part of the individuals randomly sampled from the learning table. The optimal solution is selected as the output of all optimizers by voting. After exchanging individuals with the learning table, each optimizer adds its best individual to the new population. This can not only maintain the convergence of the algorithm, but also improve the global search capability. Different from the crossover operation between two individuals, the exchange between the optimizer and the learning table belongs to the master-slave mode, which is more suitable for information exchange between multiple optimizers.

[0109] In the optimizer ensemble algorithm, each optimizer exchanges individuals with the learning table. The exchanged individuals are obtained through random sampling using the bootstrap method, and the number of exchanges is adaptively adjusted based on the fitness value. The pseudo code of the optimizer ensemble algorithm proposed in this embodiment is as follows (Algorithm 1: Optimizer ensemble algorithm):

[0110] Input: number of exchange iterations E and frequency l, dimension of search space D, population size of optimizers N, number of optimizers m;

[0111] Output: The global optimal fitness value g obtained by m optimizers and its corresponding position gx;

[0112] Begin

[0113] 1: for i=1: m do

[0114] 2: Randomly generate N individuals to initialize the i-th optimizer;

[0115] 3: end for

[0116] 4: for k=1: l do

[0117] 5: for i=1: m do

[0118] 6: for j = 1: E k -1 do

[0119] 7: The i-th optimizer runs independently according to its own search mechanism;

[0120] 8: Calculate the fitness value of each individual in the population;

[0121] 9: Update the optimal fitness value f of the i-th optimizer i and its corresponding position fx i ;

[0122] 10: end for

[0123] 11: end for

[0124] 12: for i=1: m do

[0125] 13: Normalized fitness value f i ;

[0126] 14: Adaptive calculation of the number of exchanges n i ;

[0127] 15: The i-th optimizer exchanges n with the learning table i individual;

[0128] 16: end for

[0129] 17: Update the global optimal fitness value g and its corresponding position gx obtained by m optimizers;

[0130] 18: end for.

[0131] From Algorithm 1, we can see that the maximum number of iterations is divided into l exchange iterations. First, each optimizer is initialized with N randomly generated solutions. Each optimizer runs independently according to its own search mechanism until it stops when the exchange iteration number is reached. In the exchange iteration number, the i-th optimizer randomly extracts n solutions from the learning table. i Individual, finally according to the optimal fitness value f of each optimizer and its corresponding position fx, update the global optimal fitness value g and its corresponding position gx, and the output results of m optimizers are obtained by voting method.

[0132] Specifically, if Figure 7As shown in the figure, the RANSAC algorithm is first used to estimate the parameters of the affine transformation model. Then, the optimizer ensemble algorithm is initialized, that is, the population values ​​of the selected differential evolution algorithm, particle swarm optimization algorithm, and gravitational search algorithm are set to the transformation parameters estimated by the RANSAC algorithm. Then, the optimizer ensemble algorithm is used to maximize the mutual information between images, thereby searching for the optimal transformation parameter a in the affine transformation model. 11 、a 12 , a 13 , a 21 , a 22 , a 23 Finally, the multimodal images are registered using the affine transformation model after parameter optimization.

[0133] To sum up, with the help of the above-mentioned technical scheme of the present invention, the present invention first extracts the feature points of the multimodal remote sensing image, and uses the domain adaptive method to reduce the difference of the feature vectors, maps the feature vectors of the reference image and the image to be registered to the reproducing kernel Hilbert space, minimizes the maximum mean difference between the marginal distribution and the conditional distribution, and uses the nearest neighbor method to match the feature vectors of the reference image and the image to be registered to obtain the same-name points of the two images. Finally, the values ​​of the transformation parameters of the affine transformation model are calculated, and the images are spatially aligned. When the distribution of the feature vectors of the reference image and the image to be registered are similar, the feature vectors are matched again, which can increase the correct matching point pairs and reduce the false matching rate, thereby improving the accuracy and robustness of feature matching.

[0134] In order to further improve the accuracy of registration, the present invention uses an optimizer set algorithm to search for the optimal transformation parameters. In the optimizer set, the population of all optimizers is used as a learning table. Each optimizer exchanges individuals with the learning table to achieve information exchange and sharing, and uses a bootstrap method to randomly extract individuals from the learning table. The number of individuals exchanged between the optimizer and the learning table is adaptively adjusted according to the value of fitness. The optimization algorithm is used as the optimizer, and multiple optimizers are integrated to form a strong optimizer, thereby giving full play to the search advantages of multiple optimization algorithms, improving the global search capability, and avoiding the search from falling into the local optimum.

[0135] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm, characterized in that: The method includes: Extract the feature vectors of the reference image and the image to be registered respectively, perform feature matching on the feature vectors according to the category label inclusion status of the feature vectors, and obtain the feature matching vector based on the matching results; The transformation parameters corresponding to the affine transformation model are estimated using a random sampling consensus algorithm, and the affine transformation model is optimized based on the estimation results and the similarity measure; The optimized affine transformation model is used to perform spatial transformation on the remote sensing image to be registered, so as to achieve spatial alignment between the reference image and the image to be registered.

2. The multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm according to claim 1, characterized in that: The step of extracting feature vectors of the reference image and the image to be registered respectively, performing feature matching on the feature vectors according to the category label inclusion status of the feature vectors, and obtaining a feature matching vector based on the matching results includes: According to the category label inclusion status of the feature vector, the domain adaptation method is used to reduce the difference of the feature vector, and the feature vectors of the reference image and the image to be registered are mapped to the reproducing kernel Hilbert space to minimize the maximum mean difference between the marginal distribution and the conditional distribution; A distance threshold is set and combined with the nearest neighbor method, feature matching is performed on the reference image and the image to be registered, and the matching result is used as the feature matching vector.

3. The multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm according to claim 2, characterized in that: The method of optimizing the difference between feature vectors based on the category label inclusion status of the feature vectors using a domain adaptation method and mapping the feature vectors to a reproducing kernel Hilbert space to quantize the mean difference includes: According to the judgment requirements, the category label inclusion status of the feature vector is judged, and when the feature vector does not contain the category, the edge distribution of the reference image and the image to be registered is judged respectively, and the feature vector is mapped to the reproducing kernel Hilbert space; According to the mapping results, the maximum mean difference distance between the edge distributions of the reference image and the image to be registered is obtained, and the transformation matrix is ​​calculated to minimize the maximum mean difference distance between the edge distributions; When the feature vector contains a category, the balanced distribution adaptive algorithm is used to calculate the optimal mapping function with the goal of minimizing the weighted distribution distance, and the feature vector is mapped to the reproducing kernel Hilbert space based on the calculation result; According to the mapping results, the maximum mean difference distance between the conditional distribution of the reference image and the image to be registered is obtained, and the transformation matrix is ​​calculated to minimize the weighted maximum mean difference distance between the marginal distribution and the conditional distribution.

4. The multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm according to claim 3, characterized in that: The calculation formula for the maximum mean difference distance between the edge distribution of the reference image and the image to be registered is: Where, M(X R , X S ) represents the reference image edge distribution X R and the edge distribution X of the image to be registered S The maximum mean difference distance between them, n1 represents the edge distribution X of the reference image R The number of samples in , represents the mapping function, x Ri represents the i-th eigenvector of the reference image, and n2 represents the edge distribution X of the image to be registered. S The number of samples in x Si represents the i-th eigenvector of the image to be registered, represents the reproducing kernel Hilbert space normal form.

5. The multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm according to claim 4, characterized in that: The calculation formula of the maximum mean difference distance between the conditional distribution of the reference image and the image to be registered is: Where, Dis β represents the maximum mean difference distance between the conditional distribution of the reference image and the image to be registered, C represents the number of categories, represents the feature vector belonging to category c in the reference image, represents the feature vector of category c in the image to be registered, x i represents the i-th feature vector in the feature vector of category c in the reference image, x j represents the jth eigenvector in the eigenvectors of category c in the image to be registered, represents the number of feature vectors belonging to category c in the reference image, Indicates the number of feature vectors belonging to category c in the image to be registered.

6. The multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm according to claim 1, characterized in that: The method of estimating the transformation parameters corresponding to the affine transformation model by using a random sampling consistency algorithm and optimizing the affine transformation model according to the estimation result and the similarity measure includes: Randomly sampling some feature matching vectors to estimate the affine transformation model parameters, and testing the feature matching vectors using the affine transformation model parameters according to the estimation results; Generate the transformation parameters of the affine transformation model based on the test results, integrate the optimizer set, and use the transformation parameters to initialize the optimizer set population; The transformation parameters of the affine transformation model are optimized according to the population initialization results and the optimizer ensemble algorithm, and the majority voting method is used based on the optimization results to determine the highest-ranked solution as the final optimization result.

7. The multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm according to claim 6, characterized in that: Generating the transformation parameters of the affine transformation model based on the test results, integrating the optimizer set, and initializing the optimizer set population using the transformation parameters includes: According to the test results, the feature matching vectors that satisfy the affine transformation model are selected as the inliers, and the feature matching vectors that do not satisfy the affine transformation model are selected as the outliers; By iteratively repeating random sampling and estimating model parameters, the affine transformation model with the largest number of inliers is selected as the final estimation result according to the iterative repetition results to obtain the estimated transformation parameters; several optimization algorithms are selected to integrate the optimizer set, and the optimizer set is initialized with the estimated transformation parameters. A learning table is generated to perform exchange iterative processing, and the optimizer set algorithm is determined according to the processing results.

8. The multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm according to claim 7, characterized in that: The method of optimizing the transformation parameters of the affine transformation model according to the population initialization result and the optimizer set algorithm, and determining the highest-ranked solution as the final optimization result by majority voting based on the optimization result includes: Calculate the entropy and joint entropy between the reference image and the image to be registered respectively, and use the entropy and joint entropy as the basis to analyze the mutual information between the reference image and the image to be registered to determine the degree of similarity; The mutual information is used as the fitness value, and the optimizer set algorithm is used to maximize the mutual information between the reference image and the image to be registered, and the optimal transformation parameters in the affine transformation model are searched; Based on the search results, the voting method is used to determine the solution with the highest ranking as the final optimization result, and the affine transformation model is determined according to the final optimization result.

9. The multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm according to claim 8, characterized in that: The calculation formula of the mutual information is: Where I(A, B) represents the mutual information between the reference image A and the image to be registered B, P A (a) With P B (b) indicates that the reference image A and the image to be registered B are completely independent of each other in probability density distribution, P AB (a, b) represents the joint probability density distribution between the reference image A and the image to be registered B.

10. The multimodal remote sensing image registration method based on domain adaptation and optimizer ensemble algorithm according to claim 8, characterized in that: The mutual information is used as the fitness value, and the mutual information between the reference image and the image to be registered is maximized by using the optimizer set algorithm to search for the optimal transformation parameters in the affine transformation model. The mutual information is used as the fitness value, and the number and frequency of exchange iterations in the maximization process are determined. At the same time, the search space dimension for searching the optimal transformation parameters of the affine transformation model is set. Utilizing the search mechanism of each optimizer in the optimizer ensemble algorithm, individuals are randomly selected from the learning table until the number of exchange iterations is reached. The fitness value of the individual is calculated during the iteration process. According to the optimal fitness value of each optimizer and its corresponding position, the global optimal fitness value and its corresponding position are updated, and the optimal transformation parameters in the affine transformation model are output.

Images