Optical and SAR image registration method and system based on local distortion division

Abstract

The invention relates to an optical and SAR image registration method and system based on local distortion division. The method comprises the following steps: acquiring optical and SAR images, performing superpixel segmentation, calculating superpixel feature similarity, dividing a local distortion region and a general region, inputting the local distortion region and the general region into a twin multi-feature capsule network for feature extraction, and constructing to obtain feature descriptors; and obtaining a first registration result corresponding to the local distortion region and a second registration result corresponding to the general region through the transformation model, completing super-pixel replacement reconstruction, and obtaining a registration result. By defining superpixel features and calculating the similarity of the superpixel features, local distortion regions and general regions of optical and SAR images can be quickly divided; according to the method, feature extraction is performed in a twin multi-feature capsule network, a corresponding transformation model is constructed to obtain a corresponding registration result, and replacement reconstruction is performed, so that the defect that a single registration model is easily influenced by local distortion can be overcome, and the image registration precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image registration, and particularly to a registration method and system for optical and SAR images based on local distortion division. Background Art

[0002] Optical remote sensing and synthetic aperture radar (SAR) technologies are two main remote sensing methods for obtaining information on the Earth's surface. Optical remote sensing generates images by recording the reflection or radiation of visible light, while SAR technology uses microwave signals to detect the ground surface and generate images. Optical images have high resolution and rich spectral information, capable of providing high-quality surface information of ground objects, but they cannot work in case of cloud cover or at night. In contrast, SAR technology can penetrate clouds and conduct observations at night, with unique all-weather observation capabilities. Registered optical remote sensing and SAR image data can achieve information complementarity, more reliably interpret image scenes or specific objects, and are crucial for tasks such as monitoring natural disasters and change detection. Current optical and SAR image registration technologies can be divided into two categories: traditional methods and deep learning-based methods. Traditional methods include feature-based methods and region-based methods. Feature-based methods detect key points in the image and then use feature descriptors for matching to achieve image registration; another traditional method is the region-based registration method, which uses local regions or feature regions of the image for matching and registration; Deep learning-based methods usually use deep neural networks to learn the feature representations of images and optimize the registration loss function to achieve image registration.

[0003] There are some drawbacks in traditional optical and SAR image registration methods: Region-based methods are sensitive to geometric and radiometric differences and require high computational costs; the positions of corresponding key points in feature-based methods may shift within a small range, and the point matching strategy cannot eliminate the shift of matching point pairs caused by non-linear radiometric distortion; optical images often use features such as edges and corners, while SAR images may lack obvious textures or feature points, and it is difficult for general traditional methods to effectively extract features for registration. In addition, there are some limitations in optical and SAR image registration methods based on deep learning: The processes of most deep learning-based registration methods are the same as traditional registration methods, only replacing the computational methods of feature extraction or feature matching, and they are still affected by local distortions, resulting in the failure of local detail registration in the image; general CNNs perform convolution and pooling layer by layer during feature extraction, leading to problems such as feature aliasing, information loss, and unclear semantics; a large number of local geometric distortions, including perspective contraction and layover, make it difficult to express the spatial mapping relationship between optical and SAR images with a single geometric model. At the same time, general capsule networks have certain deficiencies in extracting deep features: The network structure is simple and the feature extraction ability is insufficient; an effective feature descriptor should include the shallow features and deep features of the data; in traditional capsule networks, the dimension of digital capsules is higher than that of primary capsules, representing more complex information, bringing higher computational complexity and more abstract features, and these features may not correspond to the actual physical meaning.

[0004] Therefore, due to the simple feature extraction network structure of traditional optical and SAR image registration methods and the single geometric model adopted when facing geometric distortions, there are often difficulties in extracting features from heterogeneous images, feature aliasing, and information loss, resulting in low registration accuracy. Summary of the Invention

[0005] Based on this, in order to solve the above technical problems, a registration method and system for optical and SAR images based on local distortion division are provided, which can quickly divide the distortion region, overcome the influence of a single registration model on local distortion, and improve the image registration accuracy.

[0006] A registration method for optical and SAR images based on local distortion division, the method includes:

[0007] Collect optical and SAR images and perform superpixel segmentation, and use the mean and variance of the gray values of the segmented images as superpixel features;

[0008] Calculate the superpixel feature similarity of optical and SAR images, and divide the image region into local distortion regions and general regions according to the superpixel feature similarity;

[0009] Input the local distortion region and the general region into the Siamese multi-feature capsule network for feature extraction, and construct a local distortion region feature descriptor and a general region feature descriptor;

[0010] Construct a transformation model based on the local distortion region feature descriptor and the general region feature descriptor, and obtain a first registration result corresponding to the local distortion region and a second registration result corresponding to the general region through the transformation model;

[0011] Perform superpixel segmentation on the first registration result and the second registration result, and complete superpixel replacement and reconstruction to obtain a registration result.

[0012] In one embodiment, collect optical and SAR images and perform superpixel segmentation, and use the mean and variance of the grayscale values of the segmented images as superpixel features, including:

[0013] Collect optical and SAR images, and use a superpixel segmentation method based on the minimum spanning tree to segment the optical and SAR images to obtain a superpixel segmentation result;

[0014] Based on the superpixel segmentation result, calculate the grayscale values within the superpixel region, and calculate the mean and variance of the grayscale values as superpixel features.

[0015] In one embodiment, calculate the superpixel feature similarity of optical and SAR images, including:

[0016] In each superpixel region of the optical and SAR images, determine each superpixel feature according to the mean and variance of the grayscale values;

[0017] Determine the covariance matrix corresponding to the superpixel feature, and calculate the inverse matrix according to the covariance matrix;

[0018] According to the inverse matrix, calculate the Mahalanobis distance between each superpixel feature, and determine the similarity between each superpixel feature according to the Mahalanobis distance.

[0019] In one embodiment, divide the image region into a local distortion region and a general region according to the superpixel feature similarity, including:

[0020] Identify the region with the smallest superpixel feature value according to the Mahalanobis distance. The region with the smallest superpixel feature value indicates the highest superpixel feature similarity;

[0021] Calculate the average value and standard deviation of the superpixel feature value at the region with the smallest superpixel feature value;

[0022] Determine the region division threshold, and divide the image region into a local distortion region and a general region according to the average value and standard deviation of the superpixel feature values and the region division threshold.

[0023] In one embodiment, input the local distortion region and the general region into a Siamese multi-feature capsule network for feature extraction, and construct a local distortion region feature descriptor and a general region feature descriptor, including:

[0024] Input the local distortion region and the general region into the Siamese multi-feature capsule network, and perform feature extraction through a basic convolutional layer to obtain the extracted features;

[0025] Input the extracted features into primary capsules that densely connect and fuse shallow features and deep features. In the Siamese multi-feature capsule network, construct a feature descriptor through a dynamic routing mechanism; the feature descriptor includes a local distortion region feature descriptor and a general region feature descriptor.

[0026] In one embodiment, in the Siamese multi-feature capsule network, construct a feature descriptor through a dynamic routing mechanism, including:

[0027] In the Siamese multi-feature capsule network, assign random values to the connections between low-level capsules and high-level capsules;

[0028] The low-level capsules transmit feature information to the high-level capsules based on the random values, and the high-level capsules calculate prediction vectors according to the transmitted feature information and perform iteration;

[0029] Aggregate the output information of the low-level capsules during the iteration process, generate an activation vector of the high-level capsules, and convert it into a data vector using a non-linear activation function;

[0030] Update the routing weights according to the prediction vectors and data vectors until the weights converge to obtain the feature descriptor.

[0031] In one embodiment, perform superpixel segmentation on the first registration result and the second registration result, and complete superpixel replacement and reconstruction to obtain a registration result, including:

[0032] Perform superpixel segmentation on the first registration result and the second registration result to obtain a first superpixel segmentation result and a second superpixel segmentation result;

[0033] Determine the target superpixels in the local distortion region according to the first superpixel segmentation result;

[0034] Replace the target superpixels with the corresponding superpixels in the general region, and retain the other superpixels in the general region to complete superpixel replacement and reconstruction.

[0035] In one embodiment, the method further includes:

[0036] marking the target superpixel;

[0037] When performing superpixel replacement, determining the corresponding superpixel positions in the general region according to the marking and performing replacement.

[0038] A registration system for optical and SAR images based on local distortion division, the system includes:

[0039] A superpixel segmentation module, configured to collect optical and SAR images and perform superpixel segmentation, and use the mean and variance of the grayscale values of the segmented images as superpixel features;

[0040] A region division module, configured to calculate the superpixel feature similarity of the optical and SAR images, and divide the image regions into local distortion regions and general regions according to the superpixel feature similarity;

[0041] A feature extraction module, configured to input the local distortion region and the general region into a twin multi-feature capsule network for feature extraction, and construct a local distortion region feature descriptor and a general region feature descriptor;

[0042] A preliminary registration module, configured to construct a transformation model according to the local distortion region feature descriptor and the general region feature descriptor, and obtain a first registration result corresponding to the local distortion region and a second registration result corresponding to the general region through the transformation model;

[0043] A registration module, configured to perform superpixel segmentation on the first registration result and the second registration result, and complete superpixel replacement and reconstruction to obtain a registration result.

[0044] In one embodiment, the superpixel segmentation module is further configured to: collect optical and SAR images, and use a superpixel segmentation method based on a minimum spanning tree to segment the optical and SAR images to obtain a superpixel segmentation result; based on the superpixel segmentation result, calculate the grayscale values within the superpixel region, and calculate the mean and variance of the grayscale values as superpixel features.

[0045] The above-mentioned registration method and system for optical and SAR images based on local distortion division can quickly divide the local distortion regions and general regions of optical and SAR images by defining superpixel features and calculating superpixel feature similarity; using a twin multi-feature capsule network for feature extraction, constructing a corresponding transformation model to obtain corresponding registration results and performing replacement and reconstruction can overcome the influence of a single registration model by local distortion and improve the image registration accuracy. Description of the Drawings

[0046] Figure 1 It is an application environment diagram of the registration method for optical and SAR images divided based on local distortion in an embodiment;

[0047] Figure 2 It is a schematic flowchart of the registration method for optical and SAR images divided based on local distortion in an embodiment;

[0048] Figure 3 It is a schematic flowchart of the registration method of the twin multi - feature fusion capsule network in an embodiment;

[0049] Figure 4 It is a schematic diagram of the capsule network structure in an embodiment;

[0050] Figure 5 It is a schematic diagram of dynamic routing in an embodiment;

[0051] Figure 6 It is a schematic diagram of fusion reconstruction in an embodiment;

[0052] Figure 7 It is a schematic structural diagram of the registration method for optical and SAR images divided based on local distortion in an embodiment;

[0053] Figure 8 It is a structural block diagram of the registration system for optical and SAR images divided based on local distortion in an embodiment;

[0054] Figure 9 It is an internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0055] In order to make the purpose, technical solutions and advantages of this application clearer, the following further details this application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.

[0056] It can be understood that the terms "first", "second", etc. used in this application can be used in this article to describe the registration results, but these registration results are not limited by these terms. These terms are only used to distinguish the first registration result from another registration result. For example, without departing from the scope of this application, the first registration result can be called the second registration result, and similarly, the second registration result can be called the first registration result. Both the first registration result and the second registration result are registration results, but they are not the same registration result.

[0057] The registration method for optical and SAR images divided based on local distortion provided by the embodiments of this application can be applied to an application environment as shown in Figure 1 as shown in Figure 1As shown, the application environment includes a computer device 110. The computer device 110 can collect optical and SAR images and perform superpixel segmentation, and use the mean and variance of the grayscale values of the segmented images as superpixel features; the computer device 110 can calculate the superpixel feature similarity of the optical and SAR images, and divide the image regions into local distortion regions and general regions according to the superpixel feature similarity; the computer device 110 can input the local distortion regions and general regions into a siamese multi-feature capsule network for feature extraction, and construct a local distortion region feature descriptor and a general region feature descriptor; the computer device 110 can construct a transformation model according to the local distortion region feature descriptor and the general region feature descriptor, and obtain a first registration result corresponding to the local distortion region and a second registration result corresponding to the general region through the transformation model; the computer device 110 can perform superpixel segmentation on the first registration result and the second registration result, and complete superpixel replacement reconstruction to obtain a registration result. Among them, the computer device 110 can be, but is not limited to, various personal computers, laptop computers, smartphones, robots, tablet computers and other devices.

[0058] In one embodiment, as Figure 2 shown, a registration method for optical and SAR images based on local distortion division is provided, including the following steps:

[0059] Step 202, collect optical and SAR images and perform superpixel segmentation, and use the mean and variance of the grayscale values of the segmented images as superpixel features.

[0060] The superpixel features can be predefined. By calculating the Mahalanobis distance between superpixel feature pairs, the local distortion regions and general regions between the optical and SAR image pairs can be quickly divided.

[0061] In one embodiment, a registration method for optical and SAR images based on local distortion division can also include the process of performing superpixel segmentation. The specific process includes: collecting optical and SAR images, and using a superpixel segmentation method based on the minimum spanning tree to segment the optical and SAR images to obtain a superpixel segmentation result; based on the superpixel segmentation result, calculate the grayscale values within the superpixel regions, and calculate the mean and variance of the grayscale values as superpixel features.

[0062] The computer device can first perform superpixel segmentation on the optical and SAR images, and then use the mean and variance of the grayscale values as superpixel features.

[0063] The distortion region division is based on the minimum spanning tree-based superpixel segmentation, and the distortion conditions of each region in the optical and SAR image pairs are judged through superpixel blocks. At the same time, in this embodiment, a new superpixel feature, similarity measure, and distortion region division criterion are defined to achieve the division of local distortion regions and general regions.

[0064] Specifically, in this embodiment, the optical and SAR images can be input first, and the superpixel segmentation result of the input image can be obtained using the superpixel segmentation method based on the minimum spanning tree (MST); then, the mean and variance of the gray values are used as the features of the superpixels. For a superpixel region S, assume that this superpixel contains N pixels I1, I2,..., I n-1 , I n , the mean μ s and variance The calculation formulas can be respectively expressed as: Given two superpixel regions S1 and S2, the feature vector of each superpixel is:

[0065] Step 204, calculate the superpixel feature similarity of the optical and SAR images, and divide the image regions into local distortion regions and general regions according to the superpixel feature similarity.

[0066] The computer device can combine the Mahalanobis distance to measure the similarity between the superpixel features in the optical and SAR images. Finally, by calculating the superpixel feature similarity between the reference image and the image to be registered, each image is divided into a local distortion region and a general region according to the distortion degree. Specifically, the Superpixel-Based Local Distortion Segmentation (SLDD) algorithm based on superpixel features can be used to define the superpixel features. By calculating the Mahalanobis distance between superpixel feature pairs, the local distortion regions and general regions between the optical and SAR image pairs can be quickly divided.

[0067] In one embodiment, a registration method for optical and SAR images based on local distortion division may further include a process of calculating similarity. The specific process includes: in each superpixel region of the optical and SAR images, determine each superpixel feature according to the mean and variance of the image gray values; determine the covariance matrix corresponding to the superpixel features, and calculate the inverse matrix according to the covariance matrix; according to the inverse matrix, calculate the Mahalanobis distance between each superpixel feature, and determine the similarity between each superpixel feature according to the Mahalanobis distance.

[0068] Among them, the Mahalanobis distance D M is used to measure the similarity between these two superpixel features, and the calculation formula can be expressed as: where Σ is the covariance matrix of the feature vectors, and Σ -1 is its inverse matrix; the calculation formula of the covariance matrix Σ can be expressed as: In this embodiment, given the feature vectors ξ1 and ξ2 of two superpixel regions S1S1 and S2, the calculation formula of the superpixel feature similarity can be expressed as:

[0069] In one embodiment, a registration method for optical and SAR images based on local distortion division may further include a process of dividing regions. The specific process includes: identifying the region with the smallest superpixel feature value according to the Mahalanobis distance, and the region with the smallest superpixel feature value represents the highest superpixel feature similarity; calculating the average value and standard deviation of the superpixel feature values at the region with the smallest superpixel feature value; determining the region division threshold, and dividing the image region into a local distortion region and a general region according to the average value and standard deviation of the superpixel feature values and the region division threshold.

[0070] In this embodiment, the MST algorithm is used to perform superpixel segmentation on the reference image and the image to be registered. For each pair of superpixels, by calculating the Mahalanobis distance between the superpixel features, the position with the smallest superpixel feature value V s is identified, indicating the highest similarity between the superpixel in the reference image and the corresponding superpixel in the image to be registered. Specifically, the defined local distortion division criterion can be expressed as: where is the average value of the superpixel feature values of all regions with the smallest superpixel feature values, is its standard deviation, and λ is the region division threshold.

[0071] Step 206, input the local distortion region and the general region into the Siamese multi-feature capsule network for feature extraction, and construct a local distortion region feature descriptor and a general region feature descriptor.

[0072] Among them, the Siamese multi-feature capsule network fuses and uses shallow significant features and deep detail features through dense connections, which can not only enhance the robustness of the feature descriptor, but also reduce the network training difficulty and the number of parameters. Using the Siamese multi-feature capsule network for feature extraction can improve the feature representation ability, anti-deformation ability and robustness, and reduce information loss, solving the problems of traditional CNN feature aliasing, information loss and semantic ambiguity.

[0073] Using a twin multi-feature fusion capsule network (Multi-Feature Fusion Capsule Network, MFFCN), primary capsules are constructed by fusing shallow salient features and deep detail features, and then reconstructed into a 4D capsule-form feature descriptor that contains relevant information required for registration. In addition, the definition of the digital capsule feature descriptor is based on the Mahalanobis distance similarity metric, and a loss function is constructed by combining the idea of hard negative mining, which can enable the model to achieve the expected training effect.

[0074] In one embodiment, a registration method for optical and SAR images based on local distortion partitioning may further include a feature extraction process. The specific process includes: inputting the local distortion region and the general region into the twin multi-feature capsule network, and performing feature extraction through the basic convolutional layer to obtain the extracted features; inputting the extracted features into the primary capsules densely connected and fused with shallow features and deep features. In the twin multi-feature capsule network, a feature descriptor is constructed through a dynamic routing mechanism; the feature descriptor includes a local distortion region feature descriptor and a general region feature descriptor.

[0075] Among them, the network structure of the twin multi-feature capsule network is as Figure 3 shown. After inputting the local distortion region and the general region, first, the input passes through three 3×3 basic convolutional layers for feature extraction, and the stride of each layer is 1. Different from the traditional capsule network that only uses shallow features to construct primary capsules, in this embodiment, shallow salient features and deep detail features are fused through dense connection, as Figure 4 shown; then, the extracted features are constructed into a capsule feature descriptor. Compared with the scalar neurons in CNN, the capsule network describes the features extracted by the network in the form of capsules, which enables the network to capture pose information more sensitively and the features to have higher interpretability.

[0076] In this embodiment, the dimension of the digital capsule is reconstructed and set to 4, and a digital capsule feature descriptor containing texture, phase, structure, and amplitude information is constructed through a dynamic routing mechanism; finally, in order to keep the dimension consistent with other feature descriptors, the feature dimension is set to 32, that is, the 1×128 dimension of the ordinary vector is replaced by the 4×32 dimension in matrix form. Setting the information dimension of the digital capsule to 4 enables each digital capsule to represent four kinds of information valuable for the registration of optical and SAR images, namely texture, phase, structure, and amplitude, and solves the problem of multi-dimensional non-interpretability of the traditional capsule network.

[0077] In one embodiment, a registration method for optical and SAR images based on local distortion division may further include a process of constructing a feature descriptor through a dynamic routing mechanism. The specific process includes: in a Siamese multi-feature capsule network, assigning random values to the connections between low-level capsules and high-level capsules; the low-level capsules transmit feature information to the high-level capsules based on the random values, and the high-level capsules calculate prediction vectors according to the transmitted feature information and perform iterations; aggregating the output information of the low-level capsules during the iteration process to generate an activation vector of the high-level capsules and converting it into a data vector using a non-linear activation function; updating the routing weights according to the prediction vectors and data vectors until the weights converge to obtain the feature descriptor.

[0078] In this embodiment, by defining a distance metric between new capsule feature descriptors, efficient training of the multi-feature fusion capsule network can be achieved. Among them, the dynamic routing mechanism allows the capsule network to capture the spatial relationship between features, thereby improving the overall network performance. The schematic diagram of dynamic routing is as Figure 5 shown.

[0079] Specifically, each capsule is represented as a vector, where the length represents the probability of the feature existing, and the direction represents the attribute of the feature. The connections between low-level capsules and high-level capsules are initially assigned random values to facilitate the transmission of feature information. The high-level capsules calculate prediction vectors according to the outputs of the low-level capsules, which can be expressed as: Where is the prediction vector of the upper-level capsule, W ij is the connection weight, and u i is the output of the lower-level capsule.

[0080] Dynamic routing updates the connection weights through iteration. In each iteration, the outputs of the lower-level capsules are aggregated to generate an activation vector of the higher-level capsule, which can be expressed as: Where c ij represents the weight; the higher-level capsule uses a non-linear activation function to convert the aggregated vector into an output, which can be expressed as: Subsequently, the routing weights are updated according to the consistency between the output of the upper-level capsule and the prediction vector: c ij ←c ij +b ij . Finally, repeat the iteration multiple times until the routing weights converge, so as to obtain the output of the higher-level capsule, thereby effectively representing complex features.

[0081] Step 208: Construct a transformation model according to the local distortion region feature descriptor and the general region feature descriptor, and obtain a first registration result corresponding to the local distortion region and a second registration result corresponding to the general region through the transformation model.

[0082] Since the feature descriptors extracted by the capsule network have special forms and meanings, they cannot be simply transformed into one-dimensional vectors and the classical Euclidean distance can be used for calculation. In this embodiment, in order to effectively train the multi-feature fusion capsule network, a feature descriptor in the optical image is defined as v i and the corresponding feature descriptor in the SAR image is u i , where For the descriptor v of the optical image i and the descriptor u of the SAR image i , the Mahalanobis distance loss function based on hard negative mining is defined as: where is the Mahalanobis distance between positive examples (correctly matched optical and SAR images); is the Mahalanobis distance between negative examples (incorrectly matched optical and SAR images); Through hard negative mining, the most difficult-to-distinguish negative examples are selected here, that is, the incorrectly matched ones with a relatively close distance; α is a boundary used to control the distance difference between positive and negative examples to ensure that the distance of positive examples is less than that of negative examples; [x] + is an operation that takes positive values, indicating that when x > 0, x is returned, otherwise 0 is returned.

[0083] Step 210, perform superpixel segmentation on the first registration result and the second registration result, and complete superpixel replacement reconstruction to obtain the registration result.

[0084] In the traditional image registration process, a global transformation model is usually used to describe the spatial relationship between images. There are huge differences between optical images and SAR images in terms of land cover representation and imaging principles, and a single model is difficult to effectively handle local distortions.

[0085] In one embodiment, a registration method for optical and SAR images based on local distortion division may further include a process of feature fusion reconstruction. The specific process includes: performing superpixel segmentation on the first registration result and the second registration result to obtain the first superpixel segmentation result and the second superpixel segmentation result; determining the target superpixels in the local distortion region according to the first superpixel segmentation result; replacing the target superpixels with the corresponding superpixels in the general region, and retaining other superpixels in the general region to complete superpixel replacement reconstruction.

[0086] That is, in this embodiment, as Figure 6As shown, local-global thinking is adopted for registration in fusion reconstruction. First, the distortion conditions of different regions of the optical and SAR images are evaluated, the image distortion is estimated, and the image regions are divided into general regions and local distortion regions; then, feature descriptor matching is performed on each region, and transformation models are constructed separately to obtain registration results for the two types of distortion regions; among them, the registration result of the local distortion region is denoted as conseqL, and the registration result of the general region is denoted as conseqN; finally, based on the transformed images of the two region types and their superpixel segmentation results, the final transformed image to be registered is obtained through superpixel replacement.

[0087] In one embodiment, the computer device can mark the target superpixels; when performing superpixel replacement, the corresponding superpixel positions in the general region are determined according to the marks and replaced.

[0088] Since the two resampling results of the image to be registered are both based on the reference image, at the boundaries of different categories, the superpixel replacement reconstruction method in this embodiment will not have obvious seams. Specifically, first, superpixel segmentation is performed on the registration results of the local distortion region conseqL and the general region conseqN to obtain their respective superpixel segmentation results; next, the superpixels marked in red in the local distortion region of conseqN are replaced with the corresponding superpixels in conseqL, while the superpixels in the normal region of conseqN are retained, and through fusion reconstruction, the registration results of the two regions are integrated to obtain the final registration result.

[0089] In one embodiment, a registration method for optical and SAR images based on local distortion division is provided as Figure 7 shown. By combining SLDD and MFFCN, the distortion conditions of image regions are estimated and divided through superpixel features, different transformation models are adopted for different distortion regions, and then the registration of the entire image is completed through fusion reconstruction, thereby overcoming the problem that a single registration model is easily affected by local distortion; it solves the defect that the features of general registration methods are easily affected by local distortion, resulting in the failure of local detail registration of a single registration transformation model, and at the same time solves the problems of feature loss and poor interpretability of CNN feature descriptors; it has obvious advantages in processing optical and SAR image pairs with a large amount of local geometric distortion, and can effectively register local distortion regions and global overall regions; and it enhances the feature extraction ability, making the feature descriptors contain sufficient information. It reduces the number of parameters and the network training cost, and improves the registration efficiency and model versatility.

[0090] Moreover, in this application, a capsule network is used to replace the traditional CNN of the Siamese network, which significantly improves its feature representation ability, anti-deformation ability, and robustness, and reduces information loss. At the same time, shallow significant features and deep detail features are fused, and the dimensions of digital capsules are reconstructed to construct a digital capsule feature descriptor containing texture, phase, structure, and amplitude information, making the features interpretable; in addition, the feature descriptors extracted by the capsule network have special forms and meanings, and by defining a new distance metric between capsule feature descriptors, the efficient training of the multi-feature fusion capsule network is achieved; finally, by fusing the registration results of each region, a global registration image is generated.

[0091] It should be understood that although the steps in the above flowchart are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, there is no strict order limit for the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the above flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0092] In one embodiment, as Figure 8 shown, a registration system for optical and SAR images based on local distortion division is provided, including: a superpixel segmentation module 810, a region division module 820, a feature extraction module 830, a preliminary registration module 840, and a registration module 850, where:

[0093] The superpixel segmentation module 810 is used to collect optical and SAR images and perform superpixel segmentation, and use the mean and variance of the grayscale values of the segmented images as superpixel features;

[0094] The region division module 820 is used to calculate the superpixel feature similarity of optical and SAR images, and divide the image regions into local distortion regions and general regions according to the superpixel feature similarity;

[0095] The feature extraction module 830 is used to input the local distortion regions and general regions into a Siamese multi-feature capsule network for feature extraction, and construct local distortion region feature descriptors and general region feature descriptors;

[0096] The preliminary registration module 840 is used to construct a transformation model according to the local distortion region feature descriptors and general region feature descriptors, and obtain a first registration result corresponding to the local distortion region and a second registration result corresponding to the general region through the transformation model;

[0097] The registration module 850 is configured to perform superpixel segmentation on the first registration result and the second registration result, and complete superpixel replacement and reconstruction to obtain a registration result.

[0098] In one embodiment, the superpixel segmentation module 810 is further configured to: collect optical and SAR images, and segment the optical and SAR images using a superpixel segmentation method based on a minimum spanning tree to obtain a superpixel segmentation result; based on the superpixel segmentation result, calculate the image gray value within the superpixel region, and calculate the mean and variance of the image gray value as superpixel features.

[0099] In one embodiment, the region division module 820 is further configured to, in each superpixel region of the optical and SAR images, determine each superpixel feature according to the mean and variance of the image gray value; determine the covariance matrix corresponding to the superpixel feature, and calculate the inverse matrix according to the covariance matrix; according to the inverse matrix, calculate the Mahalanobis distance between each superpixel feature, and determine the similarity between each superpixel feature according to the Mahalanobis distance.

[0100] In one embodiment, the region division module 820 is further configured to identify the region with the smallest superpixel feature value according to the Mahalanobis distance, and the region with the smallest superpixel feature value represents the highest similarity of superpixel features; calculate the average value and standard deviation of the superpixel feature value at the region with the smallest superpixel feature value; determine a region division threshold, and divide the image region into a local distortion region and a general region according to the average value of the superpixel feature value, the standard deviation, and the region division threshold.

[0101] In one embodiment, the feature extraction module 830 is further configured to input the local distortion region and the general region into a Siamese multi-feature capsule network, perform feature extraction through a basic convolutional layer to obtain the extracted features; input the extracted features into primary capsules with dense connection and fusion of shallow features and deep features, and in the Siamese multi-feature capsule network, construct a feature descriptor through a dynamic routing mechanism; the feature descriptor includes a local distortion region feature descriptor and a general region feature descriptor.

[0102] In one embodiment, the feature extraction module 830 is further configured to, in the Siamese multi-feature capsule network, assign random values to the connections between low-level capsules and high-level capsules; the low-level capsules transmit feature information to the high-level capsules based on the random values, and the high-level capsules calculate prediction vectors according to the transmitted feature information and perform iteration; aggregate the output information of the low-level capsules during the iteration process to generate an activation vector of the high-level capsules and convert it into a data vector using a non-linear activation function; update the routing weights according to the prediction vectors and the data vectors until the weights converge to obtain a feature descriptor.

[0103] In one embodiment, the registration module 850 is further configured to perform superpixel segmentation on the first registration result and the second registration result to obtain a first superpixel segmentation result and a second superpixel segmentation result; determine target superpixels in the local distortion region according to the first superpixel segmentation result; replace the target superpixels with corresponding superpixels in the general region, and retain other superpixels in the general region to complete superpixel replacement reconstruction.

[0104] In one embodiment, the registration module 850 is further configured to mark the target superpixels; when performing superpixel replacement, determine the positions of the corresponding superpixels in the general region according to the marks and perform replacement.

[0105] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structural diagram may be as Figure 9 shown. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a registration method for optical and SAR images based on local distortion division. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device may be a touch layer covered on the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, a touchpad, or a mouse, etc.

[0106] Those skilled in the art can understand that Figure 9 the structure shown in

[0107] is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0108] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, it implements the steps of a registration method for optical and SAR images based on local distortion division.

[0109] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0110] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0111] The above-described embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A registration method for optical and SAR images based on local distortion division, characterized in that The method includes: Collect optical and SAR images and perform superpixel segmentation, and use the mean and variance of the gray values of the segmented images as superpixel features; Calculate the superpixel feature similarity of the optical and SAR images, and divide the image regions into local distortion regions and general regions according to the superpixel feature similarity; Input the local distortion regions and general regions into a Siamese multi-feature capsule network for feature extraction, and construct a local distortion region feature descriptor and a general region feature descriptor; Construct a transformation model according to the local distortion region feature descriptor and the general region feature descriptor, and obtain a first registration result corresponding to the local distortion region and a second registration result corresponding to the general region through the transformation model; Perform superpixel segmentation on the first registration result and the second registration result, and complete superpixel replacement reconstruction to obtain a registration result.

2. The registration method of optical and SAR images based on local distortion division according to claim 1, characterized in that Collect optical and SAR images and perform superpixel segmentation, and use the mean and variance of the gray values of the segmented images as superpixel features, including: Collect optical and SAR images, and use a superpixel segmentation method based on a minimum spanning tree to segment the optical and SAR images to obtain a superpixel segmentation result; Based on the superpixel segmentation result, calculate the gray values of the images within the superpixel regions, and calculate the mean and variance of the gray values as superpixel features.

3. The registration method of optical and SAR images based on local distortion division according to claim 1, characterized in that Calculate the superpixel feature similarity of the optical and SAR images, including: In each superpixel region of the optical and SAR images, determine each superpixel feature according to the mean and variance of the gray values; Determine the covariance matrix corresponding to the superpixel feature, and calculate the inverse matrix according to the covariance matrix; According to the inverse matrix, calculate the Mahalanobis distance between each superpixel feature, and determine the similarity between each superpixel feature according to the Mahalanobis distance.

4. The registration method of optical and SAR images based on local distortion division according to claim 3, characterized in that Divide the image regions into local distortion regions and general regions according to the superpixel feature similarity, including: Identify the region with the smallest superpixel feature value according to the Mahalanobis distance, and the region with the smallest superpixel feature value indicates the highest superpixel feature similarity; Calculate the average value and standard deviation of the superpixel feature values at the region with the smallest superpixel feature value; Determine a region division threshold, and divide the image regions into local distortion regions and general regions according to the average value and standard deviation of the superpixel feature values and the region division threshold.

5. The registration method of optical and SAR images based on local distortion division according to claim 1, characterized in that Input the local distortion regions and general regions into a Siamese multi-feature capsule network for feature extraction, and construct a local distortion region feature descriptor and a general region feature descriptor, including: Input the local distortion regions and general regions into a Siamese multi-feature capsule network, and perform feature extraction through a basic convolutional layer to obtain the extracted features; Input the extracted features into primary capsules that densely connect and fuse shallow features and deep features. In the Siamese multi-feature capsule network, construct a feature descriptor through a dynamic routing mechanism; the feature descriptor includes a local distortion region feature descriptor and a general region feature descriptor.

6. The registration method of optical and SAR images based on local distortion division according to claim 5, characterized in that, In the twin multi-feature capsule network, a feature descriptor is constructed through a dynamic routing mechanism, including: In the twin multi-feature capsule network, random values are assigned to the connections between low-level capsules and high-level capsules; The low-level capsules transmit feature information to the high-level capsules based on the random values, and the high-level capsules calculate prediction vectors according to the transmitted feature information and perform iterations; Aggregate the output information of the low-level capsules during the iteration process to generate an activation vector of the high-level capsules and convert it into a data vector using a non-linear activation function; Update the routing weights according to the prediction vectors and data vectors until the weights converge to obtain the feature descriptor.

7. The registration method of optical and SAR images based on local distortion division according to claim 1, characterized in that Perform superpixel segmentation on the first registration result and the second registration result, and complete superpixel replacement reconstruction to obtain the registration result, including: Perform superpixel segmentation on the first registration result and the second registration result to obtain a first superpixel segmentation result and a second superpixel segmentation result; Determine the target superpixels in the local distortion region according to the first superpixel segmentation result; Replace the target superpixels with the corresponding superpixels in the general region, and retain the other superpixels in the general region to complete the superpixel replacement reconstruction.

8. The registration method of optical and SAR images based on local distortion division according to claim 7, characterized in that, The method further includes: Mark the target superpixels; When performing superpixel replacement, determine the corresponding superpixel positions in the general region according to the marks and perform replacement.

9. An optical and SAR image registration system based on local distortion division, characterized in that The system includes: A superpixel segmentation module, configured to collect optical and SAR images and perform superpixel segmentation, and use the mean and variance of the grayscale values of the segmented images as superpixel features; A region division module, configured to calculate the superpixel feature similarity of the optical and SAR images, and divide the image regions into local distortion regions and general regions according to the superpixel feature similarity; A feature extraction module, configured to input the local distortion region and the general region into the twin multi-feature capsule network for feature extraction, and construct a local distortion region feature descriptor and a general region feature descriptor; A preliminary registration module, configured to construct a transformation model according to the local distortion region feature descriptor and the general region feature descriptor, and obtain a first registration result corresponding to the local distortion region and a second registration result corresponding to the general region through the transformation model; A registration module, configured to perform superpixel segmentation on the first registration result and the second registration result, and complete superpixel replacement reconstruction to obtain the registration result.

10. The registration system for optical and SAR images divided based on local distortion according to claim 1, wherein, The superpixel segmentation module is further configured to: collect optical and SAR images, and segment the optical and SAR images using a superpixel segmentation method based on a minimum spanning tree to obtain a superpixel segmentation result; based on the superpixel segmentation result, calculate the grayscale values within the superpixel regions, and calculate the mean and variance of the grayscale values as superpixel features.

Images