Multi-source remote sensing image registration method based on cross-modal robust feature decoupling

Abstract

The invention discloses a multi-source remote sensing image registration method based on cross-modal robust feature decoupling, and belongs to the technical field of remote sensing image processing. Comprising the following steps: 1, extracting features in a multi-source remote sensing image by using a multi-scale rotation invariance feature encoder; 2, capturing stably expressed feature descriptors in the multi-source remote sensing image by using a depth structure feature decoder, and fusing the multi-scale key point thermodynamic diagrams step by step by using a key point decoder to obtain a key point detection result graph; and 3, calculating a spatial transformation matrix by using a joint loss function to realize self-adaptive image registration. The introduction of the multi-scale rotation invariant feature encoder effectively solves the problem of scale and rotation difference coupling in the multi-source remote sensing image; the introduction of the depth self-similar structure feature decoder solves the problems of nonlinear radiation distortion and cross-modal difference in the multi-source remote sensing image; the joint loss function constructed based on the clustering perception contrast constraint solves the problem that the feature distinguishability between the same-name key point pairs is poor.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of remote sensing image processing, and particularly relates to a multi-source remote sensing image registration method based on cross-modal robust feature decoupling. BACKGROUND

[0002] Multi-source remote sensing image registration aims to align the remote sensing images of the same scene acquired by different time, sensors or platforms in space domain, and provides a basis for subsequent fusion processing (such as image fusion, change detection and super-resolution reconstruction). With the increasing information richness of data acquired by multi-source remote sensing platforms in time, space, spectral band and modalities, multi-source data fusion technology has become a research hotspot in the field of remote sensing by mining complementary information for deep analysis. However, the mutual coupling between the radiation differences (such as the differences in electromagnetic wave reflection characteristics between optical and SAR), geometric differences (different resolutions and imaging angles), viewing angle differences (changes in sensor observation direction) and time differences (dynamic changes of ground objects over time) between multi-source images makes the existing methods face the severe challenge of verifying registration deviation.

[0003] Traditional image registration methods mainly fall into two categories. One is a region-based registration method, which mainly relies on the pixel gray scale statistical information of the image, is easy to implement and has strong interpretability, but is sensitive to the radiation distortion between multi-source images. The other is a feature-based registration method, which can improve the robustness to radiation distortion, but is limited by the expression ability of artificial designed features to complex structural information in remote sensing images, and is difficult to adapt to image registration tasks facing complex geometric distortion and cross-modal structural difference.

[0004] In recent years, deep learning methods use convolutional neural networks to extract deep image features, enhance the cross-modal stability of descriptors, and alleviate modal differences through style transfer strategies. However, existing deep learning-based registration methods often replace one or several steps in traditional registration methods with neural network modules, lack joint registration optimization of scale scaling-angle rotation-radiation distortion-modal difference in multi-source remote sensing scenes, and the feature extraction module lacks targeted design, resulting in insufficient cross-modal feature robustness and discriminability. Therefore, constructing a robust feature self-adaptive extraction network that takes into account geometric deformation, spatial misplacement, radiation distortion and cross-modal difference is a key breakthrough to improve the registration accuracy of multi-source remote sensing images in complex scenes. SUMMARY

[0005] The purpose of the application is to face the application demand of multi-source remote sensing image adaptive registration, aiming at the multiple deviation coupling problems such as geometric deformation, spatial misplacement, radiation distortion and cross-modal difference between multi-source remote sensing images, a twin network image registration framework based on cross-modal robust feature decoupling is proposed, by mining the key point features in multi-source remote sensing images considering scale, rotation and cross-modal robustness, and carrying out end-to-end joint optimization learning, the adaptive high-precision registration of optical, SAR multi-source remote sensing images is realized.

[0006] The technical scheme adopted by the application is:

[0007] A multi-source remote sensing image registration method based on cross-modal robust feature decoupling, comprising the following steps:

[0008] Step one: using a lightweight multi-scale rotation invariant feature encoder, sharing its calculation and cross-modal representation ability through a twin network, extracting features in multi-source remote sensing images, and ensuring robustness under different scales and angles;

[0009] Step two: using a deep structure feature decoder based on deep structure self-similarity to capture stable feature descriptors in multi-source remote sensing images, using a key point decoder to gradually fuse multi-scale key point heat maps to obtain a key point detection result map;

[0010] Step three: using clustering-aware contrastive learning and key point detection loss to construct a joint loss function, and through end-to-end learning, extracting key point pairs in the image to be registered with high spatial accuracy and high repeatability in time domain, and calculating the spatial transformation matrix to realize adaptive image registration.

[0011] Compared with the prior art, the application has the following beneficial effects:

[0012] 1. The introduction of the multi-scale rotation invariant feature encoder effectively solves the problem of scale and rotation difference coupling in multi-source remote sensing images:

[0013] The application is a novel lightweight twin neural network model for multi-source image registration, which constructs a rotation invariant multi-scale feature space and mines deep structure features with stable cross-modal expression, and obtains high repeatability key point pairs through end-to-end constraint learning;

[0014] The application uses an efficient lightweight backbone network to extract features of multi-source remote sensing images, fuses a VGG type reparameterization multi-layer encoding structure and a rotation invariant convolution layer, and considers the scale and rotation robustness of the feature map, which is used for subsequent deep structure feature modeling and key point detection.

[0015] 2. The introduction of the deep self-similar structure feature decoder solves the problems of non-linear radiation distortion and cross-modal difference in multi-source remote sensing images:

[0016] The application proposes a deep structure feature descriptor based on deep structure self-similarity, which captures long-distance and stable expression of complex structure information of ground objects from remote sensing images, and solves the problems of nonlinear radiation distortion and cross-modal difference of multi-source remote sensing image features.

[0017] The application decodes the structure feature modeling and key point detection process in parallel, enhances the cross-modal robustness of feature description and the positioning accuracy of key point detection by introducing an extended neighborhood and constructing a deep self-similarity structure feature descriptor.

[0018] 3. The joint loss function constructed based on clustering perception contrast constraint solves the problem of poor feature distinguishability between the same name key point pairs:

[0019] The application proposes an end-to-end joint optimization strategy based on clustering perception contrast learning, which simultaneously improves the feature similarity of the same name key point pairs and the feature distinguishability of different key point pairs by joint constraint learning of deep structure features and key point positions, and ensures the uniform distribution of key points. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is the image registration flowchart of the application;

[0021] Figure 2 is the sampling point selection schematic diagram in the convolution layer;

[0022] Figure 3 is the deep structure feature decoder schematic diagram of cross-modal stable expression;

[0023] Figure 4 is the key point pair matching connection diagram;

[0024] Figure 5 is the chessboard visualized diagram of the registered image; DETAILED DESCRIPTION

[0025] In order to better understand the purpose, structure and function of the application, the application will be further described in detail below in combination with the drawings.

[0026] The application is based on solving the image registration problem of multi-source remote sensing image fusion processing task, and proposes a multi-source remote sensing image deep learning registration framework. The framework is divided into two stages of low-level feature dense coding and deep structure sparse decoding. First, the designed multi-scale rotation invariant feature encoder improves the scale robustness and rotation invariance of the encoded features without increasing the model parameters; second, the designed deep self-similarity structure feature decoder can not only establish a cross-modal robust structure feature model for complex structure information in remote sensing images, but also effectively suppress abnormal value noise interference in remote sensing images; finally, the designed joint loss function decodes the feature extraction and key point detection process in parallel, while improving the spatial high accuracy of the feature and the temporal high repeatability of the key point.

[0027] Step one: using a lightweight multi-scale rotation invariant feature encoder, sharing its calculation and cross-modal representation ability through a twin network, extracting features in multi-source remote sensing images, and ensuring robustness at different scales and angles;

[0028] and respectively represent the input reference image and the perception image, . input and into the twin four-stage encoder, each stage of the four-stage encoder is stacked in a single branch by a step-2 convolution layer and several step-1 convolution layers, as shown in Figure 1 . In the step-2 convolution layer, the input feature map is processed by two parallel 3x3 convolutions, 1x1 convolution and feature encoder layer, and finally processed by a ReLU activation function. The step-2 convolution can obtain a feature map with half size. In the step-1 convolution layer, the output feature map size remains unchanged. The step-1 convolution layer adds a residual connection identity branch to suppress the loss of deep convolution information. The above process is expressed by mathematical formula as:

[0029]

[0030]

[0031] In the formula, and respectively represent the input and output feature maps. represents the batch normalization layer. represents the pixel-wise addition. represents the ReLU activation function. represents the convolution layer with the convolution kernel size and the step size s. The number of convolution layers of the four-stage encoder is designed as 1, 2, 4 and 6 respectively. The feature map size of the four-stage encoder is , , , After passing through an efficient multi-layer encoder, a densely distributed local feature pyramid was obtained. Used for later feature decoding and keypoint estimation.

[0032] Traditional standard convolution operations are used to characterize local image features and have good translation invariance, but lack rotation invariance. This invention introduces a special rotation-invariant convolution (RIConv) to replace the standard convolutional layers in the encoder, enabling the proposed encoder to capture rotation-invariant multi-level features in remote sensing images.

[0033] First, define a fixed point on the feature map F. For convolution kernel size of Traditional convolutions distribute sampling points across a regular network. In the middle, n is a non-negative integer. For example, when n=3, It can be represented as

[0034]

[0035] effect Traditional convolution at the point The surrounding samples are sampled and weighted, and then summed, which can be expressed as:

[0036]

[0037] In the formula, express Sampling points in Represent a Learnable convolutional kernels. For convolutional kernels with a size of... Rotation-invariant convolution, whose sampling points are distributed in the order of... A circle with center r and radius r superior, .like Figure 2 As shown. The number of sampling points is Set the origin of the coordinate system. and The intersection of the extended line with the circumference is recorded as the first sampling point. Eightr sampling points are evenly distributed counter-clockwise on each circumference. (Except for...) In addition, other sampling points It can be expressed by a mathematical formula as follows

[0038]

[0039] In the formula, express With the origin of the coordinate system The angle between the connection and the coordinate axis, i.e. . Assuming F rotates degrees around , the feature map G is obtained. The corresponding point in G is , and the convolution sampling point at is obtained

[0040]

[0041] wherein, denotes the angle The corresponding rotation transformation matrix. Then, the convolution operation after rotation can be expressed as

[0042]

[0043] Step two: capture the stable expression feature descriptors in the multi-source remote sensing image using the deep structure self-similarity-based deep structure feature decoder, and use the key point decoder to gradually fuse the multi-scale key point heat map to obtain the key point detection result map;

[0044] A deep structure feature decoder with cross-modal stable expression is designed to calculate the cross-modal robust long-distance context structure features, as shown in Figure 3 It is proposed to excavate deep structure information from the feature level, which can not only capture high-order structure features outside the local neighborhood context, but also suppress the interference of remote sensing image noise through sparse sampling.

[0045] For the jth layer of encoded feature map and the defined 2D neighborhood layout, the mathematical formula of the neighborhood distribution similarity descriptor is as follows:

[0046]

[0047] In the formula, p is the center position, q and represent the non-repeated feature elements in the neighborhood N(x) of x. The square sum of feature difference is used to perceive the context structure features. In order to perceive the structural similarity between multi-source images across modalities, the noise estimator in the neighborhood distribution is defined as

[0048]

[0049] wherein, denotes the square difference mean of all feature elements in the neighborhood distribution, which is used for similarity descriptor normalization. According to the neighborhood construction method, a 2D neighborhood layout N is created, and the calculation range of feature elements is limited within 8 neighborhoods, and the distance between each other is 2 and ​This generates 8 unique sampling distances. To refine the perception of structural similarity, an extended neighborhood distribution of 8-NH was constructed. ,like Figure 3 As shown. The sampling points are located within a 5×5 neighborhood of x, and the distances between the sampling points involved in the calculation are 4 and 5. . use and These two neighborhood distributions compute context structure feature descriptors, forming a 4-dimensional feature map. Feature map It includes stable long-range contextual structure features of multi-source remote sensing images.

[0050] Will The input is fed into a linear projection layer, resulting in a compact intermediate layer feature representation. Finally, feedforward convolutional layers are used to... Further decoding into deep structural features A non-learning upsampling operation is used to upsample the feature map. Bicubic interpolation is performed to restore the spatial distribution of the feature maps, and then L2 normalization is used to activate the feature maps to obtain unlearned structural feature maps. .

[0051] The keypoint detection output layer first concatenates the heatmaps calculated by the four decoder layers along the channel dimension. Then, it uses a 3×3 convolutional layer to adjust the dimension of the heatmap back to 1 dimension and uses Softmax to obtain the keypoint heatmap. .

[0052] Step 3: Construct a joint loss function using cluster-aware contrastive learning and keypoint detection loss. Extract keypoint pairs with high spatial accuracy and high temporal repeatability from the image to be registered through end-to-end learning, and calculate the spatial transformation matrix to achieve adaptive image registration.

[0053] To further filter out the correct key point pairs, this invention introduces a cluster-aware contrastive learning paradigm.

[0054] Let D and D' represent the depth structure features of the two images to be registered, respectively. Let Y and Y' represent the pseudo-ground truth labels of keypoints in the two images to be registered, respectively. This indicates the number of keypoints in the pseudo-ground truth labels. Indicates D in spatial index The feature vector at, where Similarly, Indicates D' in spatial index The feature vector at that location. The clustering contrastive loss function is defined as:

[0055]

[0056] wherein, is a constant for adjusting the similarity distribution. When i = j, and are positive pairs of the same class, when , and are negative pairs of different classes. That is, the clustering contrast loss function can constrain the minimum intra-class distance and the maximum inter-class distance.

[0057] In order to simultaneously constrain the learning of feature descriptors and key point position information, a joint loss function is designed. The joint loss function includes a clustering-aware loss function and a key point detection loss function . Due to the large difference in the number of key points and non-key points in remote sensing images, a weighted binary cross-entropy loss is introduced to construct the key point detection loss function, and its mathematical formula is represented as:

[0058]

[0059] wherein, is a constant for balancing the difference in the number of key points and non-key points; represents the value of the heat map at the coordinate , represents the value of the ground truth label at the coordinate .

[0060] The joint loss function combines the clustering-aware loss function and the key point detection loss function in an additive manner, and is represented as:

[0061]

[0062] wherein, represents a weight parameter of the clustering contrast constraint, which is set to 0.6 by default in the experiment.

[0063] The image registration method in the present application will be applied to a specific embodiment below to show the technical effects achieved.

[0064] The specific network structure of the image registration method in the present embodiment is shown in Figure 1 , which has been described in detail above and will not be repeated here. The specific process of the present embodiment can be divided into three stages, namely, data set preparation, model training and verification, testing and visualization. The testing and visualization results are shown in Figure 4 and Figure 5 , wherein,Figure 4 a is an urban scene image; Figure 4 b is an airport scene image; Figure 5 a is an urban scene image; Figure 5 b is an airport scene image.

[0065] 1. Dataset preparation

[0066] This embodiment takes a multi-source remote sensing image registration dataset SOPatch as an example.

[0067] The SOPatch dataset is an optical-SAR image dataset, which is derived from Sentinel-2A / B, Sentinel-1A / B, Gaofen-3 satellite and Google Earth database, and the image resolution includes 10m and 1m, covering multiple cities in a global range, including forest, desert, mountain, city and lake and other scene types. On the basis of the above to-be-registered image pairs, preprocessing is performed, including random flipping, random scaling [0.8, 1.2], random rotation [-35°, 35°] and random translation [-20 pixels, 20 pixels]. The image is cropped to 512x512, and the dataset is divided into training set, validation set and test set according to the ratio of 7:1:2.

[0068] 2. Model training and validation

[0069] Step 1: obtain the training set data and validation set data as described above, and divide the training set and the validation set into batches according to a certain batch size, and the total number is N.

[0070] Step 2: sequentially select a batch of samples with index i from the training dataset and the validation dataset, wherein i∈{0, 1,..., N}. The image registration network of the present application is trained using each batch of training samples, and is verified and evaluated after each training. In the training process, the joint loss function of each training sample is calculated. On this basis, the parameters of the network are comprehensively adjusted according to the total loss of each batch of training samples, until the training is completed. After a given number of iterations is completed, the model converges, the training is completed, and the validation results in the training process are obtained. At the same time, the input image is randomly flipped horizontally, vertically and rotated during training for data augmentation.

[0071] 3. Test and visualization

[0072] After the image registration network of the present application is trained, the to-be-registered image pairs of the test dataset are input, and the output image registration results include the to-be-registered image after spatial alignment transformation, the connection line diagram between the image key points, and the chessboard distribution diagram of the registered image, thereby completing the image registration task.

[0073] The present application uses the classical evaluation indicators to quantitatively evaluate on the above remote sensing image registration data set, respectively, root mean square error (RMSE), accuracy (Acc), the number of registration image pairs with RMSE lower than a certain threshold The ratio between the total number of images completed registration is defined as the Acc at this time, for example, when , the accuracy is represented by .

[0074] In this embodiment, part of the test visualization results are shown in Figure 4 and Figure 5 , and the test quantitative results are shown in Table 1.

[0075] Table 1 Test quantitative results

[0076]

[0077] According to the quantitative results in Figure 4 , Figure 5 and Table 1, it can be known that the image registration method of the present application can well complete the image registration task for optical-SAR multi-source remote sensing images. As shown in Figure 4 , for multi-source remote sensing images in complex scenes, the image registration method of the present application can obtain uniformly distributed correct matching key point pairs, which benefits from the proposed multi-scale rotation invariant feature encoder and deep structure feature decoder, which ensures the stable expression ability of the extracted key point pairs in complex scenes and suppresses the false matching situation. It can be seen from Figure 5 that the method of the present application can perform fine spatio-temporal alignment on the texture details in the image, and has good image registration performance.

[0078] It can be understood that the present application is described through some embodiments, and those skilled in the art know that various changes or equivalent replacements can be made to these features and embodiments without departing from the spirit and scope of the present application. In addition, these features and embodiments can be modified to adapt to specific conditions and materials under the guidance of the present application without departing from the spirit and scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application are within the scope of the present application.

Claims

1. A multi-source remote sensing image registration method based on cross-modal robust feature decoupling, characterized by: The method comprises the following steps: Step one: using a lightweight multi-scale rotation invariant feature encoder, sharing its calculation and cross-modal representation ability through a twin network, extracting features in multi-source remote sensing images, and ensuring robustness at different scales and perspectives; Step two: using a deep structure feature decoder based on deep structure self-similarity to capture stable feature descriptors in multi-source remote sensing images, and using a key point decoder to gradually fuse multi-scale key point heat maps to obtain a key point detection result map; Step three: using a clustering-aware contrastive learning and key point detection loss to construct a joint loss function, extracting key point pairs with high spatial accuracy and high temporal repeatability in the image to be registered through end-to-end learning, and calculating a spatial transformation matrix to realize adaptive image registration.

2. The multi-source remote sensing image registration method based on cross-modal robust feature decoupling according to claim 1, characterized in that: The specific process of S1 is: The reference image and the perception image The input twin four-stage encoder, each stage is stacked in a single branch manner by one step 2 convolution layer and several step 1 convolution layers, in the step 2 convolution layer, the input feature map is subjected to two parallel 3*3 convolution, 1*1 convolution and feature encoder layer, and finally subjected to a ReLU activation function, and the step 2 convolution can obtain a feature map with a size halved; In the convolution layer with a step size of 1, the size of the output feature map remains unchanged, and the convolution layer with a step size of 1 adds a residual connection identity branch to suppress the loss of deep convolution information, and the above process is expressed by a mathematical formula as: wherein, and denote input and output feature maps, respectively; denotes a batch normalization layer; denotes a pixel-wise addition; denotes a ReLU activation function; denotes a convolution layer with a kernel size of and a stride of s; The convolutional layer numbers of the four-stage encoder are respectively designed as 1, 2, 4 and 6, and the feature map sizes of the four-stage encoder are respectively , , , ; After the efficient multi-layer encoder, the dense distribution of local feature pyramid is obtained For late feature decoding and key point estimation; Replace the standard convolution layer in the encoder with a special rotation invariant convolution, so that the proposed encoder can capture multi-level features that are rotation invariant in remote sensing images.

3. The multi-source remote sensing image registration method based on cross-modal robust feature decoupling according to claim 2, characterized in that: The specific process of replacing the standard convolution layer in the encoder with a special rotation invariant convolution in S1 is: First, define a fixed point on the feature map F. For convolution kernel size Traditional convolutions distribute sampling points across a regular network. In this context, n is a non-negative integer. Action Traditional convolution at a point samples the neighborhood around and sums the weighted contributions, expressed as: In the formula, express Sampling points in Represent a Learnable convolutional kernels, for convolutional kernels with a size of Rotation-invariant convolution, whose sampling points are distributed in the order of... A circle with center r and radius r superior, The number of sampling points is Set the origin of the coordinate system and The intersection of the extended line with the circumference is recorded as the first sampling point. Eightr sampling points are evenly distributed counter-clockwise on each circumference, except... In addition, other sampling points Expressed mathematically as follows: In the formula, denotes the angle between the line connecting the origin of the coordinate system and the coordinate axis, i.e. , assuming that F rotates degrees around to obtain a feature map G, the corresponding point in G is , and further obtaining the convolution sampling point at is wherein, denotes an angle The corresponding rotation transformation matrix, the rotated convolution operation is represented as 。 4. The multi-source remote sensing image registration method based on cross-modal robust feature decoupling according to claim 1, characterized in that: The specific process of S2 is: For the j-th layer encoded feature map And the defined 2D neighborhood layout, the mathematical formula representation of the neighborhood distribution similarity descriptor is as follows: where p is the center position, q and representing the non-repeated feature elements in the neighborhood N(x) of x, using the squared sum of feature differences to perceive the context structure feature, The noise estimator in the neighborhood distribution is defined as wherein, represents the mean of squared differences of all feature elements in the neighborhood distribution, used for similarity descriptor normalization, according to the neighborhood construction method, create a 2D neighborhood layout N, limit the calculation range of feature elements to 8 neighborhoods, and the distance between each other is 2 and , 8 unique sampling distances are generated, Constructing 8-NH extended neighborhood distribution , Established in the 5 × 5 neighborhood of x, and the sampling points involved in the calculation are 4 and , using and Contextual structure feature descriptors are calculated to form a 4-dimensional feature map , the feature map contains stable long-distance contextual structure features of multi-source remote sensing images, Will The input is fed into a linear projection layer, resulting in a compact intermediate layer feature representation. Finally, feedforward convolutional layers are used to... Further decoding into deep structural features A non-learning upsampling operation is used to upsample the feature map. Bicubic interpolation is performed to restore the spatial distribution of the feature maps, and then L2 normalization is used to activate the feature maps to obtain unlearned structural feature maps. , The key point detection output layer first splices the heat maps calculated by the 4-layer decoder along the channel dimension, and then adjusts the dimension of the heat map to 1 dimension again through a convolution layer with a convolution kernel of 3*3, and obtains the key point heat map by using Softmax .

5. The multi-source remote sensing image registration method based on cross-modal robust feature decoupling according to claim 1, characterized in that: In the S3, the joint loss function includes a clustering-aware loss function and a keypoint detection loss function ; The clustering contrastive loss function is defined as: wherein represents a constant used to adjust the similarity distribution; represents the number of keypoints in the pseudo ground truth label, represents a feature vector of D at a spatial index , wherein , represents a feature vector of D' at a spatial index ; D and D' represent the deep structure features of two images to be registered, respectively; When i = j, and are positive pairs of the same class, when , and are negative pairs of different classes; that is, the clustering contrast loss function can constrain the minimum intra-class distance and the maximum inter-class distance. The key point detection loss function The mathematical formula is represented as: wherein, is a constant to balance the difference in the number of key points and non-key points; denotes a heat map denotes the value of the coordinate in the ground truth label denotes a ground truth label denotes the value of the coordinate in the ground truth label The joint loss function combines the clustering-aware loss function and the key point detection loss function in an additive manner, and is expressed as: In the formula, denotes a weight parameter of the clustering contrast constraint.

Images