A Multi-Temporal Remote Sensing Image Registration Method Based on Dual-View Fully Convolutional Neural Network

Abstract

This invention relates to a multi-temporal remote sensing image registration method based on a dual-view fully convolutional neural network. This invention combines transfer learning by adding an AlexNet branch to the existing pre-trained VGG16 network. The VGG16 network focuses on local image features, while the AlexNet network focuses on global image features. A dense connection structure is used to improve the AlexNet network, enhancing feature reuse. The dual-branch network improves the grid loss function by modifying the original single-branch grid loss function based on the four parameters obtained from the regression of the two networks, avoiding errors inherent in single-branch grid loss functions. Low-level semantic information is transferred to high-level semantic information, thereby enhancing feature fusion between the two. This preserves more edge and shape information while ensuring sufficiently rich extracted semantic information, further improving the accuracy of multi-temporal remote sensing image registration.

Description

Technical Field

[0001] This invention relates to an image processing technique, and more particularly to a multi-temporal remote sensing image registration method based on a dual-view fully convolutional neural network. Background Technology

[0002] Multi-temporal remote sensing image registration is the process of matching and aligning pairs of remote sensing images captured under different time conditions in the same or similar scenes using remote sensing camera tools, and finding the geometric correspondence between the image pairs. It is the foundation of visual tasks such as remote sensing image change detection and remote sensing image ground object recognition. How to design a high-precision and high-efficiency deep learning model is one of the current hot topics.

[0003] Multi-temporal remote sensing image registration is mainly divided into two methods: image grayscale-based and image feature-based. Image grayscale-based methods register by calculating the grayscale similarity between image pairs, but the registration results are poor for two images with significant grayscale changes. Image feature-based methods are efficient, accurate, and robust, and have become the mainstream approach. Yang proposed a registration method using a pre-trained VGG16 network for feature extraction, but it performs poorly and is time-consuming for multi-temporal image pairs with large viewpoint differences. Zhang established and trained a Siamese fully convolutional network using a novel loss function. By connecting the two branches of the Siamese network through convolution operations, the similarity score between the two input image patches is obtained, thus completing the registration process. This method achieved good registration results for image pairs with different modalities, but performed poorly for multi-temporal image pairs with large viewpoint differences. Park uses a pre-trained SE-ResNeXt101 network with channel attention as the feature extraction network and introduces an ensemble method based on bidirectional networks. He proposes a two-stream network registration method that combines the source image, the target image, and the enhanced target image, which solves the asymmetry of image registration to some extent and improves the accuracy and efficiency of image registration. Summary of the Invention

[0004] To address the issues of low accuracy and inefficiency in multi-temporal remote sensing image registration, a multi-temporal remote sensing image registration method based on a dual-view fully convolutional neural network is proposed to improve the accuracy and efficiency of multi-temporal remote sensing image registration.

[0005] The technical solution of this invention is: a dual-view fully convolutional neural network for multi-temporal remote sensing image registration, which is composed of VGG16 network branches and ALexNet network branches;

[0006] The VGG16 network branch is a network that removes the fifth convolutional layer and all fully connected layers of the VGG16 network; the AlexNet network branch removes the fifth convolutional layer and all fully connected layers of the AlexNet network, and uses bilinear interpolation to upsample the image before the image input. Dense connection structures across layers are added to the second and third convolutional layers of the AlexNet network, and a smoothing layer is added to the fourth convolutional layer of the AlexNet network. This smoothing layer consists of three convolutional layers with the same kernel size.

[0007] The outputs of the two network branches are merged into the output of a dual-view fully convolutional neural network.

[0008] A multi-temporal remote sensing image registration system based on a dual-view fully convolutional neural network is proposed. The target image and the source image are used as the first image pair, and the enhanced target image and the source image are used as the second image pair. The two image pairs are respectively fed into a VGG16 network branch and a dual-view fully convolutional neural network for feature extraction, feature matching and correlation parameter adjustment. The feature parameters of the source image and the feature parameters of the target image extracted by the VGG16 network branch are arithmetically averaged to obtain the synthesized transformation parameters, which are used to guide the affine transformation of the source image to complete the image registration.

[0009] A multi-temporal remote sensing image registration method based on a dual-view fully convolutional neural network is proposed. A multi-temporal remote sensing image registration system based on a dual-view fully convolutional neural network is established, and the network is trained. The trained network is then used for multi-temporal remote sensing image registration. The network training method is as follows:

[0010] 1) Data augmentation: The source image and the target image are used as a pair of input samples for model training. Then, random color jitter is applied to the target image to obtain the augmented target image. The source image and the augmented target image are used as another pair of input samples for model training.

[0011] 2) A dual-view fully convolutional neural network is used to extract the feature information of the input image pair and outputs the extracted source image feature S0, target image feature T0, and enhanced target image feature T′0 to the next feature matching layer; a VGG16 network branch is used to extract the feature information of the input image pair and outputs the extracted source image feature S, target image feature T, and enhanced target image feature T′ to the next feature matching layer.

[0012] 3) For the six image features obtained in step 2), the correlation between source image feature S and target image feature T, source image feature S and enhanced target image feature T′, source image feature S0 and target image feature T0, and source image feature S0 and enhanced target image feature T′0 are calculated bidirectionally using the bidirectional correlation matching method. The resulting eight bidirectional relationships are input into the parameter regression network layer.

[0013] 4) Input the bidirectional relationship obtained in step 3) into the regression network to perform parameter regression, thereby obtaining eight parameters. Use the eight parameters to improve the grid loss function, train the network, and adjust the network parameters.

[0014] Furthermore, the specific implementation method of step 3) bidirectional correlation matching method is as follows:

[0015] Based on the coordinate positions f of all features of the target image feature T respectively T (i,j), the coordinate positions of all features of the enhanced target image feature T′ f T′ (i,j) are successively compared with the k-th feature f in the source image feature S. S (i k ,j k The relevant mapping is performed to form the corresponding correlation. Then, the ReLU activation function and normalization are used to channelize the two correlations mentioned above, thereby generating the corresponding correlation C from the source image features S to the target image features T. ST The correspondence between source image features S and enhanced target image features T′ is C ST′ As shown below:

[0016] C ST (i,j,k)=f T (i,j) k f S (i k ,j k )

[0017] C ST′ (i,j,k)=f T′ (i,j) k f S (i k ,j k )

[0018] Similarly, in the feature matching layer, based on the coordinate positions f of all features of the source image feature S... S (i,j) are compared one by one with the k-th feature f in the target image feature T. T (i k ,j k ), enhance the k-th feature f of the target image feature T′ T′ (ik ,j k The two correlation relationships are then mapped separately to form corresponding relationships. Next, the ReLU activation function and normalization are used to channelize these two relationships, thereby generating the corresponding correlation C from source image features S to target image features T. TS Correlation C between source image S and enhanced target image T′ T′S As shown below:

[0019] C TS (i,j,k)=f S (i,j) k f T (i k ,j k )

[0020] C T′S (i,j,k)=f S (i,j) k f T′ (i k ,j k )

[0021] All features are sequentially mapped to obtain four corresponding correlation relationships: C ST C TS C ST' and C T'S .

[0022] Similarly, by repeating the correlation acquisition operation above on the features obtained from the dual-view fully convolutional neural network, four other corresponding correlation relationships are obtained: and

[0023] Furthermore, the mesh loss function in step 4) is improved as follows:

[0024] The grid loss function for the single-branch VGG16 network is shown below:

[0025] This is the feature loss between the source image features S and the target image features T;

[0026] This is the feature loss between the source image features S and the enhanced target image features T′;

[0027] This is the feature loss between the source image features S, the target image features T, and the enhanced target image features T′;

[0028] L1=αL a1 +βL b1 +γLc1 This is the total loss, where α, β, and γ are the weights assigned to the three feature losses. The grid loss function for a dual-view fully convolutional neural network is similarly as follows:

[0029]

[0030]

[0031]

[0032] L2=αL a2 +βL b2 +γL c2 The final model mesh loss function is shown below:

[0033]

[0034] The beneficial effects of this invention are as follows: This invention is based on a multi-temporal remote sensing image registration method using a dual-view fully convolutional neural network. This invention combines transfer learning, adding an AlexNet branch to the original pre-trained VGG16 network. The VGG16 network focuses on local image features, while the AlexNet network focuses on global image features. A dense connection structure is used to improve the AlexNet network, enhancing feature reuse. The dual-branch network improves the grid loss function. Based on the four parameters obtained from the regression of the two networks, the original single-branch network grid loss function is improved, avoiding errors from a single network grid loss function and improving registration accuracy. Attached Figure Description

[0035] Figure 1 This is a diagram illustrating the algorithm framework of the method of this invention;

[0036] Figure 2 This is a diagram of the dual-view fully convolutional network structure of the present invention;

[0037] Figure 3 This is a diagram showing the qualitative comparison experimental results of the first set of multi-temporal bay images of the present invention;

[0038] Figure 4 This is a diagram showing the results of a qualitative comparison experiment of the second set of multi-temporal river channel images in this invention. Detailed Implementation

[0039] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0040] Combination Figure 1 , 2The network structure diagram shown illustrates the implementation details of each step in the method of this invention.

[0041] Step 1: Data Augmentation Processing

[0042] Because training deep learning models requires large datasets, and the manual cost of creating datasets is very high, it is essential to expand the dataset sample size using certain processing methods. Based on the target image and source image as the first image pair, this invention enhances the target image by using random color jittering in each round of deep learning model training. The resulting enhanced target image and source image form a second image pair, significantly increasing the number of dataset samples.

[0043] Step 2: Feature extraction based on dual-view fully convolutional neural network

[0044] The described Dual-View Fully Convolutional Neural Network (DFFCNN) combines transfer learning and adds a new network branch, ALexNet, to the original pre-trained VGG16 network (13 convolutional layers and 3 fully connected layers), and improves it with a dense connection structure.

[0045] To reduce the training cost of the model, a VGG16 network pre-trained on ImageNet is used to process local feature information of the image. To retain as much edge and shape information as possible, and to ensure sufficient feature extraction and improve the model's inference speed, the depth of the feature extraction network must be appropriate. Therefore, this invention removes the fifth convolutional layer and all fully connected layers of the VGG16 network. Then, a new network branch, AlexNet (typically 5 convolutional layers and 3 fully connected layers), is added, forming a Dual-View Fully Convolutional Neural Network (DFFCNN) together with the VGG16 network. Based on the requirements for the final output feature map size, the fifth convolutional layer and all fully connected layers of the AlexNet network are removed, and bilinear interpolation is used to upsample the image before input. To utilize shallow, low-level feature information to enhance the network's learning ability, dense connections across input layers are added to the second and third convolutional layers of the AlexNet network. Furthermore, to improve feature map processing capabilities, a smoothing layer is added to the fourth convolutional layer of the AlexNet network. The smoothing layer here consists of three 3×3 convolutional layers with output channel parameters of 384, 256, and 256, respectively. The structure of the Dual-View Fully Convolutional Neural Network (DFFCNN) is as follows: Figure 2 As shown, the same sample is fed into a VGG16 network with 4 convolutional layers and an AlexNet network with 4 convolutional layers and 1 smoothing layer, respectively. The outputs of the two network branches are merged as the extracted features.

[0046] Low-level semantic information contains more fine-grained features, while high-level semantic information contains richer semantic information. As convolutional neural networks increase in depth, low-level semantic information gradually transforms into abstract high-level semantic information, reducing some fine-grained features and causing blurring of edge and shape information. However, in image registration, it is crucial to retain as much low-level semantic information as possible, such as edges and shapes, to ensure alignment of image details during the registration stage, while simultaneously ensuring sufficient feature extraction and improving the model's inference speed. Therefore, the depth of the feature extraction network must be appropriate. This invention improves the AlexNet network with a densely connected structure, inter-transferring low-level and high-level semantic information, thereby enhancing the feature fusion of low-level and high-level semantic information. This preserves more edge and shape information while ensuring sufficiently rich extracted semantic information, further improving the accuracy of multi-temporal remote sensing image registration.

[0047] Step 3: Bidirectional Correlation Feature Matching

[0048] The source image, target image, and enhanced target image are each fed into two network branches for feature extraction. Correlation feature matching is performed on the features output by the VGG16 network branch, and correlation feature matching is performed on the features output by the combined output of the two network branches.

[0049] Based on the feature extraction of the VGG16 network, this invention generates source image features S and target image features T, as well as source image features S and enhanced target image features T′, respectively, and then performs feature matching on the two sets of corresponding correlations.

[0050] The following explanation uses bidirectional correlation feature matching based on feature information extracted from the VGG16 network.

[0051] In the feature matching layer, based on the coordinate positions f of all features of the target image T... T (i,j), the coordinate positions of all features of the enhanced target image T′ f T′ (i,j) are successively compared with the k-th feature f in the source image S. S (i k ,j k The relevant mapping is performed to form the corresponding correlation. Then, the ReLU activation function and normalization are used to channelize the two correlations mentioned above, thereby generating the corresponding correlation C from the source image S to the target image T. ST Correlation C between source image S and enhanced target image T′ ST′ As shown below:

[0052] C ST (i,j,k)=f T (i,j) k fS (i k ,j k )

[0053] C ST′ (i,j,k)=f T′ (i,j) k f S (i k ,j k )

[0054] Similarly, in the feature matching layer, based on the coordinate positions f of all features in the source image S... S (i,i) successively compare with the k-th feature f in the target image T. T (i k ,j k ), enhance the k-th feature f of the target image T' T' (i k ,j k The two correlation relationships are then mapped separately to form corresponding relationships. Next, the ReLU activation function and normalization are used to channelize these two relationships, thereby generating the corresponding correlation C from the source image S to the target image T. TS The correspondence between the source image S and the enhanced target image T' is C T'S As shown below:

[0055] C TS (i,j,k)=f S (i,j) k f T (i k ,j k )

[0056] C T′S (i,j,k)=f S (i,j) k f T′ (i k ,j k )

[0057] All features are sequentially mapped to obtain four corresponding correlation relationships: C ST C TS C ST' and C T'S .

[0058] Similarly, this invention performs an addition operation on the features extracted by the VGG16 network and the AlexNet network to generate the source image features S0 and target image features T0, and the source image features S0 and enhanced target image features T′0 for the corresponding networks. Then, the above correlation acquisition operation is repeated to obtain four other corresponding correlation relationships: and

[0059] Step 4: Network Loss Function

[0060] Improvement of the mesh loss function. Because a dual-branch network is used to extract image features, the original mesh loss function applied to the single-branch network is improved to avoid errors caused by the mesh loss function of the single-branch network. The original mesh loss function applied to the single-branch VGG16 network is as follows:

[0061] This is the feature loss between the source image features S and the target image features T;

[0062] This is the feature loss between the source image features S and the enhanced target image features T′;

[0063] This is the feature loss between the source image features S, the target image features T, and the enhanced target image features T′; L1 = αL a1 +βL b1 +γL c1 This is the total loss, and α, β, and γ are the weights assigned to the three feature losses.

[0064] After adding a new network branch, the VGG16+AlexNet mesh loss function, obtained by adding the two network branches, is similar to the previous one, as shown below:

[0065]

[0066]

[0067]

[0068] L2=αL a2 +βL b2 +γL c2 The final model mesh loss function is shown below:

[0069]

[0070] Step 5: Parameter processing during the training of the original VGG16 network. This involves processing the parameters obtained from the parameter regression network layer of the VGG16 network. and parameters The final transformation parameters θ are obtained by performing synthesis. VGG16 This is used to guide the affine transformation of the source image to complete image registration. The formula for calculating the synthesis parameters is shown below:

[0071]

[0072] The deep learning algorithm proposed in this invention is trained and tested using publicly available aerial remote sensing image datasets and multi-temporal remote sensing image datasets. It is compared with the CNN-registration algorithm and the RMNet algorithm proposed in recent years.

[0073] The system is based on the PyTorch deep learning framework, uses Python version 3.6, and is equipped with a GTX1080Ti graphics card with 8GB of video memory and an Intel Core i7 (4.20GHz) processor.

[0074] Experimental Analysis and Comparison:

[0075] Figure 3 The results of the comparative experiments on the first set of multi-temporal bay image pairs are presented, showcasing the CNN-registration algorithm, the RMNet algorithm, and the algorithm proposed in this invention. In the comparison of registered and overlapping images, the CNN-registration method clearly fails to register, while the other two methods show no difference in registration performance. However, in the comparison of checkerboard pattern details, the method proposed in this invention achieves better registration results, slightly outperforming the RMNet algorithm in bay line alignment, and significantly outperforming the CNN-registration algorithm.

[0076] Figure 4 This presentation showcases the comparative experimental results of a second set of multi-temporal river channel image pairs obtained by three algorithms: CNN-registration, RMNet, and the algorithm proposed in this invention, under conditions of significant viewpoint angle differences. In comparison of registered and overlaid images, our proposed method significantly outperforms the others. However, in the checkerboard detail image, it can be observed that our proposed method essentially aligns the shape of the river channel in the image.

[0077] The registration accuracy of the three algorithms is quantitatively evaluated using the Correct Keypoint Estimation (PCK) metric, the Root Mean Square Error (RMSE) metric, and the Time (TIME: in seconds) metric. The formulas for calculating PCK and RMSE are shown below:

[0078]

[0079] In the formula, α·max(h,w) represents a region of size α in an image region with height h and width w. For the key points in the source image obtained by transforming the i-th image pair, For the manually labeled key points in the target image of the i-th image pair, Here, d represents the distance between the transformed point and the original point within a certain range, and M represents the number of points.

[0080]

[0081] In the formula, M represents the pixel in the image, (x i ,y i (x′) represents the pixel obtained in the registered image. i ,y′ i () are manually marked pixels in the target image.

[0082] The higher the values ​​of PCK and RMSE, the higher the accuracy and the better the registration effect; the lower the value of time, the higher the efficiency of image registration. Table 1 shows a quantitative comparison of different algorithms, indicating that the proposed algorithm has the highest values ​​of PCK and RMSE and the lowest value of time, effectively improving the accuracy and efficiency of multi-temporal remote sensing image registration to a certain extent.

[0083] Table 1

[0084]

[0085] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A multi-temporal remote sensing image registration method based on a dual-view fully convolutional neural network, characterized in that, Specifically, the steps include the following: 1) Data augmentation: The source image and the target image are used as a pair of input samples for model training. Then, random color jitter is applied to the target image to obtain the augmented target image. The source image and the augmented target image are used as another pair of input samples for model training. 2) The AlexNet network is used to extract the feature information of the input image pair and outputs the extracted source image feature S0, target image feature T0 and enhanced target image feature T0′ to the next feature matching layer; the VGG16 network branch is used to extract the feature information of the input image pair and outputs the extracted source image feature S, target image feature T and enhanced target image feature T′ to the next feature matching layer. 3) For the six image features obtained in step 2), the correlation between source image feature S and target image feature T, source image feature S and enhanced target image feature T′, source image feature S0 and target image feature T0, and source image feature S0 and enhanced target image feature T0′ are calculated bidirectionally using the bidirectional correlation matching method. The resulting eight bidirectional relationships are input into the parameter regression network layer. 4) Input the bidirectional relationship obtained in step 3) into the regression network to perform parameter regression, thereby obtaining eight parameters. Use the eight parameters to improve the grid loss function, train the network, and adjust the network parameters. 5) The source image feature parameters and target image feature parameters extracted from the VGG16 network branch are arithmetically averaged to obtain the synthesized transformation parameters, which are used to guide the source image to perform affine transformation and complete image registration.

2. The multi-temporal remote sensing image registration method based on a dual-view fully convolutional neural network according to claim 1, characterized in that, Improvement of the mesh loss function in step 4): The grid loss function for the single-branch VGG16 network is shown below: This is the feature loss between the source image features S and the target image features T; This is the feature loss between the source image features S and the enhanced target image features T′; This is the feature loss between the source image features S, the target image features T, and the enhanced target image features T′; L1=αL a1 +βL b1 +γL c1 This is the total loss, and α, β, and γ are the weights assigned to the three feature losses. The grid loss function for a dual-view fully convolutional neural network is similar to the one above, as shown below: L2=αL a2 +βL b2 +γL c2 The final model mesh loss function is shown below:

Images