Deep Learning SAR Image Registration Method Based on Dual Transformation Network

Abstract

The present invention discloses a deep learning SAR image registration algorithm based on a dual transformation network, which mainly solves the problems of difficult acquisition of matching image pairs, low matching accuracy, too high cost of manually labeled data sets and class imbalance in existing deep learning SAR registration methods. The implementation scheme is as follows: 1) Generate training and test data sets. 2) Set the number of categories of the classification head of the deep image classification network and construct two branches of the dual transformation network; 3) Train the two branches of the dual transformation network and perform preliminary matching using the trained dual transformation network, 4) Perform fine matching on the basis of the preliminary matching; 5) Take the point pairs jointly recognized by the two branches and update the point pairs according to the results of the fine matching to obtain the final matching result. The present invention is simple in obtaining image pairs and does not require manual annotation, the data set has balanced categories, and has high matching accuracy, and can be used for geological mineral resource detection and digital terrain model extraction.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and further relates to a SAR image registration method, which can be used for geological mineral resource detection and digital terrain model extraction. Background Art

[0002] SAR image registration mainly deals with two or more SAR images captured under different conditions of the same scene. By exploring the corresponding relationships between the images, the most suitable transformation model is found, so as to transform the SAR images obtained under different times and different physical environment states by using the transformation model and transform them into the same coordinate system. Under different imaging conditions, there are some differences in many aspects such as the spatial position and imaging mode of SAR images. Only by transforming them into the same reference system can they be conveniently processed. Therefore, the SAR image registration technology is an important technical prerequisite for SAR image target detection, image fusion, and feature extraction.

[0003] Scholars such as Zitova published a paper in 2003, summarizing the registration method. The image registration technology is mainly implemented in four steps, namely feature extraction, feature matching, estimating the transformation model, and transforming the image. Generally speaking, image registration is divided into two types. One is the traditional registration method, such as the gray-based image registration method, such as the SIFT method proposed by Low et al. in 2004. The other type is the deep learning-based image registration method, which uses a deep neural network to extract features to obtain matching point pair information. For example, scholars such as Han proposed a Siamese network, which extracts features through two exactly the same convolutional neural networks to obtain the matching relationship of image patches. The SAR image registration method based on deep learning is the most common image registration method in recent years.

[0004] Xidian University disclosed a "Fine Registration Method for SAR Images Based on Deep Learning" in its patent application document with the application number: 201910943154.7. The implementation steps are as follows: First, obtain multiple SAR images of the same scene observed from different perspectives, and form a dataset from these images; then construct a neural network model for fine registration of SAR images, which is composed of a sub-convolutional neural network for correcting the overall deformation between SAR images and a sub-residual neural network for correcting the local deformation between SAR images in sequence; finally, construct a loss function Loss for the neural network for fine registration of SAR images; then use the constructed dataset to train the network to obtain a trained network model; finally, directly input the reference SAR image and the SAR image to be registered, and the network will output the registered image. The network constructed by this method can correct the overall deformation and local distortion between SAR images, thus improving the registration performance. However, the deficiencies of this method are as follows: It directly randomly selects samples from the constructed dataset for training, and the acquisition of high-quality SAR images is costly, so this will lead to an increase in the cost of obtaining the dataset. In addition, the computational cost of this method is relatively large.

[0005] Shasha Mao et al. proposed a framework for remote sensing image registration based on deep learning in the paper "Multi-Scale Fused SAR Image Registration Based on Deep Forest" published in the international top journal Remote Sensing in the field of remote sensing. It uses deep forest to train samples at multiple scales, extracts more effective feature information, and uses deep forest for binary classification to distinguish matching point pairs and non-matching point pairs, thus obtaining the final result. There are three deficiencies in this method: First, because it relies too much on matching sample pairs when constructing sample pairs, it will lead to an increase in the cost of constructing sample pairs; second, because the classification results still contain noise points, the normalized cross-correlation method NCC must be used to filter out noise points; third, because the process of constructing the training sample set is lengthy and requires a relatively long training time, it reduces the efficiency of the image registration task. Summary of the Invention

[0006] The purpose of the present invention is to propose a deep learning SAR image registration method based on a dual transformation network in view of the above deficiencies of the prior art, so as to improve the efficiency of the SAR image registration task, reduce the cost of obtaining matching sample pairs, improve the registration accuracy, and achieve accurate registration at the sub-pixel level.

[0007] The technical idea of the present invention is as follows: By using traditional methods to find key points on the reference image and the image to be registered, and intercepting image patches centered on these key points, regarding each key point and the intercepted image patch as a category to solve the problem of difficult acquisition of matching image pairs; by dividing the training of the network into a reference image - image to be registered branch R - S and an image to be registered - reference image branch S - R, and selecting the point pairs recognized by both branches as the final matching result to solve the problem of category mismatch; by designing fine matching to solve the problem of insufficient matching accuracy.

[0008] According to the above idea, the implementation steps of the present invention are as follows:

[0009] (1) Generate a dataset:

[0010] 1a) Use the SIFT method to obtain m key points on the reference image R and n key points on the image S to be registered 1 ≤ i ≤ m, 1 ≤ j ≤ n, and perform preliminary registration on these two, obtaining the initial transformation matrix from the coordinate system of the image to be registered to the coordinate system of the reference image and the initial transformation matrix from the coordinate system of the reference image to the coordinate system of the image to be registered

[0011] 1b) Respectively take P R and P S these key points as the center, and capture the reference image R and the image S to be registered according to the size of 64×64, delete the points that cannot be captured, and the remaining a points of the reference image and b points of the image to be registered are obtained, getting the reference image set and the image set to be registered 1 ≤ C ≤ a, 1 ≤ D ≤ b;

[0012] 1c) Perform t times of affine transformation including 0° - 20° and gray - scale transformation on and respectively. The combination of the and its transformed result gets the reference image set The combination of the and its transformed result gets the image set to be registered Then perform v times of affine transformation with the angle randomly in the range of 0° - 20° on and respectively. The combination of the and its transformed result gets the sample set of the reference image set for verification The combination of the and its transformed result forms the image set to be registered for verification where 150 ≤ t ≤ 250; 40 ≤ v ≤ 100;

[0013] (2) Select an existing depth image classification network, set the number of categories of the classification head, and construct two branches of the double transformation network:

[0014] 2a) Take the key points on the reference image as categories, and on the branch from the reference image to the image to be registered, take the same number of categories as the number of pictures in the reference image set FR. Input these categories into the classification head of the depth image classification network to form the first branch "R-S" of the double transformation network;

[0015] 2b) Take the key points on the image to be registered as categories, and on the branch from the image to be registered to the reference image, take the same number of categories as the number of pictures in the image to be registered set FS. Input these categories into the classification head of the depth image classification network to form the second branch "S-R" of the double transformation network;

[0016] (3) Train the two branches of the network to obtain the double transformation network:

[0017] 3a) Use the reference image set as the training set, and use the reference image set after affine transformation as the validation set. Use the gradient descent method to train the first branch R-S, and after training all the samples of the training data set once, use the validation set to perform a validation. When the accuracy rate reaches the target or above, save the trained model to obtain the trained first branch of the double transformation network;

[0018] 3b) Use the image to be registered set as the training set, and use the image to be registered set after affine transformation as the validation set. Use the gradient descent method to train the second branch S-R, and after training all the samples of the training data set once, use the validation set to perform a validation. When the accuracy rate reaches the target or above, save the trained model to obtain the trained second branch of the double transformation network;

[0019] 3c) Connect the trained first branch and the trained second branch in parallel to form the double transformation network;

[0020] (4) Use the trained double transformation network for preliminary matching:

[0021] 4a) Use the initial transformation matrix transforming from the coordinate system of the image to be registered to the coordinate system of the reference image in 1a) to transform the key points on the image to be registered to the reference image, and delete the points that cannot be captured to generate a point set Take Make a test image set with the points in Input it into the first branch of the trained double transformation network for image classification. If it is classified as the same class, it means that the points on the to-be-registered image corresponding to the test set image and the points on the reference image as the class are matching points, and delete the points whose corresponding image patch similarity is lower than the threshold to complete the preliminary registration of the R-S branch;

[0022] 4b) Use the initial transformation matrix from the to-be-registered image coordinate system to the reference image coordinate system in 1a) Transform the key points on the reference image to the to-be-registered image, and delete the points that cannot be screenshot to generate a point set With Make a test image set with the points in Input it into the second branch of the trained double transformation network for image classification. If it is classified as the same class, it means that the points on the reference image corresponding to the test set image and the points on the to-be-registered image as the class are matching points, and delete the points whose corresponding image patch similarity is lower than the threshold to complete the preliminary registration of the S-R branch;

[0023] (5) Fine matching:

[0024] 5a) Re-select points for screenshot next to the training set points of the preliminary registered R-S branch, update the preliminary registration obtained in 4a) to obtain the points on the R-S branch training set in the point pair set;

[0025] 5b) Re-select points for screenshot next to the training set points of the preliminary registered S-R branch, update the preliminary registration obtained in 4b) to obtain the points on the S-R branch training set in the point pair set;

[0026] 5c) Select the point pairs recognized by both branches, and update them according to the updated matching results in 5a) and 5b) to obtain the final registration result.

[0027] Compared with the prior art, the present invention has the following advantages:

[0028] First, the present invention starts from the perspective of image multi-classification for the first time, regards the image patches intercepted by each key point as a class, and thus trains a multi-class neural network for image classification tasks, abandoning the traditional method of using binary classification for image registration, making the training of the neural network no longer rely on a large number of matching image pairs and improving the training efficiency.

[0029] Second, after obtaining the matching results of the network, the present invention introduces a fine matching link. By selecting new points at intervals of two pixels in the eight directions of up, down, left, right, upper left, lower left, upper right, and lower right of the matching point pair to intercept image blocks for similarity comparison, and then selecting the point with the highest similarity among the nine points as the candidate for the final matching result. Compared with the traditional method, the registration accuracy is further improved.

[0030] Third, different from the single-branch registration of the previous deep learning SAR image registration method, the present invention introduces a dual transformation network, starting from two branches and selecting the results recognized by both branches as the final results, which greatly reduces the proportion of bad points and also solves the problem of class imbalance.

[0031] Fourth, the present invention completes the construction of the data set through simple image enhancement, without requiring a large amount of manpower and material resources, saving costs. Brief Description of the Drawings

[0032] Figure 1 is the implementation flowchart of the present invention;

[0033] Figure 2 is the overall framework diagram of the existing "Swin Transformer" network used in the present invention;

[0034] Figure 3 is the SAR image data set diagram used in the experiment of the present invention. Detailed Embodiment

[0035] The following further describes the embodiments and effects of the invention in detail with reference to the accompanying drawings.

[0036] Refer to Figure 1 , the implementation steps of this example are as follows:

[0037] Step 1, generate training and validation data sets.

[0038] This step generates a data set based on key points. Existing methods for obtaining key points include the SIFT method, the LIFT method, and the SAR-SIFT method. In this example, the SIFT method is used but not limited to generating the data set. The specific implementation is as follows:

[0039] 1.1) Use the SIFT method to obtain m key points in the reference image R and n key points on the image S to be registered 1 ≤ i ≤ m, 1 ≤ j ≤ n, and perform preliminary registration on the two, obtaining the initial transformation matrix from the coordinate system of the image to be registered to the coordinate system of the reference image and the initial transformation matrix from the coordinate system of the reference image to the coordinate system of the image to be registered which are respectively expressed as follows:

[0040]

[0041]

[0042] Among them, a 11 、a 12 、a 13 、a 21 、a 22 、a 23 and b 11 、b 12 、b 13 、b 21 、b 22 、b 23 are all floating-point numbers, which are obtained by least squares method;

[0043] 1.2) Respectively, with and these key points as the center, screenshots of the reference image R and the image S to be registered are taken according to the size of 64X64, and the points that cannot be screenshot are deleted. The remaining a reference image points and b image points to be registered that can be screenshot are obtained, and the reference image set and the image set to be registered are respectively expressed as follows:

[0044]

[0045]

[0046] Among them, 1 ≤ C ≤ a, 1 ≤ D ≤ b, is the image block intercepted from the i-th reference image, 1 ≤ i ≤ a, is the image block intercepted from the j-th image to be registered, 1 ≤ j ≤ b. The size of the screenshot includes but is not limited to 64×64. In this example, these reference images and images to be registered are all 64×64 in size;

[0047] 1.3) Perform t affine transformations and gray-scale transformations on and respectively, including 0° - 20°, 150 ≤ t ≤ 250. The and its transformed results are combined to obtain the reference image set The and the combined transformed results obtain the image set to be registered They are respectively expressed as follows:

[0048]

[0049]

[0050] Among them, Each subset after the f-th random affine transformation, the number of pictures contained in this subset is the same as And the number of its categories is the same as Consistent; ; Each subset after the e-th random affine transformation, the number of pictures contained in this subset is the same as And the number of its categories is the same as Consistent; In this example, t = 200 is taken; ;

[0051] 1.4) Perform v times of affine transformations on And respectively, with the angle randomly in the range of 0° - 20°, 40 ≤ v ≤ 100. The And its transformed result are combined to obtain the reference image set sample set for verification The And its transformed result form the set of images to be registered for verification Are respectively expressed as follows:

[0052]

[0053]

[0054] Among them, Each subset after the l-th random affine transformation, the number of pictures contained in this subset is the same as And the number of its categories is the same as Consistent; ; Each subset after the h-th random affine transformation, the number of pictures contained in this subset is the same as And the number of its categories is the same as Consistent, and in this example, v = 50 is taken. ;

[0055] Step 2, select an existing deep image classification network, set the number of categories of the classification head, and construct two branches of the double transformation network.

[0056] Existing deep image classification networks include "Swin Transformer", "ResNet101", "YOLO-v5", etc. In this example, the "Swin Transformer" network is selected but not limited to it, as Figure 2 Shown.

[0057] The specific implementation of this step is as follows:

[0058] 2.1) Construct the first branch "R-S" of the double transformation network:

[0059] Take the key points on the reference image as categories, and on the branch from the reference image to the image to be registered, take the number of categories equal to the number of images contained in the reference image set Input these categories into the classifier of the depth image classification network to form the first branch "R-S" of the double transformation network:

[0060] This first branch "R-S" consists of a patch generation module, a linear mosaicking module, a first two-level cascaded window-internal self-attention mechanism module, a first patch fusion module, a second two-level cascaded window-internal self-attention mechanism module, a second patch fusion module, a six-level cascaded window-internal self-attention mechanism module, a third patch fusion module, a third two-level cascaded window-internal self-attention mechanism module, and an "R-S" branch classifier connected in cascade in sequence;

[0061] 2.2) Construct the first branch "R-S" of the double transformation network:

[0062] Take the key points on the image to be registered as categories, and on the branch from the image to be registered to the reference image, take the number of categories equal to the number of images contained in the image set to be registered Input these categories into the classifier of the depth image classification network to form the second branch "S-R" of the double transformation network;

[0063] This second branch "S-R" consists of a patch generation module, a linear mosaicking module, a first two-level cascaded window-internal self-attention mechanism module, a first patch fusion module, a second two-level cascaded window-internal self-attention mechanism module, a second patch fusion module, a six-level cascaded window-internal self-attention mechanism module, a third patch fusion module, a third two-level cascaded window-internal self-attention mechanism module, and an "S-R" branch classifier connected in cascade in sequence.

[0064] The parameter settings of each module of the above "R-S" branch and "S-R" branch can be adjusted according to specific situations. In this example, the preset values of the existing "Swin Transformer" network are used as the parameters of each module, but not limited to this.

[0065] Step 3, train the first branch "R-S" of the double transformation network.

[0066] The training methods include but are not limited to mini-batch gradient descent method, momentum method, Adam algorithm. In this example, the mini-batch gradient descent method is used for training, and the specific implementation is as follows:

[0067] 3.1) Take the reference image set as the training set, and take the reference image set after affine transformation as the validation set;

[0068] 3.2) Preprocess the sample images in the first branch training set and validation set, that is, and The images in are expanded from a size of 64×64 to a size of 224×224 using bicubic interpolation;

[0069] 3.3) Input the preprocessed images into the first branch "R-S" of the double transformation network in batches. After each batch is input, calculate the gradient value of its loss function, calculate the step size through the image blocks, and adjust the parameters of the first branch of the network according to the direction and step size of the gradient value;

[0070] 3.4) Repeat step 3.3) for several batches until After all image samples are trained once, execute step 3.5) for verification. To ensure the accuracy of the mini-batch gradient descent method, the batch size is generally not more than 128. In this example, the batch size is set to 64, but not limited to this;

[0071] 3.5) Verify the training effect of the first branch and calculate the accuracy Accurancy of the current image classification:

[0072]

[0073] where T Acc is the correctly classified image, and T total is the total number of images among them;

[0074] 3.6) Determine whether the accuracy Accurancy reaches the target accuracy threshold and above:

[0075] If it reaches, save the trained model to obtain the trained first branch of the double transformation network;

[0076] Otherwise, return to step 3.3) to continue training until the accuracy meets the standard,

[0077] The target accuracy threshold should be set to at least 90%. In this example, the target threshold is set to 92%, but not limited to this.

[0078] Step 4, train the second branch "S-R" of the double transformation network.

[0079] The training methods include but are not limited to the mini-batch gradient descent method, momentum method, and Adam algorithm. In this example, the mini-batch gradient descent method is used for training, and the specific implementation is as follows:

[0080] 4.1) Use the set of images to be registered as the training set, and use the set of reference images after affine transformation as the validation set;

[0081] 4.2) Preprocess the sample images in the training set and validation set of the second branch, that is, and the images in are expanded from a size of 64×64 to 224×224 using bicubic interpolation;

[0082] 4.3) Input the preprocessed images into the second branch "S-R" of the double transformation network batch by batch. After each batch is input, calculate the gradient value of its loss function, calculate the step size through the image patches, and then adjust the parameters of the second branch of the network according to the direction and step size of the gradient value;

[0083] 4.4) Repeat step 4.3) for several batches until all the image samples are trained once, then execute step 4.5) for verification. To ensure the accuracy of the mini-batch gradient descent method, the batch size is generally not more than 128. In this example, the batch size is set to 64, but not limited to this value;

[0084] 4.5) Verify the training effect of the second branch and calculate the accuracy Accurancy of the current image classification:

[0085]

[0086] where T Acc is the correctly classified image, and T total is the total number of images in;

[0087] 4.6) Determine whether the accuracy Accurancy reaches the target accuracy threshold and above:

[0088] If it reaches, save the trained model to obtain the trained second branch of the double transformation network,

[0089] otherwise, return to step 4.3) to continue training until the accuracy meets the standard.

[0090] Step 5, connect the trained first branch and the trained second branch in parallel to obtain the trained double transformation network "STDT".

[0091] Step 6, construct a test data set.

[0092] 6.1) Construct the test set of the first branch:

[0093] Use the initial transformation matrix from the coordinate system of the image to be registered to the coordinate system of the reference image in 1.1) to transform the key points on the image to be registered to the reference image, and delete the points that cannot be captured to generate a point set Then, with Taking the points in it as the center, a test image set of the first branch is made with a size of 64×64 They are respectively expressed as follows:

[0094]

[0095]

[0096] Among them, ρ e is the key point transformed from the e-th image to be registered to the reference image, is the image patch intercepted using this key point, 1 ≤ e ≤ c, and c is the number of points in the point set ;

[0097] 6.2) Construct the test set of the second branch:

[0098] Using the initial transformation matrix from the reference image coordinate system to the image coordinate system to be registered in 1.1) transform the key points on the reference image to the image to be registered, and delete the points that cannot be screenshot to generate a point set Then, taking the points in as the center, a test image set of the second branch is made with a size of 64×64 They are respectively expressed as follows:

[0099]

[0100]

[0101] Among them, ρ f is the key point transformed from the f-th reference image to the image to be registered, is the image patch intercepted using this key point, 1 ≤ f ≤ d, and d is the number of points in the point set ;

[0102] Step 7: According to the test data set, use the trained double transformation network for image classification and matching.

[0103] 7.1) Input the test set of the first branch into the first branch of the trained double transformation network to obtain the image classification result. If the test set image A is classified as category B, it means that the center point of image A and the point corresponding to category B on the reference image are a matching point pair. Do this for each image in to obtain the initial matching point pair set of the first branch;

[0104] 7.2) Use the point pairs in the initial matching point pair set of the first branch to intercept image patches on the reference image and the image to be registered respectively, and compare the similarity of the image patches. The methods for comparing similarity include the difference method, the cross-correlation coefficient method, and the normalized cross-correlation coefficient method. In this example, but not limited to, the difference method is selected. Delete the point pairs with similarity lower than the threshold to obtain the matching point pairs recognized by the first branch, and complete the preliminary registration of the R-S branch;

[0105] 7.3) Input the test set of the second branch into the second branch of the trained double transformation network to obtain the picture classification result. If the test set picture C is classified into category D, it means that the center point of picture C and the point corresponding to category D on the reference image are a matching point pair. Do this for each picture in to obtain the initial matching point pair set of the second branch;

[0106] 7.4) Use the point pairs in the initial matching point pair set of the second branch to intercept image patches on the image to be registered and the reference image respectively, and compare the similarity of the image patches. The methods for comparing similarity include the difference method, the cross-correlation coefficient method, and the normalized cross-correlation coefficient method. In this example, select but not limited to the difference method. Delete the point pairs with similarity lower than the threshold to obtain the matching point pairs recognized by the second branch, and complete the preliminary registration of the S-R branch.

[0107] Step 8, fine matching.

[0108] 8.1) In the test set of the first branch where preliminary registration is completed select a direction every 45° within the 360° range of each point, a total of 8 directions are obtained. For each direction, select 8 new points every two pixel points. Take these 8 new points as the center respectively, and intercept on the reference image according to the size of 64×64, and a total of 8 new image patches of the test set of the first branch with preliminary registration are obtained;

[0109] 8.2) Calculate the similarity between the 8 new image patches and the original matching image patch. The methods for calculating similarity include the difference method, the normalized cross-correlation information method, etc. In this example, but not limited to, the difference method is adopted;

[0110] 8.3) Update the matching point pairs of the first branch, that is, select the center point corresponding to the image patch with the largest similarity among the 8 image patches, and use this point to replace the point in the original R-S branch test set If the similarity of all points is not as good as the original one, then keep the status quo to obtain the fine matching result of the matching point pairs of the first branch;

[0111] 8.4) In the test set of the second branch where preliminary registration is completed Select a direction every 45° within the 360° range of each point in , a total of 8 directions are obtained. For each direction, select 8 new points at intervals of two pixel points. Respectively, with these 8 new points as the centers, take screenshots on the image to be registered according to the size of 64×64, and a total of 8 new image patches of the second branch test set for preliminary registration are obtained;

[0112] 8.5) Calculate the similarity between the 8 new image patches and the original matching image patch using the same method as in 8.2);

[0113] 8.6) Update the matching point pairs of the second branch, that is, select the center point corresponding to the image patch with the largest similarity among the 8 image patches, and use this point to replace the point in the original S-R branch test set If the similarity of all points is not as good as the original one, then keep the status quo to obtain the refined matching result of the matching point pairs of the second branch;

[0114] 8.7) Select the point pairs recognized by both branches in step 7 to obtain ρ final , which is as follows:

[0115] ρ final ={ρ1,...ρ x ...,ρ Z}

[0116] where ρ x is the Xth matching point pair recognized by both branches, 1≤X≤Z, where Z is the number of matching image pairs recognized by both branches.

[0117] 8.8) Update each ρ x respectively with the refined matching results of the first branch and the second branch, and calculate the similarity of the corresponding matching image patches after the update of the two branches using the difference method. Select the refined matching result of the branch with the greater similarity as the final registration result.

[0118] The effects of the present invention can be further illustrated by the following simulation experiments:

[0119] I. Simulation experiment conditions.

[0120] The dataset pictures used in this experiment are eight pictures captured by SAR under different time backgrounds and different imaging conditions. They can be pairwise registered, namely: Wuhan dataset, Yellow River R1 dataset, Yellow River R2 dataset, YAMABA dataset. As Figure 3 shown, where Figure 3 (a) is the Wuhan dataset, Figure 3 (b) is the Yellow River R1 dataset, Figure 3 (c) is the Yellow River R2 dataset,Figure 3 (d) is the YAMABA dataset. Each dataset includes a reference image and an image to be registered, with the reference image on the left and the image to be registered on the right.

[0121] This experiment is simulated through the deep learning library Pytorch. The input image size is 224×224. Smooth labels are used to prevent overfitting, with a smoothing rate of 0.1. The cosine similarity is used as the loss function, and the mini-batch gradient descent method is used for parameter optimization. The number of iterations is set to 100 rounds. Batch size = 64. After each Epoch, all samples in the validation set are used for validation, and the TOP-1 accuracy is calculated. When the lowest value reaches 0.92, the iteration ends.

[0122] The hardware devices and deep learning environment required for this experiment are: NIVIDA GTX 2080Ti, CUDA 11.0, Torch 1.7, Ubuntu 18.04. The remaining dependent packages can be installed according to the requirements of requirements.txt of the existing network "Swin Transformer".

[0123] II. Contents and Results of Simulation Experiments

[0124] Simulation Experiment 1: Under the above conditions, the four dataset images are registered using the present invention and the existing method "Multi-Scale Fused SAR Image Registration Based on Deep Forest", and their root mean square error R mse is compared, and the results are shown in Table 1.

[0125] Table 1 Comparison of the accuracy between the present invention and the prior art

[0126] Wuhan YAMABA YellowR1 YellowR2 <![CDATA[Existing method (R mse )]]> 0.4345 0.4645 0.5923 0.5051 <![CDATA[This invention (R mse )]]> 0.4490 0.4604 0.5487 0.4808

[0127] As can be seen from Table 1, except for the Wuhan dataset, the registration accuracy of the present invention is higher than that of the existing method on the other three datasets, and the present invention can also achieve good results on the Wuhan dataset.

[0128] Simulation Experiment 2: Under the above conditions, the present invention uses the R-S branch to compare the accuracy before and after fine matching on different datasets, and the results are shown in Table 2.

[0129] Table 2 Comparison of the root mean square error before and after fine matching on the R-S branch

[0130] Dataset <![CDATA[(R mse )Before fine matching]]> <![CDATA[(R mse )After fine matching]]> Wuhan 0.4598 0.4579 YABABA 0.4788 0.4690 YellowR1 0.5798 0.5525 YellowR2 0.5253 0.5185

[0131] As can be seen from the experimental results in Table 2, the present invention has achieved obvious effects on all four data sets through fine matching, and the errors after fine matching are all smaller than those without fine matching.

[0132] In summary, the method for SAR image registration based on deep learning of the present invention can efficiently complete the registration task for SAR images, effectively solve the problem of difficulty in obtaining matching image pairs in the prior art, save the cost of SAR image registration, and improve the accuracy of SAR image registration.

Claims

1. A deep learning SAR image registration method based on a dual transformation network, characterized in that, It includes the following steps: (1) Generate a dataset: 1a) Use the SIFT method to obtain m key points in the reference image R respectively and obtain n key points on the image S to be registered 1 ≤ i ≤ m, 1 ≤ j ≤ n, and perform a preliminary registration on these two to obtain the initial transformation matrix from the coordinate system of the image to be registered to the coordinate system of the reference image and the initial transformation matrix from the coordinate system of the reference image to the coordinate system of the image to be registered 1b) Respectively centered on and these key points, the reference image R and the image S to be registered are cropped according to the size of 64×64. The points that cannot be cropped are deleted, leaving a reference image points and b image points to be registered that can be cropped, obtaining the reference image set and the image set to be registered 1≤C≤a, 1≤D≤b; 1c) For and perform t - time affine transformation and grayscale transformation including 0° - 20° respectively. The is combined with its transformed result to obtain a reference image set The and its transformed result are combined to obtain a set of images to be registered Then, for and perform v - time affine transformation with the angle randomly in the range of 0° - 20° respectively. The is combined with its transformed result to obtain a sample set of reference image sets for verification The and its transformed result form a set of images to be registered for verification where 150 ≤ t ≤ 250; 40 ≤ v ≤ 100; (2) Select an existing deep image classification network, set the number of categories of the classification head, and construct two branches of the double transformation network: 2a) Take the key points on the reference image as categories, and on the branch from the reference image to the image to be registered, take the number of categories equal to the number of images contained in the reference image set and input these categories into the classifier of the depth image classification network to form the first branch "R-S" of the double transformation network; 2b) Use the key points on the image to be registered as categories, and on the branch from the image to be registered to the reference image, take the number of categories equal to the number of images contained in the set of images to be registered in the set, and input these categories into the classifier of the depth image classification network to form the second branch "S-R" of the double transformation network; (3) Train the two branches of the network to obtain the double transformation network: 3a) Use the set of reference images as the training set, and use the set of reference images after affine transformation as the validation set. Use the gradient descent method to train the first branch R-S. After each complete pass through all the samples in the training data set, use the validation set to perform a validation. When the accuracy reaches the target or above, save the trained model to obtain the trained first branch of the double transformation network; 3b) The set of images to be registered is used as the training set, and the set of images to be registered after affine transformation is used as the validation set. The second branch S-R is trained using the gradient descent method. After each pass through all the samples in the training dataset, the validation set is used for one validation. When the accuracy reaches the target or above, the trained model is saved to obtain the trained second branch of the double transformation network; 3c) Connect the trained first branch and the trained second branch in parallel to form the double transformation network; (4) Use the trained double transformation network for preliminary matching: 4a) Use the initial transformation matrix obtained in 1a) to transform from the coordinate system of the image to be registered to the coordinate system of the reference image Transform the key points on the image to be registered to the reference image, and delete the points that cannot be captured to generate a point set Take the points in as the center to capture and produce a test image set Input it into the first branch of the trained double transformation network for image classification. If it is classified as the same class, it means that the points on the image to be registered corresponding to the test set image and the points on the reference image as the class are matching points, and delete the points whose corresponding image block similarity is lower than the threshold to complete the preliminary registration of the R-S branch; 4b) Use the initial transformation matrix obtained in 1a) to transform from the reference image coordinate system to the image coordinate system to be registered Transform the key points on the reference image to the image to be registered, and delete the points that cannot be captured to generate a point set Take the points in as the center to capture and produce a test image set Input it into the second branch of the trained double transformation network for image classification. If it is classified as the same class, it means that the points on the reference image corresponding to the test set image and the points on the image to be registered as the class are matching points, and delete the points whose corresponding image patch similarity is lower than the threshold to complete the preliminary registration of the S-R branch; (5) Fine matching: 5a) Re-select points and take screenshots next to the training set points of the R-S branch after preliminary registration, update the preliminary registration obtained in 4a), and obtain the points on the R-S branch training set in the point pair set; 5b) Re-select points and take screenshots next to the training set points of the S-R branch after preliminary registration, update the preliminary registration obtained in 4b), and obtain the points on the S-R branch training set in the point pair set; 5c) Select the point pairs recognized by both branches and update them according to the updated matching results in 5a) and 5b) to obtain the final registration result.

2. The method according to claim 1, wherein: The initial transformation matrix obtained in step 1a) for transforming from the coordinate system of the image to be registered to the coordinate system of the reference image and the initial transformation matrix for transforming from the coordinate system of the reference image to the coordinate system of the image to be registered are respectively expressed as follows: Among them, a 11 、a 12 、a 13 、a 21 、a 22 、a 23 and b 11 、b 12 、b 13 、b 21 、b 22 、b 23 are all floating-point numbers, obtained by least squares calculation.

3. The method according to claim 1, wherein: The set of reference images obtained in step 1b) and the set of images to be registered are respectively represented as follows: Among them, is an image patch intercepted from the i-th reference image, where 1 ≤ i ≤ a, is an image patch intercepted from the j-th image to be registered, where 1 ≤ j ≤ b. These reference images and images to be registered are all pictures with a size of 64×64.

4. The method according to claim 1, characterized in that: Those obtained in step 1c) and as well as and are respectively represented as follows: Among them, each subset after the f-th random affine transformation, the number of pictures contained in this subset is exactly the same as and the number of categories is the same as consistent; after the e-th random affine transformation each subset, the number of pictures contained in this subset is the same as exactly the same, and the number of its categories is the same as consistent; after the l-th random affine transformation For each subset, the number of images contained in the subset is exactly the same as and the number of categories is the same as consistent; after the h-th random affine transformation For each subset, the number of images contained in the subset is the same as exactly the same, and the number of categories is the same as identical.

5. The method according to claim 1, wherein: In step 2a), the first branch "R-S" of the network is obtained, which consists of a patch generation module, a linear mosaic module, a first two-level cascaded window self-attention mechanism module, a first patch fusion module, a second two-level cascaded window self-attention mechanism module, a second patch fusion module, a six-level cascaded window self-attention mechanism module, a third patch fusion module, a third two-level cascaded window self-attention mechanism module, and an "R-S" branch classifier in cascade. The number of classes of this classifier is the same as the number of pictures contained in the picture set obtained in 1b), and the parameter settings of each module other than this classifier are the same as the preset values of the existing "Swin Transformer" network. The number of pictures contained in the picture set obtained in 1b), and the parameter settings of each module other than this classifier are the same as the preset values of the existing "Swin Transformer" network.

6. The method according to claim 1, characterized in that: In step 2b), the second branch "S-R" of the network is obtained, which consists of a patch generation module, a linear mosaicking module, a first two-level cascaded window self-attention mechanism module, a first patch fusion module, a second two-level cascaded window self-attention mechanism module, a second patch fusion module, a six-level cascaded window self-attention mechanism module, a third patch fusion module, a third two-level cascaded window self-attention mechanism module, and an "S-R" branch classifier connected in cascade in sequence. The number of categories of this classifier is the same as the number of images contained in the image set obtained in 1b), and the parameter settings of each module except this classifier are the same as the preset values of the existing "Swin Transformer" network. The number of images contained in the image set obtained in 1b), and the parameter settings of each module except this classifier are the same as the preset values of the existing "Swin Transformer" network.

7. The method according to claim 1, wherein: Those obtained in step 4a) are respectively represented as follows: Among them, ρ e is the key point transformed from the e-th image to be registered onto the reference image, is the image patch intercepted with a size of 64x64 using this key point, where 1 ≤ e ≤ c and c is the number of points in the point set ; The test set of the S-R branch is obtained in step 4b). They are respectively represented as follows: Among them, ρ f is the key point transformed onto the image to be registered on the f-th reference image, is the image patch intercepted in the size of 64x64 using this key point, where 1 ≤ f ≤ d and d is the number of points in the point set of points.

8. The method according to claim 1, characterized in that: In step 3a), the gradient descent method is used to train the first branch R-S, and after each complete pass through all the samples of the training data set, its validation set is used for one validation, as follows: 3a1) Preprocess the sample images in the training set of the R-S branch, that is, The images in are expanded from a size of 64×64 to 224×224 using bicubic interpolation; 3a2) Preprocess the sample images in the validation set of the R-S branch, that is, the images in it are expanded from a size of 64×64 to 224×224 using bicubic interpolation; 3a3) Set the batch size of the training pictures input into the network each time to 64, input the 224×224-sized image into the first branch "R-S" of the double transformation network, calculate the gradient value of its loss function, and use the mini-batch gradient descent method to update the weights of the first branch R-S; 3a4) After training all the images in once, use all the image samples in the validation set to perform a validation once and calculate its accuracy Accurancy: where T Acc is the correctly classified image, and T total is the total number of images therein. 3a5) Record the Accurancy value obtained each time during verification. When the Accurancy value exceeds 92%, save the network model as the trained first branch "R-S" of the double transformation network.

9. The method according to claim 1, wherein In 3b), the gradient descent method is used to train the second branch S-R, and after each complete training of all samples in the training dataset, the validation set is used for validation once, and the implementation is as follows: 3b1) Preprocess the sample images in the training set of the S-R branch, that is, The images in are expanded from a size of 64×64 to a size of 224×224 using bicubic interpolation; 3b2) Preprocess the sample images in the validation set of the S-R branch, that is, The images in are expanded from a size of 64×64 to a size of 224×224 using bicubic interpolation; 3b3) Set the batch size of the training pictures input into the network each time to 64, input the 224×224-sized image into the second branch "S-R" of the double transformation network, calculate the gradient value of its loss function, and use the mini-batch gradient descent method to update the weights of the second branch S-R; 3b4) After training all the images among them, use the validation set to perform a validation on all the image samples among them and calculate the accuracy Accurancy: Among them, T Acc is the correctly classified picture, while T total is the total number of pictures therein; 3b5) Record the Accurancy value obtained each time during verification. When the Accurancy value exceeds 92%, save the network model as the trained second branch "S-R" of the double transformation network.

10. The method according to claim 1, characterized in that: The implementation of step 5a) and step 5b) is as follows: In step 5a), re-select points and take screenshots on the image to be registered of the R-S branch after preliminary registration to update the point set: 5a1) In the R-S branch test set select one direction every 45° within the 360° range of each point, a total of 8 directions are obtained, and select 8 new points every two pixel distances in each direction; 5a2) Respectively take screenshots on the reference image in the size of 64×64 with the eight new points obtained in 5a1) as the centers, a total of eight new image patches are obtained, and the similarity is calculated by taking the difference between the eight new images and the image patches on the image to be registered corresponding to the original category; 5a3) Select the center point corresponding to the image patch with the maximum similarity, and use this point to replace the point in the original R-S branch test set and update the matching point pairs. If the similarity of all points is not as good as the original one, keep the status quo. In step 5b), re-select points and take screenshots on the reference image of the S-R branch after preliminary registration to update the point set, and its implementation steps are the same as those of 5a).

Images