Optical and sar image registration method based on similar edge feature extraction and ofast-brisk algorithm

Abstract

The application discloses an optical and SAR image registration method based on similar edge feature extraction and OFAST-BRISK algorithm, and mainly applies to the technical field of satellite remote sensing image processing and application.The application aims to solve the problems of low accuracy and poor stability of existing optical and SAR image registration algorithms.The application combines the advantages of deep learning algorithm and traditional feature matching algorithm.The improved Log operator is used in combination with the Deeplabv3+ network to realize the similar edge feature extraction of optical and SAR images, and the OFAST-BRISK algorithm is used in combination with the improved RANSAC algorithm to realize the extraction and matching of feature points.The optical and SAR image registration method based on similar edge feature extraction and OFAST-BRISK algorithm can realize the efficient and accurate registration of optical and SAR images with 10m resolution.

Description

Technical Field

[0001] This invention belongs to the technical field of satellite remote sensing image processing and application. Background Technology

[0002] Optical and synthetic aperture radar (SAR) sensors are widely used in multimodal remote sensing. Optical sensors can acquire rich texture and spectral information of ground features, possessing good ground feature identification and classification capabilities, but are easily affected by atmospheric attenuation and weather conditions. In contrast, SAR sensors have strong penetrating power and can acquire backscattered signals of ground features regardless of weather conditions. Since optical and SAR images each have their advantages and can capture different features of the same target, high-precision registration of optical and SAR images is crucial in multimodal remote sensing applications. However, although optical and SAR images of the same scene can be obtained, significant geometric and nonlinear radiometric differences exist between them due to their different imaging mechanisms, making multimodal image registration of optical and SAR images a challenging task.

[0003] Existing image registration techniques can be divided into two main categories: region-based registration methods and feature-based registration methods (see Zeng L, Du Y, Lin H, Wang J, Yin J and Yang J. 2021. A novel region-based image registration method for multisource remote sensing images via CNN. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14:1821-1831). The former method mainly relies on pre-established similarity metrics and transformation models. It calculates image similarity based on shallow information such as image grayscale or phase, and then adjusts the transformation model parameters through optimization strategies to achieve an ideal similarity threshold, ultimately realizing high-precision image alignment. These methods have a certain degree of robustness, but their registration efficiency is low and time-consuming. The latter method first extracts and matches geometric features, then uses the spatial correspondence of geometric features to solve for the model transformation parameters, achieving image registration. The advantage of these traditional algorithms is that feature point extraction is accurate and stable, enabling efficient registration. However, these traditional registration algorithms were originally designed for optical images, and they cannot achieve satisfactory results when there are differences in the acquisition and spatial resolution of multimodal images. To address this, several improved registration algorithms have been proposed for optical and SAR images, such as the optical-SAR image registration algorithm based on the improved SIFT algorithm (OS-SIFT) (see Xiang, Y; Wang, F; You, H OS-SIFT: A Robust SIFT-like Algorithm for High-Resolution Optical to SAR Image Registration in Suburban Areas[J]. EEE Transactions on Geoscience and Remote Sensing. 2018, 56(6), 3078--3090.) and the Oriented Phase Consistency Histogram (HOPC) algorithm (see Rahmani H, Mahmood A, Huynh D, et al. Histogram of Oriented Principal Components for Cross-View Action Recognition[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2016: 2430-2443.).Although these improved algorithms can achieve good image registration results, they still only utilize the low-level features of the image and cannot automatically adjust to improve accuracy, which is a limitation of traditional algorithms.

[0004] With the rapid development of deep neural network technology, deep learning-based registration methods can learn from a large number of samples, extract deep semantic information from images, and construct complex spatial mapping relationships between images, thus achieving high efficiency in remote sensing image registration. Deep learning-based registration algorithms are divided into region-based registration and feature-based registration. In region-based registration methods, the construction of a robust similarity measure is crucial. Yang et al. used a DNN network for image similarity estimation, learning an image similarity measure and the relationship between image patches and deformation parameters to achieve image registration (see Yang X, Kwitt R, Styner M, et al. Fast predictive multimodal image registration: IEEE, 10.1109 / ISBI.2017.7950652[P].2017.). Li et al. also embedded CNN networks into the traditional region-based registration framework to improve the robustness of the conversion model (see Li H, Fan Y. Non-Rigid Image Registration Using Self-Supervised Fully Convolutional Networks without Training Data: IEEE, 10.1109 / ISBI.2018.8363757[P].2018.). Furthermore, GAN networks can also be used to reduce the differences between optical and SAR images, construct similarity in spectral information, and reduce the difficulty of registration, such as in Deep learning with improved Harris (DIH) (see Zhang, W. Robust registration of SAR and optical images based on deep learning and improved Harris algorithm. Sci Rep 12, 5901(2022).). Feature-based registration methods in deep learning generally include the following two strategies: one is feature point extraction based on Score Map. Typically, a CNN output Score Map is used, and then the points with the maximum local response values ​​of the Score Map are extracted as feature points. A representative method is Description and Detection Net (D2-NET) (see Dusmanu M, Rocco I, Pajdla T, et al. D2-Net: A Trainable CNN for Joint Detection and Description of Local Features[J].arXiv,2019).Second, based on fuzzy matching feature point extraction, this method treats feature extraction and matching as a whole, simulating traditional similarity estimation methods, and proposes a matching strategy based on an association layer. When applying this method, a Siamese neural network is first used to generate feature maps of two images, and then the similarity of each point in the feature maps is calculated to obtain matching point pairs (see Rocco I, R). SivicJ. Convolutional Neural Network Architecture for Geometric Matching[J]. PatternAnalysis and Machine Intelligence, IEEE Transactions on, 2018.).

[0005] In summary, traditional feature-based registration algorithms are limited by fixed feature extractors, working only with specific data and unable to dynamically adjust based on different input data. Deep learning-based registration methods can extract deep semantic information from images and learn complex spatial mapping relationships between images. Furthermore, because deep learning can provide negative feedback, network parameters can be adjusted according to different input data, resulting in a wider range of applications than traditional algorithms. While deep learning algorithms improve the robustness and practicality of registration to some extent, they remain susceptible to image noise and low spatial resolution. In addition, a sufficient number of high-quality training and annotated datasets are required. Summary of the Invention

[0006] To address the issues of low registration accuracy and poor robustness in existing optical image-synthetic aperture radar (SAR) image registration algorithms, this invention combines the advantages of deep learning algorithms and traditional feature algorithms. It proposes a method for optical and SAR image registration based on similar edge feature extraction and the OFAST-BRISK algorithm, which can accurately and efficiently achieve registration between optical and SAR images.

[0007] The specific steps of the technical solution adopted in this invention are as follows:

[0008] Step 1: Select the optical and SAR image pair dataset as the training and validation data for the deep learning network, and randomly select six pairs of images from the dataset to perform translation and rotation transformations to simulate real optical and SAR image registration scenarios.

[0009] Step 2: Replace the Gaussian filter in the original Log detection algorithm with NLM filter to obtain an improved Log edge detection algorithm. Extract the edges of SAR images in the dataset and use the edge extraction results as shared edge label data for the deep learning network.

[0010] Step 3: The optical image and shared edge label data, and the SAR image and shared edge label data are respectively fed into the deep learning network for training. After training, the deep learning network can extract the similarity edge features between the optical and SAR images. The deep learning network is an improved Deeplabv3+ deep learning network, in which Inception-ResNet-v2 is used as the backbone network of the encoder.

[0011] Step 4: Based on the extraction of similarity edge features between optical and SAR images, the proposed OFAST-BRISK algorithm is used to extract and describe feature points in the optical and SAR images.

[0012] Step 5: Propose an improved RANSAC algorithm to match feature points and complete image registration.

[0013] The specific process of edge extraction for SAR images in the dataset in step two is as follows:

[0014] 1) The filtered image obtained by using an NLM filter is I(i), which can also be represented by I(x,y). The specific steps are as follows: Assuming a discrete noisy image is f(x,y), it can also be expressed as: f={f(i)|i∈Ω}, where Ω represents the image neighborhood and i is the pixel index. After NLM filtering of the noisy image, it can be normalized as:

[0015]

[0016] Ω i Let represent a search window with center pixel i and size q×q; ω(i,j) represents the weights assigned to the noisy image, which can be expressed as:

[0017]

[0018]

[0019]

[0020] Where h represents the filter coefficient that controls the smoothness of the filter, which is usually the standard deviation of the noise. N(i) and N(j) represent image blocks centered at pixel i and pixel j with a window size of p, respectively; d(i,j) represents the Euclidean weighted distance between N(i) and N(j). G represents the convolution between two matrices; α It is a Gaussian matrix.

[0021] 2) The second-order directional derivative of the filtered image I(x,y) is processed using the Laplacian operator, and the resulting image M(x,y) is the edge extraction result.

[0022]

[0023] Edge extraction was performed on SAR images selected from the dataset, which were then used as label data for the deep learning network.

[0024] Step four of the OFAST-BRISK algorithm can be divided into two parts:

[0025] 1) The OFAST algorithm is used to extract feature points from optical and SAR edge images. The coordinates of the feature points and the centroid form a vector as the direction of the feature points, and the overall direction θ of the feature points is determined.

[0026] The FAST feature point extraction algorithm is widely used due to its high computational speed. However, the FAST algorithm did not consider directional characteristics in its design. Therefore, the Directed FAST (OFAST) algorithm was later proposed, which uses the gray centroid method to obtain the rotational neighborhood of the feature point orientation. This algorithm first uses the FAST method to detect feature points, then uses a spatial pyramid to generate multi-scale features, and finally uses a dense centroid algorithm to give the overall orientation of the feature points. Assuming that each feature point has a centroid with radius r, the coordinates of the feature point and the centroid form a vector, which serves as the orientation of the feature point. The moment is defined as follows:

[0027]

[0028] Where I(x,y) is the gray value at position (x,y) in the image.

[0029] Therefore, these moments are used to determine the centroid of the local neighborhood of the image:

[0030]

[0031] Where m 00 m represents the sum of gray levels of all pixels in the image's neighborhood window. 10 and m 01 These are the first moments with respect to x and y, respectively. The coordinates of the feature point and the centroid form a vector that serves as the direction of the feature point, determining the overall direction of the feature point as follows:

[0032] θ = arctan(m) 01 ,m 10 (8)

[0033] Through the above operations, OFAST feature points exhibit robust scale invariance and rotation invariance, significantly improving the detection performance of feature points.

[0034] 2) The BRISK algorithm is used to extract binary vector descriptors;

[0035] The main direction θ determined by the OFAST algorithm is used as the main direction of the feature descriptor extracted by the BRISK algorithm. Before the feature point description, the sampling pattern is rotated according to the overall direction θ determined by the feature points to ensure the rotation invariance of the feature descriptor.

[0036] Based on the scale and orientation of the feature points, the gray values ​​of all two points in short-distance pairs of the feature point sampling points are compared. This comparison yields a 512-bit binary vector descriptor, where each bit b of the binary vector descriptor corresponds to:

[0037]

[0038] This represents the sampling point pair (p) after rotating each point on the sampling template by an angle θ. i ,p j The smoothing intensity value after Gaussian filtering with a standard deviation of σ.

[0039] A subset S of short-distance point pairs:

[0040]

[0041] The set of all sampling point pairs Ω:

[0042] Ω={(p i ,p j )∈R 2 ×R 2 ∣i<N, j<i, (i,j)∈N} (11)

[0043] The threshold distance is set to δ max = 9.75t, where t is the scale of the feature point.

[0044] The specific steps of the improved RANSAC algorithm in step five are as follows:

[0045] (1) Coarse matching

[0046] O = {O1,...,O} n S = {S1,...,S} n Feature points are extracted from optical and SAR images, respectively. First, the Euclidean distance between all points in set O and all points in set S is calculated, and all matching point pairs are sorted in ascending order of Euclidean distance. Four pairs of points at the same location are randomly selected from the two sets. These eight feature points can form a homology matrix, resulting in:

[0047]

[0048] Where (x,y) and (x′,y′) are the remaining corresponding feature points in sets O and S, respectively, and Z is the scale parameter. Then, the projection error of all feature-matching point pairs in the dataset is calculated. If the projection error of the data is less than the set threshold, the feature points are placed into sets O' and S', respectively. The error calculation formula is as follows:

[0049]

[0050] After repeatedly and randomly selecting four pairs of feature points, the case with the most feature point pairs is retained in a set, denoted as set O'. max and S' max Let be the initial matching set. Although the RANSAC algorithm filters out some mismatches in the original set, some mismatches still remain due to the significant differences between multimodal images. Therefore, it is necessary to further remove obviously erroneous correspondences in the initial matching set to find more correct matching points.

[0051] (2) Precise matching

[0052] Based on prior knowledge, the slopes of the lines between correctly matched feature points are the same. Therefore, feature points are accurately matched based on the consistency of the slopes. Thus, O' max and S' max The slope parameter k between two candidate points in the set is calculated as follows:

[0053]

[0054] Then O' max and S' max All slope values ​​k i All of these can be calculated, k mean It is the average of all slope values. Based on this, select the interval [k]. mean -0.5,k mean Point pairs with slopes within [+0.5] are used to form a refined matching set. Finally, based on these correct correspondences, image registration is completed through affine transformation.

[0055] The beneficial effects of this invention are:

[0056] This invention combines the advantages of deep learning algorithms and traditional feature matching algorithms. Through training with an improved deep learning algorithm, Deeplabv3+, the network can extract similarity information between optical and SAR images. Based on similarity edge features, the OFAST-BRISK registration algorithm is used to extract image feature points, which are then combined with an improved RANSAC algorithm to achieve registration between optical and SAR images. This invention proposes an improved Log operator, an improved Deeplabv3+ network, an OFAST-BRISK registration algorithm, and an improved RANSAC algorithm to further improve the accuracy of image registration. This invention can efficiently and accurately achieve registration between optical and SAR images, with the registration error controlled within one pixel, providing important technical support for the collaborative application of optical and SAR images. Attached Figure Description

[0057] Figure 1 This is the SEN1-2 dataset used in this invention;

[0058] Figure 2 This is an overall flowchart of the present invention;

[0059] Figure 3 These are comparison images before and after NLM filtering; where (a) is the original image and (b) is the filtered image.

[0060] Figure 4 These are the effect diagrams of the improved Log operator used in this invention; where (a) is a SAR image and (b) is the result of edge extraction by the Log operator;

[0061] Figure 5 This is a schematic diagram of the improved Deeplabv3+ network structure used in this invention;

[0062] Figure 6 This is a schematic diagram of the improved Inception-Resnet-v2 network structure used in this invention;

[0063] Figure 7 These are the similar edge extraction results using the improved Deeplabv3+ network used in this invention; (a) shows the SAR image and the optical image, and (b) shows the edge extraction results.

[0064] Figure 8 This is the sampling template used in the BRISK algorithm of this invention;

[0065] Figure 9 This is the optical and SAR image registration result of the translation transformation of the present invention;

[0066] Figure 10 This is a chessboard diagram of the registration result of this invention;

[0067] Figure 11 This is the optical and SAR image registration result of translation and slight rotation transformations according to the present invention; Detailed Implementation

[0068] The technical solution of the present invention will be further explained and described below with reference to specific embodiments.

[0069] Example 1:

[0070] Most registration studies choose high-resolution optical images and meter- or sub-meter-resolution SAR images for registration. The reason for choosing these images is their high resolution; optical and SAR images contain more texture features and ground cover information, making them easier to register with remote sensing images. However, such high-resolution data is not readily available and requires complex preprocessing, resulting in a lack of suitable large-scale high-resolution registration datasets. Therefore, this paper selects images such as... Figure 1 The optical and SAR images shown are used as experimental data. The data pairs are from the Sen1-2 dataset, with a size of 256×256 pixels and a resolution of 10 meters. It contains features such as cities, rivers, and farmland, with clear characteristics and strong representativeness. The Sen1-2 dataset consists of Sen-1 and Sen-2 images. This dataset includes 282,384 optical and SAR image pairs from all regions of the world.

[0071] Since there is a slight translation between optical and SAR images in most practical applications, the SEN1 SAR images in the dataset are randomly translated to verify the performance of the proposed method in real-world multimodal image registration scenarios. Furthermore, the SAR images are slightly rotated to verify the rotational image registration performance of the proposed algorithm. See the flowchart for details. Figure 2 .

[0072] Step 1: Dataset preprocessing;

[0073] Six pairs of optical and SAR images were randomly selected. The optical image was used as the reference image, and the SAR image was used as the image to be registered. For the SAR image, a translation transformation of approximately 15 pixels in any direction was performed to simulate optical and SAR images in real-world applications. In addition, based on the translation transformation, the SAR image was rotated approximately 10° in any direction to test the performance of the algorithm for rotating image matching.

[0074] Step 2: Replace the Gaussian filter in the original Log detection algorithm with NLM filter to obtain an improved Log edge detection algorithm. Extract the edges of SAR images in the dataset and use the edge extraction results as shared edge label data for the deep learning network.

[0075] The filtered image obtained by using an NLM filter is I(i), which can also be represented by I(x,y). The specific steps are as follows: Assuming a discrete noisy image is f(x,y), it can also be expressed as: f={f(i)|i∈Ω}, where Ω represents the image neighborhood and i is the pixel index. After NLM filtering of the noisy image, it can be normalized as:

[0076]

[0077] Ω i Let represent a search window with center pixel i and size q×q; ω(i,j) represents the weights assigned to the noisy image, which can be expressed as:

[0078]

[0079]

[0080]

[0081] Where h represents the filter coefficient that controls the smoothness of the filter, which is usually the standard deviation of the noise. N(i) and N(j) represent image blocks centered at pixel i and pixel j with a window size of p, respectively; d(i,j) represents the Euclidean weighted distance between N(i) and N(j). G represents the convolution between two matrices; α Let I be a Gaussian matrix. Using a Gaussian convolution kernel is similar to first filtering noise using Gaussian filtering. Its advantage is that it reduces the impact of noise on the similarity calculation between two image patches, thus making the subsequent calculation results more accurate. The resulting filtered image is I(i), which can also be represented by I(x,y).

[0082] Then, the second-order directional derivative of the filtered image I(x,y) is processed using the Laplacian operator, and the resulting image M(x,y) is the edge extraction result.

[0083]

[0084] Edge extraction was performed on SAR images selected from the dataset, which were then used as label data for the deep learning network.

[0085] In step two, an edge extraction algorithm is used to extract edge features from the SAR images in the dataset to construct edge label data for the deep learning network. However, due to the large amount of noise in SAR images, existing edge extraction algorithms are affected by the noise. Therefore, NLM filtering is used to replace the Gaussian filtering in the original Log detection algorithm. Compared with Gaussian filtering, NLM filtering can more effectively remove noise that affects edge extraction.

[0086] In this embodiment, the SAR image is first fed into an NLM filter to remove noise. The NLM filter search window size q is set to 25×25, the window size p is set to 5×5, and the smoothing factor h is set to 25. After filtering, the image is obtained... Figure 3 The results are shown. Edge extraction is performed on the filtered image using the Laplacian operator, yielding... Figure 4 The results are shown.

[0087] Step 3: Improved Deeplabv3+ network training:

[0088] DeepLabv3+ employs the Atrous Spatial Pyramid Pooling (ASPP) model, applying multiple Atrous convolutions with varying dilation rates in parallel to fully utilize multi-scale information. DeepLabv3+ proposes a novel encoder-decoder architecture, using a CNN as the backbone encoder and adding a simple and efficient decoder. In the original DeepLabv3+ network, ResNet, VGGNet, and MobileNet are frequently chosen as the backbone networks for feature extraction. However, when using these backbone networks for edge extraction in optical and SAR images, the network training accuracy is low, and the segmentation results do not achieve ideal results. The main reasons affecting segmentation accuracy are that the area of ​​the edges to be extracted accounts for a small proportion of the total image area, and noise in SAR images affects network training. To extract features accurately and efficiently, the depth of the backbone network can be increased; however, with increasing network depth, problems such as vanishing gradients and overfitting occur.

[0089] Inception-ResNet-v2 effectively combines the advantages of both ResNet and Inception architectures by deepening the network hierarchy and reducing noise interference. The Inception structure, composed of multiple convolutional and pooling layers, can obtain sparse or non-sparse features on the same layer. The ResNet structure effectively summarizes features from higher layers to lower layers, accelerating training and preventing gradient vanishing. Therefore, using Inception-ResNet-v2 as the backbone network of Deeplabv3+ avoids the gradient vanishing problem. The network structure and improvements of Deeplabv3+ in this invention are as follows:

[0090] (1) Encoder structure:

[0091] The Inception-ResNet-v2 backbone network is used to extract high-level and low-level semantic features from the input image. Low-level semantic features are directly sent to the decoder for further processing, while high-level semantic features are input into an Atrous Spatial Pyramid Pooling (ASPP) model. The ASPP model consists of one spatial pyramid pooling layer, three Atrous convolutional layers operating in parallel with different dilation rates, and a 1×1 convolutional layer. After a series of convolution and pooling operations in the ASPP model, the high-level semantic features of the image are generated into a feature map of depth 5, which is then sent to the decoder through a 1×1 convolutional layer.

[0092] (2) Decoder structure;

[0093] For low-level semantic features from the Inception-ResNet-v2 backbone, a 1×1 convolutional layer is first used in the decoder to reduce the number of channels. For high-level semantic features output by the ASPP model, they are first bilinearly upsampled and then concatenated with the corresponding low-level features in the Inception-ResNet-v2 backbone. After concatenation, the features are refined using a 3×3 convolutional layer, followed by simple bilinear upsampling to output the final segmentation result.

[0094] In this embodiment, the Deeplabv3+ network is used as the optical and SAR image similarity feature extraction network. The network diagram is shown below. Figure 5 The Deeplabv3+ network mainly consists of two structures: an encoder and a decoder. In the ASPP structure, the sampling rates of the 3×3 dilated convolutions are set to 6, 12, and 18, respectively. In the decoder, the upsampling rate is set to 4 for all components. The Inception-ResNet-v2 network is used as the backbone network. The network diagram is shown below. Figure 6 The network mainly consists of a Stem Block structure, 5 Inception ResNet A structures, 10 Inception ResNet B structures, 5 Inception ResNet C structures, and Reduction A and Reduction B structures. 10,000 pairs of images from the SEN1-2 dataset were used as training data, and 2,000 pairs were used as validation data. Label data was obtained from the corresponding SAR images using an improved Log operator edge extraction. The batch size was set to 2, the initial learning rate was set to 0.0001, and the training lasted for 30 epochs. The trained Deeplabv3+ network can generate edge contour images that show similarity between optical and SAR images, such as... Figure 7 As shown.

[0095] Step 4: OFAST-BRISK Feature Point Extraction

[0096] 1) The OFAST algorithm is used to extract feature points from optical and SAR edge images. The coordinates of the feature points and the centroid form a vector as the direction of the feature points, and the overall direction θ of the feature points is determined.

[0097] The FAST feature point extraction algorithm is widely used due to its high computational speed. However, the FAST algorithm did not consider directional characteristics in its design. Therefore, the Directed FAST (OFAST) algorithm was later proposed, which uses the gray centroid method to obtain the rotational neighborhood of the feature point orientation. This algorithm first uses the FAST method to detect feature points, then uses a spatial pyramid to generate multi-scale features, and finally uses a dense centroid algorithm to give the overall orientation of the feature points. Assuming that each feature point has a centroid with radius r, the coordinates of the feature point and the centroid form a vector, which serves as the orientation of the feature point. The moment is defined as follows:

[0098]

[0099] Where I(x,y) is the gray value at position (x,y) in the image.

[0100] Therefore, these moments are used to determine the centroid of the local neighborhood of the image:

[0101]

[0102] Where m 00 m represents the sum of gray levels of all pixels in the image's neighborhood window. 10 and m 01 These are the first moments with respect to x and y, respectively. The coordinates of the feature point and the centroid form a vector that serves as the direction of the feature point, determining the overall direction of the feature point as follows:

[0103] θ = arctan(m) 01 ,m 10 (8)

[0104] Through the above operations, OFAST feature points exhibit robust scale invariance and rotation invariance, significantly improving the detection performance of feature points.

[0105] In this embodiment, the FAST algorithm is first used to determine feature points. The main method is to define a circle with a radius of 3 pixels centered at any point P. If 12 consecutive points on the circumference have gray values ​​greater or less than the gray value of point P, then P is identified as a feature point. Next, a spatial pyramid is constructed to ensure the scale invariance of the feature points. The spatial pyramid has 8 layers, and each layer is scaled at a ratio of 1.2. Finally, the principal direction of the feature points is determined based on moments.

[0106] 2) Use the BRISK algorithm to obtain the feature descriptors of the feature points:

[0107] The BRISK descriptor uses a defined sampling pattern to generate uniform sampling points within a given radius around the feature point. A Gaussian filter is applied to the surrounding uniform sampling points, centered on the feature point. I(p,σ) represents the grayscale value of the uniform sampling point p after Gaussian filtering with a standard deviation of σ. Assuming there are N sampling points, there are N·(N-1) / 2 pairs of sampling points (p...). i ,p j The smoothing intensity values ​​at these points are I(p) i ,σ i ) and I(p j ,σ j ), local gradient g(p) i ,p j It can be estimated in the following ways:

[0108]

[0109] Consider the set Ω of all sample point pairs:

[0110] Ω={(p i ,p j )∈R 2 ×R 2 ∣i<N, j<i, (i,j)∈N} (10)

[0111] Define a subset of short-range pairs as follows:

[0112]

[0113] The threshold distance is set to δ max = 9.75t, where t is the scale of the feature point.

[0114] In order to better combine the OFAST algorithm and the BRISK algorithm, the main direction θ determined by the OFAST algorithm in formula (8) is used as the main direction of the feature point.

[0115] Before feature point description, the sampling template (such as...) Figure 8 As shown, the feature descriptor is rotated according to the overall direction θ determined by the feature points to ensure rotation invariance. Based on the scale and orientation of the feature points, the grayscale values ​​of all two points in short-distance pairs are compared. This comparison yields a 512-bit binary vector descriptor, with each bit b corresponding to:

[0116]

[0117] This represents the grayscale value of each point on the sampling template after rotation by an angle θ.

[0118] In this embodiment, the number of sampling points N is set to 60, the feature point scale t is set to 1, and finally a 512-bit feature descriptor is obtained.

[0119] Step 5: Feature point matching using the improved RANSAC algorithm

[0120] This invention proposes an improved RANSAC algorithm, which enhances the matching accuracy of feature points. The improved RANSAC algorithm performs two matching operations on feature points: 1) A coarse matching is performed using the Random Sample Consensus (RANSAC) algorithm. However, due to the limitations of the RANSAC algorithm, the correctness of the obtained matching points cannot be guaranteed. 2) A model corresponding to the feature points is rebuilt, and the slope between corresponding points is used as the criterion for a second, precise matching. The improved RANSAC algorithm is described as follows:

[0121] (1) Coarse matching

[0122] O = {O1,...,O} n S = {S1,...,S} n Feature points are extracted from optical and SAR images, respectively. First, the Euclidean distance between all points in set O and all points in set S is calculated, and all matching point pairs are sorted in ascending order of Euclidean distance. Four pairs of points at the same location are randomly selected from the two sets. These eight feature points can form a homology matrix, resulting in:

[0123]

[0124] Where (x,y) and (x′,y′) are the remaining corresponding feature points in sets O and S, respectively, and Z is the scale parameter. Then, the projection error of all feature-matching point pairs in the dataset is calculated. If the projection error of the data is less than the set threshold, the feature points are placed into sets O' and S', respectively. The error calculation formula is as follows:

[0125]

[0126] After repeatedly and randomly selecting four pairs of feature points, the case with the most feature point pairs is retained in a set, denoted as set O'. max and S' max Let be the initial matching set. Although the RANSAC algorithm filters out some mismatches in the original set, some mismatches still remain due to the significant differences between multimodal images. Therefore, it is necessary to further remove obviously erroneous correspondences in the initial matching set to find more correct matching points.

[0127] (2) Precise matching

[0128] Based on prior knowledge, the slopes of the lines between correctly matched feature points are the same. Therefore, feature points are accurately matched based on the consistency of the slopes. Thus, O' max and S' max The slope parameter k between two candidate points in the set is calculated as follows:

[0129]

[0130] Then O' max and S' max All slope values ​​k i All of these can be calculated, k mean It is the average of all slope values. Based on this, select the interval [k]. mean -0.5,k mean Point pairs with slopes within [+0.5] are used to form a refined matching set. Finally, based on these correct correspondences, image registration is completed through affine transformation.

[0131] To address the issue of feature point matching errors in the RANSAC algorithm, this experiment proposes an improvement. First, feature points are sorted based on their normal distance. Then, the RANSAC algorithm is used for coarse matching, with 100 iterations and a projection error threshold of 0.01. Next, precise matching is performed based on the principle that correctly matched points have the same slope, eliminating incorrectly matched points.

[0132] This invention uses the following metrics for image registration evaluation:

[0133] (1) Matching accuracy: A metric used to measure the accuracy of a registration algorithm. It is usually described by the root mean square error (RMSE). RMSE represents the root mean square value of the deviation between the positions of a set of feature points in the image to be registered, the positions in the reference image after geometric transformation, and the actual positions of the feature point set in the reference image. RMSE is defined as follows:

[0134]

[0135] In the formula, N is the number of selected control points, (x i ,y i () represents the location of feature points in the reference image, and (x) represents the location of feature points in the image to be registered. i ',y i The location of the feature point.

[0136] (2) Correct Match Rate (CMR), defined as

[0137] CMR = N c / N (17)

[0138] Where N is the total number of matching point pairs, N c This represents the number of correctly matched point pairs after the matching process.

[0139] The results of the experiment are shown in the table below:

[0140] Table 1

[0141]

[0142] From the Sen1-2 dataset, three main scenarios—city, farmland, and river—were selected for comparison and evaluation. Figure 9 Six examples of optical and SAR image registration performance using the proposed algorithm are presented. It can be observed that accurate registration results are obtained for each image pair, and the obtained feature points are basically evenly distributed at the image edges, effectively improving the stability of the registration algorithm. For each pair of test images, a checkerboard image is selected to evaluate the registration algorithm, such as... Figure 10 As shown in the table. The results show that the SAR image and the optical image are accurately aligned, and the DEF-OB method has good accuracy. In summary, the algorithm of this invention, by extracting similar features from optical and SAR images, achieved satisfactory registration results in various test images. The registration performance of the algorithm was further evaluated using evaluation metrics. The quantitative results are shown in Table 1. It can be seen that the algorithm of this invention can achieve sub-pixel-level relative positioning accuracy for all test images. According to the analysis, an average of about 69 feature matching points can be obtained for each test image, with an average RMSE of 0.697. Overall, the registration accuracy of farmland and river images is higher than that of urban images. Its main advantage is that the edge information of farmland and river images is clearer than that of urban images, and the edge information generated by the improved Deeplabv3+ network has high similarity, which can achieve high-precision image registration. The registration results for slightly rotated images are shown in Table 1. Figure 11 As shown, the DEF-OB method can detect and describe the feature points of the registered image using the OFAST-BRISK algorithm, thus possessing the ability to match rotated images.

[0143] Experimental Results: To demonstrate the superiority of the proposed optical-SAR image registration algorithm, we compared it with traditional feature-based optical-SAR image registration algorithms HOPC and OS-SIFT, as well as deep learning-based algorithms DIH and D2-NET. Experimental results show that all compared methods suffer from issues such as low registration accuracy, low feature point dispersion, and mismatches. Our proposed method fully combines the advantages of both traditional and deep learning image registration algorithms. In the similarity feature extraction model, NLM-Log edge extraction and an improved Deeplabv3+ model are first used to extract more stable similar edge information between optical and SAR images. Then, the OFAST-BRISK algorithm is used to accurately and effectively extract a sufficient number of feature points from the similar edges. In the image matching model, the proposed improved RANSAC algorithm further ensures the accuracy and quantity of the final matched point pairs, providing a basis for the conversion model. Experimental results on optical and SAR images with translation and rotation operations show that the proposed method can obtain robust and uniformly distributed feature point matching pairs, and the registration accuracy is better than other representative traditional registration methods and deep learning registration methods.

Claims

1. A method for optical and SAR image registration based on similar edge feature extraction and OFAST-BRISK algorithm, characterized in that, The steps of this method are as follows: Step 1: Select the optical and SAR image pair dataset as the training and validation data for the deep learning network, and randomly select six pairs of images from the dataset to perform translation and rotation transformations to simulate real optical and SAR image registration scenarios. Step 2: Replace the Gaussian filter in the original Log detection algorithm with NLM filter to obtain an improved Log edge detection algorithm. Extract the edges of SAR images in the dataset and use the edge extraction results as shared edge label data for the deep learning network. Step 3: Feed the optical image and shared edge label data, and the SAR image and shared edge label data into the deep learning network for training. After training, the deep learning network can extract the similarity edge features between the optical and SAR images. The deep learning network is an improved Deeplabv3+ deep learning network, in which Inception-ResNet-v2 is used as the backbone network of the encoder. Step 4: Based on the extraction of similarity edge features between optical and SAR images, the proposed OFAST-BRISK algorithm is used to extract and describe feature points in the optical and SAR images. Step 5: Use an improved RANSAC algorithm to match feature points and complete image registration; In step four, the OFAST-BRISK algorithm can be divided into two parts: 1) The OFAST algorithm is used to extract feature points from optical and SAR edge images. The coordinates of the feature points and the centroid form a vector as the direction of the feature points, and the overall direction θ of the feature points is determined. 2) The BRISK algorithm is used to extract binary vector descriptors; The main direction θ determined by the OFAST algorithm is used as the main direction of the feature descriptor extracted by the BRISK algorithm. Before the feature point description, the sampling mode is rotated according to the overall direction θ determined by the feature point to ensure the rotation invariance of the feature descriptor. Based on the scale and orientation of the feature points, the gray values ​​of all two points in short-distance pairs of the feature point sampling points are compared. This comparison yields a 512-bit binary vector descriptor, where each bit b of the binary vector descriptor corresponds to: This represents the sampling point pair (p) after rotating each point on the sampling template by an angle θ. i ,p j The smoothed intensity value after Gaussian filtering with a standard deviation of σ; A subset S of short-distance point pairs: The set of all sampling point pairs Ω: Ω={(p i ,p j )∈R 2 ×R 2 ∣i<N,j<i,(i,j)∈N} (11) The threshold distance is set to δ max = 9.75t, where t is the scale of the feature point; The specific steps of the improved RANSAC algorithm in step five are as follows: (1) Coarse matching O = {O1,...,O} n S = {S1,...,S} n Feature points are extracted from optical and SAR images, respectively. First, the Euclidean distance between all points in set O and all points in set S is calculated, and all matching point pairs are sorted from smallest to largest Euclidean distance. Four pairs of points at the same location are randomly selected from the two sets. These eight feature points can form a homology matrix, resulting in: Where (x,y) and (x′,y′) are the remaining corresponding feature points in sets O and S, respectively, and Z is the scale parameter; then, the projection error of all feature matching point pairs in the dataset is calculated; if the projection error of the data is less than the set threshold, the feature points are placed into sets O' and S' respectively; the error calculation formula is as follows: After repeatedly and randomly selecting four pairs of feature points, the case with the most feature point pairs is retained in a set, denoted as set O'. max and S' max , defined as the initial matching set; although the RANSAC algorithm filters out some mismatches in the original set, some mismatches still exist due to the large differences between multimodal images; therefore, it is necessary to further remove obviously erroneous correspondences in the initial matching set and find more correct matching points; (2) Precise matching Based on prior knowledge, the slopes of the lines between correctly matched feature points are the same; therefore, feature points are accurately matched based on the consistency of the slopes; therefore, O' max and S' max The slope parameter k between two candidate points in the set is calculated as follows: Then O' max and S' max All slope values ​​k i All of these can be calculated, k mean It is the average of all slope values; based on this, select the interval [k] mean -0.5,k mean Point pairs with slopes within [+0.5] are used to form a refined matching set; finally, based on these correct correspondences, image registration is completed through affine transformation.

2. The optical and SAR image registration method based on similar edge feature extraction and OFAST-BRISK algorithm according to claim 1, characterized in that, The process of replacing the Gaussian filter in the original Log edge detection algorithm with an NLM filter in step two to obtain an improved Log edge detection algorithm is as follows: 1) The filtered image obtained by using an NLM filter is I(i), which can also be represented by I(x,y). The specific steps are as follows: Assuming a discrete noisy image is f(x,y), it can also be expressed as: f={f(i)|i∈Ω}, where Ω represents the image neighborhood and i is the pixel index. After NLM filtering of the noisy image, it can be normalized as: Ω i Let represent a search window with center pixel i and size q×q; ω(i,j) represents the weights assigned to the noisy image, which can be expressed as: Where h represents the filter coefficient that controls the smoothness of the filter, which is usually the standard deviation of the noise; N(i) and N(j) represent the image blocks centered at pixel i and pixel j with a window size of p, respectively; d(i,j) represents the Euclidean weighted distance between N(i) and N(j); G represents the convolution between two matrices; α It is a Gaussian matrix; 2) The second-order directional derivative of the filtered image I(x,y) is processed using the Laplacian operator, and the resulting image M(x,y) is the result of edge extraction. Edge extraction was performed on SAR images selected from the dataset, which were then used as label data for the deep learning network.

3. The optical and SAR image registration method based on similar edge feature extraction and OFAST-BRISK algorithm according to claim 1, characterized in that, Step four, in the OFAST-BRISK algorithm, the OFAST algorithm is used to extract feature points from optical and SAR edge images. The coordinates of the feature points and the centroid form a vector as the direction of the feature points. The specific steps for determining the overall direction θ of the feature points are as follows: First, the FAST method is used to detect feature points. Then, a spatial pyramid is used to generate multi-scale features. Finally, the dense centroid algorithm is used to give the overall orientation of the feature points. In the dense centroid algorithm, it is assumed that the feature point has a centroid with radius r. The coordinates of the feature point and the centroid form a vector, which serves as the orientation of the feature point. The moment is defined as follows: Where I(x,y) is the gray value at position (x,y) in the image; Therefore, these moments are used to determine the centroid of the local neighborhood of the image: Where m 00 m represents the sum of gray levels of all pixels in the image's neighborhood window. 10 and m 01 These are the first moments about x and y, respectively; the coordinates of the feature point and the centroid form a vector that serves as the direction of the feature point, determining the overall direction of the feature point as follows: θ=arctan(m 01 ,m 10 ) (8)。

Images