Improved superpoint-based sar and optical image registration method

Abstract

The application provides a SAR and visible light image registration method based on an improved SuperPoint network, takes the SuperPoint network as a basic network model, combines multi-scale feature extraction, key point detection construction, feature descriptor construction, feature matching and elimination to implement image pair registration. The method comprises the following steps: B100: joint coding stage (GhostNet extracts deep basic features, and Adaptive Spectrum Congruency (ASC) structure prior coding); B200: improved interest point decoding and descriptor decoding stage (interest point decoding based on a Sobel operator, feature descriptor based on an ASC local histogram, a deep basic feature module, a harmonic regression head and a loss function); and B300: feature matching and error elimination (initial registration, error point elimination and final matching effect). The application effectively solves the problems of low registration accuracy and poor robustness caused by large imaging mechanism difference, large noise influence and geometric distortion in SAR and visible light image registration, and provides a reliable technical scheme for multi-modal image registration.

Description

Technical Field

[0001] This invention belongs to the field of multimodal image matching, specifically involving an improved SuperPoint network-based SAR and visible light image registration method, which is suitable for image registration and detection of SAR and visible light images under complex backgrounds, such as camouflaged target detection, urban construction monitoring, agricultural and forestry resource monitoring, and ecological protection monitoring. Background Technology

[0002] With the continuous iteration of intelligent, connected, and spatial information technologies, multimodal image registration and fusion technology has entered a stage of rapid development. In particular, the continuous maturation and popularization of intelligent technologies such as autonomous driving, smart transportation, and intelligent tracking have further expanded the application scenarios of multimodal image registration technology. Currently, it widely covers multiple military-civilian integration fields such as battlefield reconnaissance and surveillance, urban intelligent monitoring, natural disaster relief, emergency response to public emergencies, industrial non-destructive testing, product quality control, and medical non-invasive imaging, with broad market application prospects.

[0003] In multimodal remote sensing image processing and practical applications, high-precision image registration is crucial for ensuring the success of subsequent data fusion, feature extraction, and target recognition. Heterogeneous image registration, a key branch of multimodal remote sensing image processing, specifically refers to the technical process of matching and overlaying two or more remote sensing images of different modalities acquired from the same scene using specific algorithms, achieving accurate multimodal image registration. Due to the fundamental differences in spatial geometric parameters and physical radiative transfer mechanisms between the sensors acquiring different modalities of remote sensing images, multimodal remote sensing images generally exhibit complex geometric distortions, radiative distortions, and nonlinear geometric differences. This increases the difficulty of identifying and matching corresponding feature points between heterogeneous images. Furthermore, ground targets exhibit significant differences in response values ​​to sensors of different bands, with each response representing different physical characteristics of the ground object. In addition, remote sensing images are susceptible to factors such as terrain undulations, complex textures, and illumination variations, further exacerbating the differences in pixel representation between heterogeneous images. This severely restricts the accuracy and efficiency of remote sensing image registration, making it difficult to meet the needs of high-end application scenarios.

[0004] Currently, in the field of intelligent registration technology for SAR and optical heterogeneous images, the commonly used registration algorithms in the industry are mainly divided into three categories: registration algorithms based on feature points, registration algorithms based on regions, and registration algorithms based on structural features. These three types of algorithms have obvious limitations and shortcomings in practical applications, and are difficult to adapt to the high-precision and real-time registration requirements of complex urban scenarios. Feature-point-based registration algorithms rely on the extraction and matching of stable feature points in the image. Their advantages lie in high computational efficiency and fast registration speed. However, these algorithms are extremely sensitive to feature differences between heterogeneous images and are easily affected by factors such as image noise, repetitive textures, and illumination changes. Their registration stability is poor in complex urban scenes. Region-based registration algorithms traverse the image through a sliding window and calculate the similarity of corresponding regions in different images to achieve registration. They are highly sensitive to changes in local regions and are greatly affected by the differences in radiometric characteristics between SAR and optical images, resulting in poor real-time registration and difficulty in meeting the real-time requirements of time-sensitive target monitoring. Structural feature-based registration algorithms rely on the geometric structure information of the scene for matching. They have high requirements for the integrity of the scene structure and are easily affected by factors such as differences in the structural morphology of heterogeneous images and occlusion of ground objects. Their adaptability in complex scenes is poor. To address the technical problems of existing registration methods, such as poor sensitivity to heterogeneous image features, weak noise resistance, insufficient adaptability to structural morphology differences, and poor real-time performance, this invention designs an improved SuperPoint network-based SAR and visible light image registration method. The aim is to achieve accurate extraction, efficient matching, and pixel-level registration of feature points in heterogeneous images, providing a more efficient and reliable technical solution for multi-source remote sensing performance. Summary of the Invention

[0005] The purpose of this invention is to provide an improved SuperPoint network-based SAR and visible light image registration method to address key issues such as low registration accuracy and poor robustness caused by differences in imaging mechanisms, significant noise, and geometric distortion in SAR and visible light image registration. This provides an efficient and reliable technical solution for multimodal image registration. The method includes the following steps:

[0006] B100: Joint encoding stage (GhostNet extracts deep basic features, ASC structure prior encoding);

[0007] B200: Improved joint decoding stage (interest point decoding based on Sobel operator, feature descriptor based on ASC local histogram, deep basic feature module, homology regression head, loss function);

[0008] B300: Feature matching and error removal (initial registration, error point removal, final matching result).

[0009] In step B100 of this invention, the joint encoding stage is divided into two parts: a deep basic feature extraction module and an ASC structure prior encoding module. These modules extract multi-scale features from image pairs from different depths and angles. The constructed deep basic feature module and the ASC structure prior construct a nonlinear scale space, which extracts stable structural information at multiple scales while adapting to multimodal image content.

[0010] In step B200 of the invention, the geometric consistency representation capability of network features is improved by refining the joint decoding stage. This process can effectively improve the registration effect of SAR and visible light images. The specific implementation process is as follows: First, after multi-scale feature extraction, the Sobel operator is introduced for gradient extraction, injecting structural prior information into interest point detection, enhancing the network's ability to perceive image edges, and thus improving the accuracy of interest point detection, providing high-quality interest points for subsequent registration; then, a local histogram feature descriptor based on ASC is constructed to improve the problems of nonlinear radiation differences and geometric deformation in the SAR and visible light image registration process, reduce the registration deviation caused by different imaging mechanisms, and improve the rationality and stability of feature matching; then, based on Sobel... Interest point detection results from the l-operator and the deep basic feature module are fed into the interest point decoder for interest point extraction. Simultaneously, feature descriptors based on ASC local histograms and the deep basic feature module are fed into the descriptor decoder to obtain discriminative feature descriptor representations, ensuring spatial alignment for registration. Finally, the outputs of the interest point decoder, the descriptor decoder, and the homology regression head are used to calculate the loss function, which is then revised and optimized to effectively enhance the feature representation capabilities of image pairs. This ultimately achieves improved inlier rate, reduced reprojection error, and improved spatial alignment accuracy during registration, while also strengthening the robustness of the registration results. This provides a better registration foundation for subsequent image fusion, object detection, and other tasks, improving the overall quality and efficiency of image processing tasks.

[0011] In step B300 of this invention, during the initial feature matching stage, initial feature matching point pairs are generated by combining similarity analysis, cross-term verification, and proportional matching. Next, the initial matching results are sorted and filtered. A combination of Top-K filtering and uniform grid sampling is used to eliminate low-quality matching point pairs, reducing interference in the subsequent geometric registration process and ensuring the stability and accuracy of the registration process. Then, an error removal operation is performed, using the Random Sample Consensus Algorithm (RANSAC) to impose geometric consistency constraints on candidate matching point pairs, accurately eliminating erroneous matching points and refining the matching results. Finally, geometric registration processing is performed. Based on the refined set of inliers, the optimal geometric transformation matrix is ​​re-estimated to complete the spatial alignment of the source and target images. Simultaneously, core evaluation indicators such as the number of inliers, inlier rate, reprojection error, and registration success rate are output to quantitatively evaluate the registration performance and provide data support for verifying the registration effect.

[0012] This invention implements a registration method for SAR and visible light images through an improved SuperPoint network, which improves the matching pairs of multimodal images and effectively solves the problems of low registration accuracy and poor robustness caused by geometric distortion and noise interference in the registration of SAR and visible light images. It provides a more efficient and reliable technical solution for multimodal image registration. Attached Figure Description

[0013] Figure 1 This is an overall flowchart of an improved SuperPoint network-based SAR and visible light image registration method provided in one embodiment of this disclosure.

[0014] Figure 2 This is a flowchart of the GhostNet network coding layer structure provided in one embodiment of this disclosure.

[0015] Figure 3 This is a flowchart of the ASC structure prior encoding provided in one embodiment of this disclosure.

[0016] Figure 4 This is a flowchart of interest point decoding based on the Sobel operator provided in one embodiment of this disclosure.

[0017] Figure 5 This is a flowchart illustrating the construction of feature descriptors based on ASC local histograms, provided in one embodiment of this disclosure.

[0018] Figure 6 This is a flowchart of the total loss function calculation provided in one embodiment of this disclosure.

[0019] Figure 7 This is a flowchart of image error removal and registration provided in one embodiment of the present disclosure.

[0020] Figure 8 This is a diagram showing the execution result of an improved SuperPoint network-based SAR and visible light image registration method provided in one embodiment of this disclosure. Detailed Implementation

[0021] For multimodal image registration, due to the significant differences in imaging mechanisms, noise interference, and nonlinear geometric effects, the robustness of multimodal image registration is poor, resulting in unsatisfactory target recognition performance. Currently, commonly used multimodal image registration methods include feature matching-based methods, region feature-based methods, and deep learning-based methods. However, the first two methods are unsuitable for registering SAR images with other multimodal images, performing poorly in acquiring image detail information and under complex backgrounds. Deep learning methods, due to their powerful feature learning capabilities and strong adaptability, were chosen. This study utilizes an improved SuperPoint network as a foundation to register SAR and visible light images, aiming to overcome the limitations of SAR-visible light image registration and achieve fast, comprehensive, and accurate registration of image pairs, providing a valuable reference for subsequent high-quality image pair fusion. Figure 1 This is the overall framework and process of the model, and the method includes the following steps:

[0022] B100: Joint encoding stage (GhostNet extracts deep basic features, ASC structure prior encoding);

[0023] B200: Improved joint decoding stage (interest point decoding based on Sobel operator, feature descriptor based on ASC local histogram, deep basic feature module, homology regression head, loss function);

[0024] B300: Feature matching and error removal (initial registration, error point removal, final matching result).

[0025] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0026] Reference Figure 1The joint encoding stage mainly consists of GhostNet extracting deep basic features and ASC structure prior encoding. Firstly, for deep basic feature extraction, this invention uses the GhostNet structure instead of the traditional VGG structure. This structure, as a lightweight residual network, has the advantage of rich feature information and can effectively improve the computational efficiency of the multimodal image encoder. When using the GhostNet network to perform deep basic feature extraction, the first 7 layers of the network are selected as shared encoding layers, and a feature pyramid structure is used to replace the original single convolutional kernel. Specifically, the first layer uses convolutional kernels of various sizes (3×3, 5×5, 7×7, 9×9) to replace the original 3×3 convolutional kernel of the GhostNet network, while maintaining the same number of channels. Layers 2 to 7 retain the original Ghost bottleneck (G-bneck) structure, which consists of two stacked Ghost modules. After increasing the number of channels through an extension layer, channel compression matching is performed to expand the receptive field, thereby aggregating more detailed information between multimodal images. The specific structure of the GhostNet network encoding layers is shown in the figure. Figure 2 As shown, the symbols in the image have the following meanings: They indicate that the image size has been reduced. And the number of channels is C, This represents the convolution operation. This indicates a batch standardization operation. This represents the activation function.

[0027] Reference Figure 3 For the ASC structure prior encoding stage, let the input image be... To enhance the stability of the registration structure response between SAR and visible light images, a nonlinear multi-scale space is constructed based on the SAR and visible light images, assuming the initial scale image is:

[0028] (1)

[0029] in, The standard deviation is expressed as The Gaussian kernel is then used, followed by the anisotropic diffusion equation to generate higher-scale layers, with the iterative update formula being:

[0030] (2)

[0031] The diffusion coefficient is defined as:

[0032] (3)

[0033] in, is the diffusion coefficient in the corresponding direction. These represent the differences in the north, south, west, and east directions, respectively. For diffusion control parameters, The diffusion step size allows for the construction of multi-scale image sets. .

[0034] To further characterize the consistency of multi-scale local responses, a consistency metric is defined as:

[0035] (4)

[0036] in Represented as the mean of local amplitudes at multiple scales. The standard deviation of the local amplitudes at multiple scales is represented by the sigmoid function, and therefore the consistency weights are constructed as follows:

[0037] (5)

[0038] Finally, after normalizing the ASC map at each scale, a multi-scale fusion strategy is applied to obtain the final ASC feature map:

[0039] (6)

[0040] in, This serves as a structural prior input for subsequent interest point detection and descriptor construction, acting as a branch to enhance the structural consistency of heterogeneous image registration.

[0041] Reference Figure 4 In the Sobel operator-based interest point decoding stage, the ASC structure prior feature map is first used as input, and the Sobel operator is used to calculate the horizontal gradient. and vertical gradient ,in Gaussian smoothing is then used to construct a local second-order matrix using gradient information, and Gaussian smoothing is applied to enhance the stability of the local structure. Finally, the two eigenvalues ​​of this local structure matrix are calculated. and The smaller value R is selected as the response value of the interest point to highlight corner points and structural intersection areas and suppress single edge responses. Finally, threshold filtering, non-maximum suppression, boundary removal, and maximum point number constraints are applied to the response map to obtain the final set of key points.

[0042] (7)

[0043] (8)

[0044] (9)

[0045] Reference Figure 5In the process of constructing feature descriptors based on ASC local histograms, the multi-scale fused ASC feature map is first used as input and adjusted to match the backbone features. Figure 1 To achieve the desired spatial resolution, the Sobel operator is used to calculate the horizontal and vertical gradients of the ASC features, obtaining the gradient magnitude and direction. The gradient direction is then divided into several directional channels, and soft allocation is performed using cosine projection. The gradient magnitude is then combined with these channels to construct the response for each direction. Next, average pooling is performed on the responses in their local neighborhoods to form local direction histogram features, which are then concatenated with the original ASC channels to form the LHASC-like descriptor prior. Finally, this prior is fused with the deep backbone features and input into the descriptor decoding branch to generate the final local feature descriptor. In constructing LHASC, ASC is used to calculate the gradient direction and magnitude information, followed by logarithmic polar coordinate neighborhood pooling to obtain the local histogram descriptor (LHASC), as shown in the formula:

[0046] (10)

[0047] (11)

[0048] Reference Figure 6 In this paper, the total loss is defined as the sum of the interest point detection loss function, the feature description loss function, and the homology regression loss function. The interest point detection loss function uses cross-entropy with class weights to constrain keypoint location prediction; the feature description loss function uses a comparative constraint based on positive and negative sample margins to make descriptors of corresponding points closer and descriptors of dissimilar points more separated; the homology regression loss uses Huber loss to geometrically supervise the four corner point offsets predicted by the network. Therefore, the interest point detection loss function is defined as follows: Feature description loss function Homophonic regression loss function They are respectively:

[0049] (12)

[0050] (13)

[0051] (14)

[0052] (15)

[0053] in For the first The actual category label of each cell; Output the predicted class distribution corresponding to logits to the network; Represents the softmax cross-entropy; For effective area masking; Category weights; Indicates the similarity between the original image and the feature descriptor; Indicates whether it is a geometrically corresponding positive sample; The boundary of positive samples; The boundary of negative samples; Weights for feature descriptor loss; Normalization factor; To predict corner offset; For corner offset monitoring generated from the real homography matrix; For Huber threshold; As the weights for the homophonic regression loss, based on the configuration in this paper, we take...

[0054] (16)

[0055] Reference Figure 7 To improve the effectiveness of feature matching points between SAR and visible light images, this paper adopts an overall strategy combining feature matching, RANSAC mismatch removal, and geometric registration. First, in the feature matching stage, candidate correspondences are constructed based on key points and feature descriptors extracted from the two images. Initial matching point pairs are generated through similarity calculation, cross-term constraints, and proportional matching strategies. Then, the initial matching results are sorted and filtered, combining Top-K optimal retention with grid uniform sampling to ensure that candidate matching point pairs have both high confidence and spatial uniformity, thereby reducing the interference of low-quality matches on the subsequent geometric registration process. Based on this, the Random Sample Consensus Algorithm (RANSAC) is used to apply geometric consistency constraints to the candidate matching point pairs. The geometric model is estimated through random sampling iteration, and reprojection error is used as a criterion to distinguish inliers from outliers, achieving effective removal of mismatches. Finally, the optimal geometric transformation matrix is ​​re-estimated based on the retained inlier set, completing accurate spatial alignment from the source image to the target image. Indicators such as the number of inliers, inlier rate, reprojection error, and registration success rate are output simultaneously to achieve a quantitative evaluation of registration performance.

[0056] refer to Figure 8 The images show the results of the SAR and visible light image feature registration method based on SuperPoint adaptive spectral consistency. First, the original SAR and visible light images are input. Then, the results of basic deep feature block extraction and nonlinear multi-scale feature block extraction are shown. Next, the results of Sobel operator and interest point detection and feature descriptor representation based on ASC local histogram are shown. Finally, the results of the initial matching of the image pair and the result after error removal are shown.

[0057] The above description, in conjunction with the accompanying drawings, illustrates specific embodiments of the improved SuperPoint network-based SAR and visible light image registration method of the present invention. However, it should be clarified that the scope of protection of the present invention is not limited to the detailed design of the described embodiments. These embodiments are merely exemplary cases of typical application scenarios. Their core lies in the improved SuperPoint network, which exhibits better robustness and effectiveness in SAR and visible light image registration. The algorithm increases the number of correct matches by approximately 1.83 times and the average correct matching rate by approximately 12.38%, thereby improving the accuracy of multimodal image registration and providing a valid reference for subsequent image pair fusion.

Claims

1. A method for SAR and visible light image registration based on an improved SuperPoint network. This method uses the SuperPoint network as the basic network model and combines multi-scale feature extraction, keypoint detection and construction, feature descriptor construction, feature matching, and error removal to achieve image pair registration. It includes: B100: Joint encoding stage (GhostNet extracts deep basic features, ASC structure prior encoding); B200: Improved joint decoding stage (interest point decoding based on Sobel operator, feature descriptors based on ASC local histograms, deep basic feature modules, homology regression head, loss function); B300: Feature matching and error removal (initial registration, error point removal, final matching result). This invention effectively solves the problems of low registration accuracy and poor robustness caused by large differences in imaging mechanisms, significant noise influence, and geometric distortion in SAR and visible light image registration, providing a reliable technical solution for multimodal image registration.

2. The method for SAR and visible light image registration based on the improved SuperPoint network according to claim 1, characterized in that, The joint encoding stage described in step B100 replaces the VGG structure with a lightweight GhostNet structure.

3. The method for SAR and visible light image registration based on the improved SuperPoint network according to claim 1, characterized in that, In step B100, the joint coding stage introduces ASC to achieve noise energy estimation, constructs a non-scale feature space, and extracts multi-scale structural consistency features.

4. The method for SAR and visible light image registration based on the improved SuperPoint network according to claim 1, characterized in that, The introduction of the Sobel operator for gradient extraction in the interest point decoding stage described in step B200 introduces geometric structure priors for interest point detection, improves the localization and repetition rate detection of interest points between SAR and visible light images, and suppresses noise and nonlinear differences between images.

5. The method for SAR and visible light image registration based on the improved SuperPoint network according to claim 1, characterized in that, The ASC local histogram-based feature descriptor described in step B200 effectively suppresses nonlinear radiometric differences and geometric deformations between SAR and visible light image registration.

6. The method for SAR and visible light image registration based on the improved SuperPoint network according to claim 1, characterized in that, Step B200 describes incorporating homology regression constraints into the loss function as a weak geometric regularizer to enhance the correspondence between key points between SAR and visible light images.

7. The method for SAR and visible light image registration based on the improved SuperPoint network according to claim 1, characterized in that, The joint decoding of interest point based on Sobel operator, feature descriptor based on ASC local histogram, deep basic feature module, homology regression head and loss function described in step B200 effectively improves interest point detection and feature descriptor expression in SAR and visible light image registration.

8. The method for SAR and visible light image registration based on the improved SuperPoint network according to claim 1, characterized in that, In step B300, the initial stage of feature matching uses similarity, cross term, and ratio matching strategies to generate initial matching points.

9. The method for SAR and visible light image registration based on the improved SuperPoint network according to claim 1, characterized in that, The matching results are sorted and filtered in step B300, and a Top-K retention and grid uniform sampling strategy is adopted to reduce the interference of low-quality matching on subsequent geometric registration.

10. The method for SAR and visible light image registration based on the improved SuperPoint network according to claim 1, characterized in that, The error removal described in step B300 uses the Random Sample Consensus Algorithm (RANSAC) to constrain the geometric consistency of candidate matching point pairs, removes erroneous matches, and then uses the retained set of inliers to re-estimate the optimal geometric transformation matrix and complete the spatial alignment from the source image to the target image. At the same time, it outputs evaluation indicators such as the number of inliers, inlier rate, reprojection error and success rate to quantitatively evaluate the registration performance.

Images