Visible light and SAR image registration method based on Gaussian process and Transform

Abstract

The invention relates to a visible light and synthetic aperture radar (SAR) image registration method based on a Gaussian process and a Transform, which effectively solves the problem of accurate registration of a repetitive scene of a different-source image by combining the feature refinement capability of the Transform and the probability modeling capability of the Gaussian process. According to the specific scheme, a visible light image, an SAR image training data set and a to-be-registered image are manufactured, a Gaussian process is used for embedding higher dimensionality after a feature pyramid is processed, Fourier transform is carried out, repetitive features are mapped to a specific region of a frequency domain, a plurality of matching candidates are generated, and a matching result with the highest probability of global judgment fitting is obtained. Then refining features from a large amount of image data by using a Transform module, screening a corresponding relation with high confidence, and training the network by taking cross entropy as a loss function; then, image registration can be carried out on a visible light image and an SAR image by using the network, and a transformation matrix is obtained;

Description

Technical Field

[0001] The present application relates to the technical field of image registration in computer vision, and specifically to a visible light and SAR image registration method based on Gaussian process and Transformer. Background Art

[0002] Image registration is a fundamental technology for multi-source data fusion. Its core goal is to achieve precise alignment between different images through spatial transformation. It plays a key role in fields such as autonomous driving environmental perception and remote sensing image analysis. Existing methods face challenges when applied to heterogeneous image registration, such as visible light and synthetic aperture radar (SAR). They perform poorly when dealing with similar repetitive structures, such as forests or building complexes, primarily due to limitations in their local receptive fields. Furthermore, methods for capturing global correlations between heterogeneous images need to be optimized. Summary of the Invention

[0003] In light of this, the present invention provides a visible light and SAR image registration method based on Gaussian processes and Transformers. This method organically combines the feature refinement capabilities of the Transformer with the probabilistic modeling capabilities of the Gaussian process. This collaborative design effectively handles the nonlinear mapping relationships of repetitive structures between heterogeneous images through the probabilistic framework provided by the Gaussian process, while simultaneously overcoming the inherent image variability by leveraging the powerful feature learning capabilities of the Transformer. Unlike a single matching relationship, the Gaussian process is used to determine the optimal matching model, providing a new technical solution for the precise registration of visible light and SAR images.

[0004] To achieve the above objectives, the technical solutions of the embodiments of the present application are as follows:

[0005] A Gaussian process and Transformer-based method for registering visible light and SAR images uses a Gaussian process to embed higher dimensions and perform Fourier transforms. This method maps repetitive features to specific regions in the frequency domain, generates multiple matching candidates, and then globally determines the most likely matching result. The Transformer module then refines features from a large amount of image data, screening for high-confidence correspondences. The network is then trained using cross-entropy as the loss function. The network can then be used to register visible light and SAR images and calculate the transformation matrix. The method includes the following steps:

[0006] Step S1: Collect visible light and corresponding SAR images, and perform different rotations, translations, and noise superposition to form image pairs, which serve as the visible light training dataset Train_Vis and the SAR training dataset Train_SAR. In addition, prepare pairs of visible light images to be registered Test_Vis and SAR images to be registered Test_SAR.

[0007] Step S2: Use the datasets Train_Vis and Train_SAR in step S1 to build a network feature pyramid and refine the preliminary features;

[0008] Step S3: Normalize the datasets Train_Vis and Train_SAR from step S2, concatenate them in the batch dimension, and then input them into the Gaussian process module for Fourier transformation. Modeling is used to determine the feature similarity between them, generate multiple matching candidates, and select the one with the highest similarity as the preliminary matching result.

[0009] Step S4: Input the result of step S3 into the Transformer module, refine and filter the correspondences with high confidence as matching feature points, and use cross entropy as the loss function to train the network parameters;

[0010] Step S5: Input the visible light image Test_Vis to be registered and the SAR image Test_SAR to be registered in step S1 into the network, calculate the pairs of valid feature points, and calculate the transformation matrix based on the feature points.

[0011] Specifically, the visible light image and SAR image in step S1 include image pairs of different scenes, different times, and different viewing angles.

[0012] Specifically, the network feature pyramid described in step S2 includes four convolutional layers at different stages, which can extract feature maps of four different sizes;

[0013] The input of the first convolutional layer is a 3×320×240 feature map, the size of the convolution kernel is 3×3, the stride is 1, and the output is a 8×320×240 feature map;

[0014] The input of the second convolutional layer is a feature map of 8×320×240, the size of the convolution kernel is 3×3, the stride is 2, and the output is a feature map of 16×160×120;

[0015] The input of the third convolutional layer is a 16×160×120 feature map, the size of the convolution kernel is 3×3, the stride is 2, and the output is a 32×80×60 feature map;

[0016] The input of the fourth convolutional layer is a 32×80×60 feature map, the size of the convolution kernel is 3×3, the stride is 2, and the output is a 64×40×30 feature map;

[0017] Specifically, in step S3, a normalization operation is performed before entering the Gaussian process to standardize the value of each pixel so that the mean is 0 and the variance is 1.

[0018] Specifically, the Fourier transform in step S3 uses a cosine convolution kernel operation. Since the convolution kernel is axially symmetric at y=0 and has periodicity, the corresponding pixels will show similar responses.

[0019] Specifically, in step S3, a Gaussian process is used to fit the pixel correspondence. During the image registration process, the corresponding pixels A and B in the two images should have the highest correlation, while the correlation between the pixels around B and A decreases with increasing distance, which is similar to a Gaussian distribution. The specific formula for calculating the similarity measure is as follows:

[0020]

[0021] in,<A,B> Indicates that both perform cosine calculations; τ and ε are both constants. τ is used to adjust the sensitivity of the similarity measure, and ε is a correction constant to prevent the denominator from being 0, thus ensuring the stability of the algorithm.

[0022] Specifically, the Transformer module in step S4 consists of 6 encoders and 6 decoders, where each encoder consists of a multi-head attention mechanism and a feedforward residual network, and each decoder consists of a multi-head attention mechanism and a multi-layer perceptron.

[0023] Specifically, the cross entropy loss function in step S4 is defined as follows:

[0024]

[0025] Where H represents the height of the image, W represents the width of the image, y represents the label of the pixel, that is, the value of each channel, t i Indicates the prediction result.

[0026] After the training in step S4, a feature extraction registration model is obtained, which can be used in step S5 to calculate the correspondence between the visible light image Test_Vis to be registered and the SAR image Test_SAR to be registered.

[0027] Specifically, the method for calculating the transformation matrix in step S5 is: first estimate the homography matrix based on the RANSAC algorithm, and then use the least squares method to obtain the accurate transformation matrix.

[0028] The purpose of the present invention is to solve the registration problem of visible light images and SAR images, and to propose a registration method based on Gaussian processes and Transformers. Visible light and corresponding SAR images are collected and rotated, translated, and noise-superimposed as the original training data set. In the neural network, the image is first subjected to preliminary feature extraction, then input into the Gaussian process module, Fourier transformed and embedded in high dimensions, and the initial correspondence is fitted. The Transformer is then used to refine the features, screen out corresponding feature points with high confidence, and use the cross-entropy loss function for network feedback. The trained network can be used for image registration of visible light images and SAR images.

[0029] The beneficial technical effects brought about by the present invention are:

[0030] (1) The present invention overcomes the problem of difficulty in alignment caused by imaging differences between SAR imaging and visible light imaging.

[0031] (2) The present invention reduces the dependence of existing registration algorithms on viewing angle.

[0032] (3) The present invention provides a training module, which can be subsequently trained for different scenarios to improve the accuracy of registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The present invention depicts a flow chart of the visible light and SAR image registration method based on Gaussian process and Transformer.

[0034] Figure 2 The network feature pyramid structure diagram in the present invention is depicted.

[0035] Figure 3 Depicts a schematic diagram of the Transformer module in the present invention. DETAILED DESCRIPTION

[0036] The present invention will be described in detail below with reference to the accompanying drawings.

[0037] This paper provides a visible light and SAR image registration method based on Gaussian processes and Transformers. This method organically combines the Transformer's feature refinement capabilities with the Gaussian process's probabilistic modeling capabilities. This collaborative design effectively handles the nonlinear mapping relationships of repetitive structures between heterogeneous images through the probabilistic framework provided by the Gaussian process, while simultaneously overcoming the inherent differences in the images by leveraging the Transformer's powerful feature learning capabilities. Unlike a single matching relationship, this method uses Gaussian processes to obtain the optimal matching model, providing a new technical solution for the precise registration of visible light and SAR images. To achieve the above objectives, the technical solution is as follows:

[0038] A Gaussian process and Transformer-based method for registering visible light and SAR images uses a Gaussian process to embed higher dimensions and perform Fourier transforms. This method maps repetitive features to specific regions in the frequency domain, generates multiple matching candidates, and then globally determines the most likely matching result. The Transformer module then refines features from a large amount of image data, screening for high-confidence correspondences. The network is then trained using cross-entropy as the loss function. The network can then be used to register visible light and SAR images and calculate the transformation matrix. The method includes the following steps:

[0039] Step S1: Collect visible light and corresponding SAR images, and perform different rotations, translations, and noise superposition to form image pairs, which serve as the visible light training dataset Train_Vis and the SAR training dataset Train_SAR. In addition, prepare pairs of visible light images to be registered Test_Vis and SAR images to be registered Test_SAR.

[0040] Step S2: Use the datasets Train_Vis and Train_SAR in step S1 to build a network feature pyramid and refine the preliminary features;

[0041] Step S3: Normalize the datasets Train_Vis and Train_SAR from step S2, concatenate them in the batch dimension, and then input them into the Gaussian process module for Fourier transformation. Modeling is used to determine the feature similarity between them, generate multiple matching candidates, and select the one with the highest similarity as the preliminary matching result.

[0042] Step S4: Input the result of step S3 into the Transformer module, refine and filter the correspondences with high confidence as matching feature points, and use cross entropy as the loss function to train the network parameters;

[0043] Step S5: Input the visible light image Test_Vis to be registered and the SAR image Test_SAR to be registered in step S1 into the network, calculate the pairs of valid feature points, and calculate the transformation matrix based on the feature points.

[0044] Specifically, the visible light image and the SAR image in step S1 are image pairs of different scenes, different times, and different viewing angles.

[0045] Specifically, the network feature pyramid structure described in step S2 is as follows: Figure 2 As shown in the figure, it includes 4 convolutional layers at different stages, which can extract feature maps of 4 different sizes;

[0046] The input of the first convolutional layer is a 3×320×240 feature map, the size of the convolution kernel is 3×3, the stride is 1, and the output is a 8×320×240 feature map;

[0047] The input of the second convolutional layer is a feature map of 8×320×240, the size of the convolution kernel is 3×3, the stride is 2, and the output is a feature map of 16×160×120;

[0048] The input of the third convolutional layer is a 16×160×120 feature map, the size of the convolution kernel is 3×3, the stride is 2, and the output is a 32×80×60 feature map;

[0049] The input of the fourth convolutional layer is a 32×80×60 feature map, the size of the convolution kernel is 3×3, the stride is 2, and the output is a 64×40×30 feature map;

[0050] In some embodiments, a normalization operation is performed in step S3 before entering the Gaussian process to standardize the value of each pixel so that the mean is 0 and the variance is 1.

[0051] Specifically, the Fourier transform in step S3 uses a cosine convolution kernel operation. Since the convolution kernel is axially symmetric at y=0 and has periodicity, the corresponding pixels will show similar responses.

[0052] Specifically, in step S3, a Gaussian process is used to fit the pixel correspondence. During the image registration process, the corresponding pixels A and B in the two images should have the highest correlation, while the correlation between the pixels around B and A decreases with increasing distance, which is similar to a Gaussian distribution. The specific formula for calculating the similarity measure is as follows:

[0053]

[0054] in,<A,B> Indicates that both perform cosine calculations; τ and ε are both constants. τ is used to adjust the sensitivity of the similarity measure, and ε is a correction constant to prevent the denominator from being 0, thus ensuring the stability of the algorithm.

[0055] In some embodiments, the Transformer module in step S4 is as follows: Figure 3 As shown in the figure, it consists of 6 encoders and 6 decoders, where each encoder consists of a multi-head attention mechanism and a feedforward residual network, and each decoder consists of a multi-head attention mechanism and a multi-layer perceptron.

[0056] In some embodiments, the cross entropy loss function in step S4 is defined as follows:

[0057]

[0058] Where H represents the height of the image, W represents the width of the image, y represents the label of the pixel, that is, the value of each channel, t i Indicates the prediction result.

[0059] After the training in step S4, a feature extraction registration model is obtained, which can be used in step S5 to calculate the correspondence between the visible light image Test_Vis to be registered and the SAR image Test_SAR to be registered.

[0060] Specifically, the method for calculating the transformation matrix in step S5 is: first estimate the homography matrix based on the RANSAC algorithm, and then use the least squares method to obtain the accurate transformation matrix.

[0061] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A visible light and SAR image registration method based on Gaussian process and Transformer, characterized by The combination of step S1, step S2, step S3, step S4, and step S5 is as follows: Step S1: collect visible light and corresponding SAR images, and perform data enhancement as visible light training dataset Train_Vis and SAR training dataset Train_SAR; also prepare pairs of visible light images to be registered Test_Vis and SAR images to be registered Test_SAR; Step S2: Use the datasets Train_Vis and Train_SAR in step S1 to build a network feature pyramid and refine the preliminary features; Step S3: Normalize the datasets Train_Vis and Train_SAR in step S2, concatenate them in the batch dimension, and then input them into the Gaussian process module; Step S4: Input the result of step S3 into the Transformer module, refine and filter the correspondences with high confidence as matching feature points, and use cross entropy as the loss function to train the network parameters; Step S5: Input the visible light image Test_Vis to be registered and the SAR image Test_SAR to be registered in step S1 into the network, calculate the pairs of valid feature points, and calculate the transformation matrix based on the feature points.

2. As described in claim 1, a visible light and SAR image registration method based on Gaussian processes and Transformers is characterized in that: in step S1, data enhancement can include different rotations, translations, and noise superposition to form image pairs, and can also perform grid extraction to reduce the number of true values.

3. As described in claim 1, a visible light and SAR image registration method based on Gaussian process and Transformer is characterized in that: in step S2, a feature pyramid is constructed for the training dataset, and the convolution layer may include: The input of the first convolution layer is a feature map of 3×320×240, the size of the convolution kernel is 3×3, the stride is 1, and the output is a feature map of 8×320×240; the input of the second convolution layer is a feature map of 8×320×240, the size of the convolution kernel is 3×3, the stride is 2, and the output is a feature map of 16×160×120; the input of the third convolution layer is a feature map of 16×160×120, the size of the convolution kernel is 3×3, the stride is 2, and the output is a feature map of 32×80×60; the input of the fourth convolution layer is a feature map of 32×80×60, the size of the convolution kernel is 3×3, the stride is 2, and the output is a feature map of 64×40×30.

4. As described in claim 1, a visible light and SAR image registration method based on Gaussian process and Transformer is characterized in that: in step S3, a normalization operation can be performed before entering the Gaussian process, and a cosine convolution kernel is used to perform a Fourier transform. The Gaussian process is used to model the similarity of image features, generate multiple matching candidates, and select the one with the highest similarity as the preliminary matching result.

5. As described in claim 1, a visible light and SAR image registration method based on Gaussian process and Transformer is characterized in that: in step S5, the trained network is used to perform feature extraction and feature matching, and the transformation matrix is preliminarily calculated using RANSAC, and a more accurate transformation matrix is fitted using the least squares method.

Images