Multi-angle registration model based on Fourier-Net-SAR

Abstract

The invention relates to the field of image processing, in particular to a multi-angle registration model based on Fourier-Net-SAR (Synthetic Aperture Radar). According to the technical problem, a method based on SIFI features, a method based on a deep neural network, an MU-Net multi-scale framework method, a shape context method, a method for directly matching multi-angle SAR images and a method based on semantic distribution positions are adopted to solve the registration problem of the multi-angle SAR images, and under the conditions of rare surface data and complex scenes, the registration accuracy of the multi-angle SAR images is improved. The registration performance and the robustness are insufficient to meet requirements; according to the technical scheme, the Fourier-Net-SAR-based multi-angle registration model comprises an SMA feature extraction network, a displacement field estimation layer, a DFT layer, a frequency domain optimization layer, an IDFT layer and a displacement field optimization layer; according to the SAR image registration method, the SMA feature extraction network is arranged, multiple technical modules are adopted, the technical modules cover a preprocessing strategy, STN-Attention and a multi-scale feature fusion technology, and the technical modules are combined to provide a comprehensive and efficient solution for SAR image registration.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a multi-angle registration model based on Fourier-Net-SAR. Background Art

[0002] SAR image registration is a classic problem and technical challenge in the field of SAR image processing. It aims to accurately align two or more images in space. Specifically, for two images in a set of image data sets, it maps one image to another by finding a spatial transformation method so that the corresponding spatial positions in the two images correspond one to one to achieve information fusion. This is because these images may come from different devices, different times, or different perspectives. Therefore, image registration must be performed before many image analysis tasks (such as image fusion, change detection, target recognition, and agricultural monitoring). Therefore, SAR image registration plays a vital and indispensable role in image analysis. Only by accurately aligning images acquired at different times or angles can subsequent analysis and application be continued. However, due to the different imaging times and ground observation positions of SAR images, there may be differences in radiation information and geometric distortion in the same scene, which further increases the complexity of SAR image registration. With the rapid development of deep neural network technology, deep learning-based image registration has become a hot topic in the field of SAR image registration. Compared with traditional methods, SAR registration methods based on deep learning can better capture image features and construct more sophisticated spatial mapping relationships between images. Through the feature representation of deep learning, these registration methods can mine more abstract and discriminative features from SAR images, making the common features between SAR images more obvious, thereby achieving more accurate image matching. This deep learning-based method not only performs well in cases with limited samples or complex scenes, but also shows excellent performance in large-scale data and diverse scenes. SIFI feature-based methods, deep neural network-based methods, MU-Net multi-scale framework methods, shape context methods, methods for directly matching multi-angle SAR images, and methods based on semantic distribution positions are used to solve the registration problem of multi-angle SAR images. In the case of sparse surface data and complex scenes, the registration performance and robustness are not enough to meet the needs. Summary of the invention

[0003] In order to overcome the problem of multi-angle SAR image registration solved by SIFI feature-based methods, deep neural network-based methods, MU-Net multi-scale framework methods, shape context methods, direct matching of multi-angle SAR images and semantic distribution position-based methods, the registration performance and robustness are not enough to meet the needs when the surface data is scarce and the scenes are complex.

[0004] The technical solution of the present invention is: a multi-angle registration model based on Fourier-Net-SAR, comprising:

[0005] A11: SMA feature extraction network, used to process the input SAR image and extract the features of the input SAR image. When extracting the features of the input SAR image, the characteristics of the input SAR image, such as the influence of noise and angle, are considered.

[0006] A12: Displacement field estimation layer, which is used to estimate the displacement field by adding a set of convolutional layers after feature extraction of the input SAR image, and directly estimate the displacement field from the extracted image features;

[0007] A13: DFT layer, used to further process the displacement field in the frequency domain to remove high-frequency noise in the displacement field and make the displacement field smoother;

[0008] A14: Frequency domain optimization layer, used to further optimize the frequency domain field, while making the entire process differentiable for effective back propagation;

[0009] A15: IDFT layer, used to convert the frequency domain field processed by the frequency domain optimization layer into a displacement field;

[0010] A16: Displacement field optimization layer, used to refine the displacement field to prevent unnatural image deformation and output the final displacement field.

[0011] Preferably, the Fourier-Net-SAR model is mainly used to obtain the displacement field of the image, and the image to be registered is transformed through the obtained displacement field to finally obtain the registration result. The core part of this model is the SMA feature extraction network, which uses the powerful feature extraction ability of the SMA feature extraction network to complete the SAR image registration task.

[0012] Preferably, the SMA feature extraction network includes:

[0013] A preprocessing module, used for preprocessing the input SAR image;

[0014] STN-Attention module, used to assist in extracting image features and reduce changes caused by different angles;

[0015] The multi-scale feature extraction module is used to fuse features extracted at different scales through depth-wise separable convolution to obtain full-resolution feature representation.

[0016] Preferably, when the preprocessing module preprocesses the input SAR image, the source image and the target image are first adjusted to an appropriate resolution, then the initial noise is reduced using a convolutional layer, and finally a Lee filter is used to further suppress the noise.

[0017] Preferably, the STN-Attention module includes a localization network, a grid generator and a sampler. The localization network includes a convolution layer, a pooling layer, a non-local attention module and a ReLU activation function. When the STN-Attention module is working, it includes the following steps:

[0018] S11: input the preprocessed image into the localization network;

[0019] S12: The localization network calculates the spatial transformation parameter θ through convolutional layers, pooling layers, non-local attention modules, and ReLU activation functions;

[0020] S13: The grid generator generates a sampling grid using the transformation parameters θ and the size of the input image;

[0021] S14: The sampler uses a grid to spatially sample the input image X to produce a feature map Y that integrates different angles.

[0022] Preferably, a non-local attention block is specifically introduced in the localization network to capture the long-range dependencies of the input feature map. The inclusion of the non-local attention block in the localization network is a significant enhancement because it allows the network to consider global context information when determining the spatial transformation parameters; wherein the input is a pre-processed feature map and the output is a feature map modulated by the non-local attention.

[0023] Preferably, the non-local attention module includes three convolutional layers and one output convolutional layer. When the non-local attention module is working, it includes the following steps:

[0024] S21: Use three convolutional layers to generate three feature maps: (θ(X)), (φ(X)), and (g(X));

[0025] S22: Calculate attention weight: (f ij =θ(X i ) T φ(X j ));

[0026] S23: Apply softmax normalization:

[0027] S24: Calculate the output feature map: (Y i =∑ j α ij g(X j ));

[0028] S25: Process (Y) with the output convolutional layer to produce the final attention modulation features.

[0029] Preferably, when the SMA feature extraction network processes the input SAR image and extracts the features of the input SAR image, the SMA feature extraction network includes the following steps:

[0030] S11: preprocessing the input SAR image to ensure that images with different resolutions can be processed efficiently, wherein the preprocessing method includes noise reduction processing and resolution adjustment;

[0031] S12: Process the preprocessed image using a spatial transformation network to reduce image changes caused by different observation angles;

[0032] S13: Used to extract and fuse multi-scale features of images that have passed through the spatial transformation network to enhance the expressiveness of features.

[0033] Preferably, the SMA feature extraction network processes the input SAR image and extracts the features of the input SAR image, thereby processing the SAR image to generate a full-resolution feature map to provide key information for the subsequent image registration step. Through such a processing flow, the unique properties of the SAR image are fully considered, thereby enhancing the performance of the model in the multi-angle image registration task.

[0034] As a preferred embodiment, the expression formula of multi-scale feature fusion is:

[0035] F=Concat(X,U(Xlow),D(Xhigh))

[0036] Among them, X is the feature map processed by the STN-Attention module, F is the feature map that integrates features of different scales, U and D represent upsampling and downsampling operations respectively, and Xlow and Xhigh are low-scale and high-scale feature maps respectively.

[0037] Preferably, the displacement field estimation layer includes:

[0038] A21: The first convolution layer has 1024 input channels and uses a 3x3 convolution kernel and a padding value of 1 to keep the spatial size of the feature map unchanged. The number of output channels is set to 512 to reduce the dimension of the feature while retaining key information.

[0039] A22: ReLU activation function, applied after the first convolution layer to introduce nonlinearity to capture complex displacement patterns;

[0040] A23: Additional convolutional layers, through three additional convolutional layers to process features, the second convolution layer reduces the number of output channels to 256, the third convolution layer reduces the number of output channels to 128, and the fourth layer further reduces it to 64. These layers continue to reduce the dimension and refine the features in preparation for displacement field estimation;

[0041] A24: Final displacement field output layer. The final displacement field output layer is a convolutional layer with two output channels. The two channels at each pixel position represent the displacement in the x and y directions respectively. This layer uses a 3x3 convolution kernel and a padding value of 1 to ensure that the output displacement field has the same spatial size as the input image.

[0042] Preferably, during the forward propagation of the network, the feature map first passes through the feature extraction layer and then is input into the displacement field estimation layer; the output of the displacement field estimation layer is a full-resolution displacement field, represented by a displacement vector for each pixel position; the displacement field estimation layer enables the network to accurately predict the displacement of each pixel in the image, which is crucial for achieving high-precision image registration; by using a series of convolutional layers, the network can extract the necessary displacement information from complex image features while maintaining computational efficiency.

[0043] Preferably, the definition formula of the frequency domain field in the DFT layer is: Bφ=F(G(SMA(IM, IF))); wherein IM represents the image to be registered, IF represents the reference image, F represents the DFT layer, φ represents the full-resolution spatial displacement field, G represents the displacement field estimation layer, SMA represents the SMA feature extraction network, and Bφ represents the frequency domain field.

[0044] Preferably, the positioning formula of the full-resolution spatial displacement field in the DFT layer is:

[0045]

[0046] where the size of the full-resolution spatial displacement field is H×W, i∈[0,H-1] and j∈[0,W-1] are discrete indices in the spatial domain, while k∈[0,H-1] and l∈[0,W-1] are discrete indices in the frequency domain.

[0047] Preferably, the displacement field can be further processed in the frequency domain, for example, only the signal within a specific frequency range is retained; this is very effective for removing high-frequency noise in the displacement field, especially when processing high-noise data such as SAR images. By removing high-frequency components, the displacement field can be made smoother, thereby reducing artifacts and unnatural deformations in the registered image; and the Fourier transform helps to handle nonlinear and complex deformation patterns, which may be difficult to achieve in the direct spatial domain; therefore, in order to bridge the domain gap between real-valued spatial images and the frequency domain without increasing complexity, a discrete Fourier transform (DFT) layer is a simple and effective method to generate the frequency domain without requiring the network itself to process complex values.

[0048] Preferably, the frequency domain optimization layer optimizes the frequency domain field using convolution and filtering operations of a convolutional neural network; wherein, when optimizing the frequency domain field using convolution and filtering operations of a convolutional neural network, high-frequency noise features in the frequency domain are particularly infused and low-frequency components representing the main structural information of the image are retained.

[0049] Preferably, the frequency domain optimization layer optimizes the frequency domain field through the convolution and filtering operations of CNN, which pays special attention to the high-frequency noise characteristics in the frequency domain and retains the low-frequency components, which represent the main structural information of the image; retaining this information during the registration process helps to improve the registration accuracy, especially in the case of complex structures such as SAR images, which may undergo diverse changes due to various reasons (such as different imaging angles). Processing the displacement field in the frequency domain can improve the robustness of the registration process and enable it to adapt to these changes.

[0050] Preferably, the composition structure of the frequency domain optimization layer includes:

[0051] A31: frequency domain optimization input layer, used to input the frequency domain field after being processed by the DFT layer. The input channel of the input layer is 2, and the input channels of the two frequency domain optimization input layers correspond to the real part and imaginary part of the frequency domain data respectively.

[0052] A32: The first point-by-point convolution layer has 2 input channels and 64 output channels, and uses a 1x1 convolution kernel for processing. The step size of the first point-by-point convolution layer is 1 to maintain the spatial resolution, and same padding is used to keep the input and output sizes consistent. After the first point-by-point convolution layer, batch normalization and ReLU activation operations are performed to further adjust and refine the frequency domain features.

[0053] A33: The second frequency domain specific operation layer is used to suppress the high-frequency noise in the input frequency domain field data after Fourier transformation using a Gaussian low-pass filter;

[0054] A34: The third depth-wise separable convolutional layer has 64 input channels and 64 output channels, and uses a 3x3 convolution kernel for processing, with a stride of 1 to maintain spatial resolution, and same padding to keep the input and output sizes consistent. After the third depth-wise separable convolutional layer, batch normalization and ReLU activation operations are performed to provide efficient feature extraction while maintaining computational efficiency.

[0055] A35: Output convolution layer, the input channel of this layer is 64, the output channel is 2, and it uses a convolution kernel of size 3x3 for processing, with a stride of 1 to maintain spatial resolution, and same padding is used to keep the input and output sizes consistent.

[0056] Preferably, the Gaussian low-pass filter smoothes the high-frequency components in the frequency domain in a unique way while retaining the low-frequency components, thereby denoising and enhancing the frequency domain field; the mathematical definition of the Gaussian low-pass filter is as follows:

[0057]

[0058] Among them, u and v represent the horizontal and vertical frequency coordinates in the frequency domain, respectively. σ is the standard deviation of the Gaussian kernel and is set to 2, which controls the bandwidth of the filter. When implementing a Gaussian low-pass filter, you first need to construct a Gaussian kernel of the same size as the input frequency domain field. This kernel is generated according to the above formula, and its center is aligned to the center frequency of the frequency domain field. Then, multiplying this Gaussian kernel with the input frequency domain field can achieve frequency domain filtering:

[0059] F filtered (u, v) = F (u, v) G (u, v)

[0060] Among them, F(u, v) represents the frequency domain field of the input, and Ffiltered(u, v) is the frequency domain field after filtering; the key to this frequency domain filtering process is to maintain the differentiability of the operation so that backpropagation can be performed in model training; in this way, the Gaussian low-pass filter in the frequency domain optimization layer not only effectively reduces the high-frequency noise in the frequency domain, but also due to its differentiability, it can be integrated with the deep learning model to achieve end-to-end training and optimization; it should be noted that the 64-channel output after point-by-point convolution means that for each pixel position, there are now 64 different eigenvalues, so when performing frequency domain operations (Gaussian low-pass filtering), these operations need to be performed independently on each channel.

[0061] As a preference, the principle formula of the IDFT layer is:

[0062]

[0063] Where H and W represent the height and width of the image, respectively, φ represents the full-resolution spatial displacement field, and [F(φ)]k,l is the representation in the frequency domain.

[0064] Preferably, the composition structure of the displacement field optimization layer includes:

[0065] A41: Displacement field optimization input layer, used to input the spatial displacement field output after IDFT layer processing. The input channel of the displacement field optimization input layer is 2;

[0066] A42: The first convolution layer of the displacement field optimization has 2 input channels and 64 output channels. It uses a 3x3 convolution kernel with a stride of 1 to maintain spatial resolution and same padding to keep the input and output sizes consistent. Batch normalization and ReLU activation operations are performed after the first convolution layer of the displacement field optimization to extract high-level features of the displacement field.

[0067] A43: Displacement field optimization The second displacement field smoothing operation layer is used to smooth the displacement field using a bilateral filter to reduce discontinuities and unnatural deformations;

[0068] A44: Displacement field optimized output convolution layer. The input channel of the displacement field optimized output convolution layer is 64, the output channel is 2, and a convolution kernel of size 3x3 is used for processing. The stride is 1 to maintain the spatial resolution, and same padding is used to keep the input and output sizes consistent.

[0069] Preferably, the spatial displacement field optimization layer is designed to refine and smooth the displacement field after IDFT, reducing unnatural deformations that may occur during image registration. By adding convolutional layers and smoothing operations, this layer helps to enhance the continuity and naturalness of the displacement field, thereby improving the quality of the final registered image and outputting the final spatial displacement field. Finally, the image to be registered uses the displacement field to complete the registration task. At the same time, in the training stage, gradient regularization is achieved by calculating the gradient of the displacement field (i.e., the difference between the displacement of each pixel and the displacement of its neighborhood) and summing these gradients to ensure that the displacement field is smooth, which helps to further avoid unnatural image deformations. In addition, the result image of the image to be registered after the displacement field is also similar to the reference image to generate a Loss value, which is back-propagated back to the encoder part for parameter optimization operations.

[0070] Beneficial effects of the present invention:

[0071] 1. Compared with the existing technologies that use SIFI feature-based methods, deep neural network-based methods, MU-Net multi-scale framework methods, shape context methods, methods for directly matching multi-angle SAR images, and methods based on semantic distribution positions to solve the registration problem of multi-angle SAR images, the registration performance and robustness are not enough to meet the needs when the surface data is scarce and the scene is complex; this registration method sets up an SMA feature extraction network and adopts a variety of technical modules, which cover preprocessing strategies, STN-Attention, and multi-scale feature fusion technology. These technical modules are combined to provide a comprehensive and efficient solution for SAR image registration;

[0072] 2. The Fourier-Net-SAR model is mainly used to obtain the displacement field of the image. The image to be registered is transformed through the obtained displacement field to finally obtain the registration result. The core part of this model is the SMA feature extraction network, which uses the powerful feature extraction ability of the SMA feature extraction network to complete the SAR image registration task;

[0073] 3. During the forward propagation of the network, the feature map first passes through the feature extraction layer and then is input into the displacement field estimation layer; the output of the displacement field estimation layer is a full-resolution displacement field, represented as a displacement vector at each pixel position; the displacement field estimation layer enables the network to accurately predict the displacement of each pixel in the image, which is essential for achieving high-precision image registration; by using a series of convolutional layers, the network is able to extract the necessary displacement information from complex image features while maintaining computational efficiency;

[0074] 4. The displacement field can be further processed in the frequency domain, for example, only retaining signals within a specific frequency range; this is very effective for removing high-frequency noise in the displacement field, especially when processing high-noise data such as SAR images. By removing high-frequency components, the displacement field can be made smoother, thereby reducing artifacts and unnatural deformations that appear in the registered image; and the Fourier transform helps to deal with nonlinear and complex deformation patterns, which may be difficult to achieve in the direct spatial domain; therefore, in order to bridge the domain gap between real-valued spatial images and the frequency domain without increasing complexity, a discrete Fourier transform (DFT) layer is a simple and effective method to generate the frequency domain without requiring the network itself to process complex values. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 What is shown is a schematic diagram of the construction of the multi-angle registration model based on Fourier-Net-SAR of the present invention;

[0076] Figure 2What is shown is a schematic diagram of the construction of the SMA feature extraction network in the multi-angle registration model based on Fourier-Net-SAR of the present invention;

[0077] Figure 3 What is shown is a schematic diagram of the workflow of the STN-Attention module in the multi-angle registration model based on Fourier-Net-SAR of the present invention. DETAILED DESCRIPTION

[0078] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0079] See also Figure 1 The present invention provides an embodiment: a multi-angle registration model based on Fourier-Net-SAR, comprising:

[0080] A11: SMA feature extraction network, used to process the input SAR image and extract the features of the input SAR image. When extracting the features of the input SAR image, the characteristics of the input SAR image, such as the influence of noise and angle, are considered.

[0081] A12: Displacement field estimation layer, which is used to estimate the displacement field by adding a set of convolutional layers after feature extraction of the input SAR image, and directly estimate the displacement field from the extracted image features;

[0082] A13: DFT layer, used to further process the displacement field in the frequency domain to remove high-frequency noise in the displacement field and make the displacement field smoother;

[0083] A14: Frequency domain optimization layer, used to further optimize the frequency domain field, while making the entire process differentiable for effective back propagation;

[0084] A15: IDFT layer, used to convert the frequency domain field processed by the frequency domain optimization layer into a displacement field;

[0085] A16: Displacement field optimization layer, used to refine the displacement field to prevent unnatural image deformation and output the final displacement field.

[0086] Preferably, the Fourier-Net-SAR model is mainly used to obtain the displacement field of the image, and the image to be registered is transformed through the obtained displacement field to finally obtain the registration result. The core part of this model is the SMA feature extraction network, which uses the powerful feature extraction ability of the SMA feature extraction network to complete the SAR image registration task.

[0087] See also Figure 2-3 In this embodiment, the SMA feature extraction network includes:

[0088] A preprocessing module, used for preprocessing the input SAR image;

[0089] STN-Attention module, used to assist in extracting image features and reduce changes caused by different angles;

[0090] The multi-scale feature extraction module is used to fuse features extracted at different scales through depth-wise separable convolution to obtain full-resolution feature representation.

[0091] Preferably, when the preprocessing module preprocesses the input SAR image, the source image and the target image are first adjusted to an appropriate resolution, then the initial noise is reduced using a convolutional layer, and finally a Lee filter is used to further suppress the noise.

[0092] Preferably, the STN-Attention module includes a localization network, a grid generator and a sampler. The localization network includes a convolution layer, a pooling layer, a non-local attention module and a ReLU activation function. When the STN-Attention module is working, it includes the following steps:

[0093] S11: input the preprocessed image into the localization network;

[0094] S12: The localization network calculates the spatial transformation parameter θ through convolutional layers, pooling layers, non-local attention modules, and ReLU activation functions;

[0095] S13: The grid generator generates a sampling grid using the transformation parameters θ and the size of the input image;

[0096] S14: The sampler uses a grid to spatially sample the input image X to produce a feature map Y that integrates different angles.

[0097] Preferably, a non-local attention block is specifically introduced in the localization network to capture the long-range dependencies of the input feature map. The inclusion of the non-local attention block in the localization network is a significant enhancement because it allows the network to consider global context information when determining the spatial transformation parameters; wherein the input is a pre-processed feature map and the output is a feature map modulated by the non-local attention.

[0098] Preferably, the non-local attention module includes three convolutional layers and one output convolutional layer. When the non-local attention module is working, it includes the following steps:

[0099] S21: Use three convolutional layers to generate three feature maps: (θ(X)), (φ(X)), and (g(X));

[0100] S22: Calculate attention weight: (fij =θ(X i ) T φ(X j ));

[0101] S23: Apply softmax normalization:

[0102] S24: Calculate the output feature map: (Y i =∑ j α ij g(X j ));

[0103] S25: Process (Y) with the output convolutional layer to produce the final attention modulation features.

[0104] Preferably, when the SMA feature extraction network processes the input SAR image and extracts the features of the input SAR image, the SMA feature extraction network includes the following steps:

[0105] S11: preprocessing the input SAR image to ensure that images with different resolutions can be processed efficiently, wherein the preprocessing method includes noise reduction processing and resolution adjustment;

[0106] S12: Process the preprocessed image using a spatial transformation network to reduce image changes caused by different observation angles;

[0107] S13: Used to extract and fuse multi-scale features of images that have passed through the spatial transformation network to enhance the expressiveness of features.

[0108] Preferably, the SMA feature extraction network processes the input SAR image and extracts the features of the input SAR image, thereby processing the SAR image to generate a full-resolution feature map to provide key information for the subsequent image registration step. Through such a processing flow, the unique properties of the SAR image are fully considered, thereby enhancing the performance of the model in the multi-angle image registration task.

[0109] As a preferred embodiment, the expression formula of multi-scale feature fusion is:

[0110] F=Concat(X,U(Xlow),D(Xhigh))

[0111] Among them, X is the feature map processed by the STN-Attention module, F is the feature map that integrates features of different scales, U and D represent upsampling and downsampling operations respectively, and Xlow and Xhigh are low-scale and high-scale feature maps respectively.

[0112] Preferably, the displacement field estimation layer includes:

[0113] A21: The first convolution layer has 1024 input channels and uses a 3x3 convolution kernel and a padding value of 1 to keep the spatial size of the feature map unchanged. The number of output channels is set to 512 to reduce the dimension of the feature while retaining key information.

[0114] A22: ReLU activation function, applied after the first convolution layer to introduce nonlinearity to capture complex displacement patterns;

[0115] A23: Additional convolutional layers, through three additional convolutional layers to process features, the second convolution layer reduces the number of output channels to 256, the third convolution layer reduces the number of output channels to 128, and the fourth layer further reduces it to 64. These layers continue to reduce the dimension and refine the features in preparation for displacement field estimation;

[0116] A24: Final displacement field output layer. The final displacement field output layer is a convolutional layer with two output channels. The two channels at each pixel position represent the displacement in the x and y directions respectively. This layer uses a 3x3 convolution kernel and a padding value of 1 to ensure that the output displacement field has the same spatial size as the input image.

[0117] Preferably, during the forward propagation of the network, the feature map first passes through the feature extraction layer and then is input into the displacement field estimation layer; the output of the displacement field estimation layer is a full-resolution displacement field, represented by a displacement vector for each pixel position; the displacement field estimation layer enables the network to accurately predict the displacement of each pixel in the image, which is crucial for achieving high-precision image registration; by using a series of convolutional layers, the network can extract the necessary displacement information from complex image features while maintaining computational efficiency.

[0118] Preferably, the definition formula of the frequency domain field in the DFT layer is: Bφ=F(G(SMA(IM, IF))); wherein IM represents the image to be registered, IF represents the reference image, F represents the DFT layer, φ represents the full-resolution spatial displacement field, G represents the displacement field estimation layer, SMA represents the SMA feature extraction network, and Bφ represents the frequency domain field.

[0119] Preferably, the positioning formula of the full-resolution spatial displacement field in the DFT layer is:

[0120]

[0121] where the size of the full-resolution spatial displacement field is H×W, i∈[0,H-1] and j∈[0,W-1] are discrete indices in the spatial domain, while k∈[0,H-1] and l∈[0,W-1] are discrete indices in the frequency domain.

[0122] Preferably, the displacement field can be further processed in the frequency domain, for example, only the signal within a specific frequency range is retained; this is very effective for removing high-frequency noise in the displacement field, especially when processing high-noise data such as SAR images. By removing high-frequency components, the displacement field can be made smoother, thereby reducing artifacts and unnatural deformations in the registered image; and the Fourier transform helps to handle nonlinear and complex deformation patterns, which may be difficult to achieve in the direct spatial domain; therefore, in order to bridge the domain gap between real-valued spatial images and the frequency domain without increasing complexity, a discrete Fourier transform (DFT) layer is a simple and effective method to generate the frequency domain without requiring the network itself to process complex values.

[0123] Preferably, the frequency domain optimization layer optimizes the frequency domain field using convolution and filtering operations of a convolutional neural network; wherein, when optimizing the frequency domain field using convolution and filtering operations of a convolutional neural network, high-frequency noise features in the frequency domain are particularly infused and low-frequency components representing the main structural information of the image are retained.

[0124] Preferably, the frequency domain optimization layer optimizes the frequency domain field through the convolution and filtering operations of CNN, which pays special attention to the high-frequency noise characteristics in the frequency domain and retains the low-frequency components, which represent the main structural information of the image; retaining this information during the registration process helps to improve the registration accuracy, especially in the case of complex structures such as SAR images, which may undergo diverse changes due to various reasons (such as different imaging angles). Processing the displacement field in the frequency domain can improve the robustness of the registration process and enable it to adapt to these changes.

[0125] Preferably, the composition structure of the frequency domain optimization layer includes:

[0126] A31: frequency domain optimization input layer, used to input the frequency domain field after being processed by the DFT layer. The input channel of the input layer is 2, and the input channels of the two frequency domain optimization input layers correspond to the real part and imaginary part of the frequency domain data respectively.

[0127] A32: The first point-by-point convolution layer has 2 input channels and 64 output channels, and uses a 1x1 convolution kernel for processing. The step size of the first point-by-point convolution layer is 1 to maintain the spatial resolution, and same padding is used to keep the input and output sizes consistent. After the first point-by-point convolution layer, batch normalization and ReLU activation operations are performed to further adjust and refine the frequency domain features.

[0128] A33: The second frequency domain specific operation layer is used to suppress the high-frequency noise in the input frequency domain field data after Fourier transformation using a Gaussian low-pass filter;

[0129] A34: The third depth-wise separable convolutional layer has 64 input channels and 64 output channels, and uses a 3x3 convolution kernel for processing, with a stride of 1 to maintain spatial resolution, and same padding to keep the input and output sizes consistent. After the third depth-wise separable convolutional layer, batch normalization and ReLU activation operations are performed to provide efficient feature extraction while maintaining computational efficiency.

[0130] A35: Output convolution layer, the input channel of this layer is 64, the output channel is 2, and it uses a convolution kernel of size 3x3 for processing, with a stride of 1 to maintain spatial resolution, and same padding is used to keep the input and output sizes consistent.

[0131] Preferably, the Gaussian low-pass filter smoothes the high-frequency components in the frequency domain in a unique way while retaining the low-frequency components, thereby denoising and enhancing the frequency domain field; the mathematical definition of the Gaussian low-pass filter is as follows:

[0132]

[0133] Among them, u and v represent the horizontal and vertical frequency coordinates in the frequency domain, respectively. σ is the standard deviation of the Gaussian kernel and is set to 2, which controls the bandwidth of the filter. When implementing a Gaussian low-pass filter, you first need to construct a Gaussian kernel of the same size as the input frequency domain field. This kernel is generated according to the above formula, and its center is aligned to the center frequency of the frequency domain field. Then, multiplying this Gaussian kernel with the input frequency domain field can achieve frequency domain filtering:

[0134] F filtered (u, v) = F (u, v) G (u, v)

[0135] Among them, F(u, v) represents the frequency domain field of the input, and Ffiltered(u, v) is the frequency domain field after filtering; the key to this frequency domain filtering process is to maintain the differentiability of the operation so that backpropagation can be performed in model training; in this way, the Gaussian low-pass filter in the frequency domain optimization layer not only effectively reduces the high-frequency noise in the frequency domain, but also due to its differentiability, it can be integrated with the deep learning model to achieve end-to-end training and optimization; it should be noted that the 64-channel output after point-by-point convolution means that for each pixel position, there are now 64 different eigenvalues, so when performing frequency domain operations (Gaussian low-pass filtering), these operations need to be performed independently on each channel.

[0136] As a preference, the principle formula of the IDFT layer is:

[0137]

[0138] Where H and W represent the height and width of the image, respectively, φ represents the full-resolution spatial displacement field, and [F(φ)]k,l is the representation in the frequency domain.

[0139] Preferably, the composition structure of the displacement field optimization layer includes:

[0140] A41: Displacement field optimization input layer, used to input the spatial displacement field output after IDFT layer processing. The input channel of the displacement field optimization input layer is 2;

[0141] A42: The first convolution layer of the displacement field optimization has 2 input channels and 64 output channels. It uses a 3x3 convolution kernel with a stride of 1 to maintain spatial resolution and same padding to keep the input and output sizes consistent. Batch normalization and ReLU activation operations are performed after the first convolution layer of the displacement field optimization to extract high-level features of the displacement field.

[0142] A43: Displacement field optimization The second displacement field smoothing operation layer is used to smooth the displacement field using a bilateral filter to reduce discontinuities and unnatural deformations;

[0143] A44: Displacement field optimized output convolution layer. The input channel of the displacement field optimized output convolution layer is 64, the output channel is 2, and a convolution kernel of size 3x3 is used for processing. The stride is 1 to maintain the spatial resolution, and same padding is used to keep the input and output sizes consistent.

[0144] Preferably, the spatial displacement field optimization layer is designed to refine and smooth the displacement field after IDFT, reducing unnatural deformations that may occur during image registration. By adding convolutional layers and smoothing operations, this layer helps to enhance the continuity and naturalness of the displacement field, thereby improving the quality of the final registered image and outputting the final spatial displacement field. Finally, the image to be registered uses the displacement field to complete the registration task. At the same time, in the training stage, gradient regularization is achieved by calculating the gradient of the displacement field (i.e., the difference between the displacement of each pixel and the displacement of its neighborhood) and summing these gradients to ensure that the displacement field is smooth, which helps to further avoid unnatural image deformations, and the result image obtained by the displacement field of the image to be registered will also perform similarity calculations with the reference image to generate Loss values, and back-propagate back to the encoder part for parameter optimization operations.

[0145] When working, first of all, the processing of the original SAR image includes resolution adjustment and noise suppression to reduce the interference of noise on subsequent feature extraction; the specific operation is as follows: the input original SAR image is subjected to a series of processing steps, including convolution layer for preliminary denoising and Lee filter for further denoising, and finally the image processed by denoising and Lee filtering is output; the purpose of this process is to reduce the noise level in the image while maintaining the same image resolution, so as to improve the accuracy of subsequent feature extraction;

[0146] Next, the processed image passes through the STN-Attention module, whose main function is to assist in extracting image features and reduce changes caused by different angles. In STN-Attention, non-local attention blocks are also introduced to better capture long-distance dependencies in the image, thereby enhancing the performance of STN-Attention in processing complex geometric transformations. The key to this step is to normalize and align images through STN-Attention so that the network can better understand the spatial relationship between images and reduce deformation caused by angle changes.

[0147] After STN-Attention processing, deep SAR feature extraction is performed to further extract useful information from the image. This process includes multi-scale feature fusion such as multi-layer convolution, pooling structure, and up- and down-sampling. The core of multi-scale feature fusion is to fuse the features extracted at different scales through deep separable convolution to obtain full-resolution feature representation. This fusion method ensures that the network can obtain information at different scales, thereby improving the diversity and expression ability of features.

[0148] Through the above steps, the entire image processing pipeline includes steps such as resolution adjustment, noise suppression, STN-Attention assisted feature extraction, and multi-scale feature fusion, which is designed to deal with the specific challenges of SAR images to achieve better registration performance.

[0149] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of those skilled in the art without departing from the spirit of the present invention.

Claims

1. A multi-angle registration model based on Fourier-Net-SAR; characterized by: Included are: A11: SMA feature extraction network, used to process the input SAR image and extract the features of the input SAR image. When extracting the features of the input SAR image, the characteristics of the input SAR image, such as the influence of noise and angle, are considered. A12: Displacement field estimation layer, which is used to estimate the displacement field by adding a set of convolutional layers after feature extraction of the input SAR image, and directly estimate the displacement field from the extracted image features; A13: DFT layer, used to further process the displacement field in the frequency domain to remove high-frequency noise in the displacement field and make the displacement field smoother; A14: Frequency domain optimization layer, used to further optimize the frequency domain field, while making the entire process differentiable for effective back propagation; A15: IDFT layer, used to convert the frequency domain field processed by the frequency domain optimization layer into a displacement field; A16: Displacement field optimization layer, used to refine the displacement field to prevent unnatural image deformation and output the final displacement field.

2. The multi-angle registration model based on Fourier-Net-SAR according to claim 1, characterized in that: The SMA feature extraction network includes: A preprocessing module, used for preprocessing the input SAR image; STN-Attention module, used to assist in extracting image features and reduce changes caused by different angles; The multi-scale feature extraction module is used to fuse features extracted at different scales through depth-wise separable convolution to obtain full-resolution feature representation.

3. The multi-angle registration model based on Fourier-Net-SAR according to claim 2, characterized in that: The STN-Attention module consists of a localization network, a grid generator, and a sampler. The localization network includes a convolutional layer, a pooling layer, a non-local attention module, and a ReLU activation function. When the STN-Attention module is working, it includes the following steps: S11: input the preprocessed image into the localization network; S12: The localization network calculates the spatial transformation parameter θ through convolutional layers, pooling layers, non-local attention modules, and ReLU activation functions; S13: The grid generator generates a sampling grid using the transformation parameters θ and the size of the input image; S14: The sampler uses a grid to spatially sample the input image X to produce a feature map Y that integrates different angles.

4. The multi-angle registration model based on Fourier-Net-SAR according to claim 3, characterized in that: The non-local attention module includes three convolutional layers and one output convolutional layer. When the non-local attention module works, it includes the following steps: S21: Use three convolutional layers to generate three feature maps: (θ(X)), (φ(X)), and (g(X)); S22: Calculate attention weights: S23: Apply softmax normalization: S24: Calculate the output feature map: (Y i =∑ j α ij g(X j )); S25: Process (Y) with the output convolutional layer to produce the final attention modulation features.

5. The multi-angle registration model based on Fourier-Net-SAR according to claim 4, characterized in that: The displacement field estimation layer includes: A21: The first convolution layer has 1024 input channels and uses a 3x3 convolution kernel and a padding value of 1 to keep the spatial size of the feature map unchanged. The number of output channels is set to 512 to reduce the dimension of the feature while retaining key information. A22: ReLU activation function, applied after the first convolution layer to introduce nonlinearity to capture complex displacement patterns; A23: Additional convolutional layers, through three additional convolutional layers to process features, the second convolution layer reduces the number of output channels to 256, the third convolution layer reduces the number of output channels to 128, and the fourth layer further reduces it to 64. These layers continue to reduce the dimension and refine the features in preparation for displacement field estimation; A24: Final displacement field output layer. The final displacement field output layer is a convolutional layer with two output channels. The two channels at each pixel position represent the displacement in the x and y directions respectively. This layer uses a 3x3 convolution kernel and a padding value of 1 to ensure that the output displacement field has the same spatial size as the input image.

6. The multi-angle registration model based on Fourier-Net-SAR according to claim 5, characterized in that: The definition formula of the frequency domain field in the DFT layer is: Bφ=F(G(SMA(IM, IF))); where IM represents the image to be registered, IF represents the reference image, F represents the DFT layer, φ represents the full-resolution spatial displacement field, G represents the displacement field estimation layer, SMA represents the SMA feature extraction network, and Bφ represents the frequency domain field; the positioning formula of the full-resolution spatial displacement field in the DFT layer is: where the size of the full-resolution spatial displacement field is H×W, i∈[0,H-1] and j∈[0,W-1] are discrete indices in the spatial domain, while k∈[0,H-1] and l∈[0,W-1] are discrete indices in the frequency domain.

7. The multi-angle registration model based on Fourier-Net-SAR according to claim 6, characterized in that: The frequency domain optimization layer optimizes the frequency domain field using the convolution and filtering operations of the convolutional neural network; wherein, when optimizing the frequency domain field using the convolution and filtering operations of the convolutional neural network, the high-frequency noise features in the frequency domain are particularly infused and the low-frequency components representing the main structural information of the image are retained.

8. The multi-angle registration model based on Fourier-Net-SAR according to claim 7, characterized in that: The composition structure of the frequency domain optimization layer includes: A31: frequency domain optimization input layer, used to input the frequency domain field after being processed by the DFT layer. The input channel of the input layer is 2, and the input channels of the two frequency domain optimization input layers correspond to the real part and imaginary part of the frequency domain data respectively. A32: The first point-by-point convolution layer has 2 input channels and 64 output channels, and uses a 1x1 convolution kernel for processing. The step size of the first point-by-point convolution layer is 1 to maintain the spatial resolution, and same padding is used to keep the input and output sizes consistent. After the first point-by-point convolution layer, batch normalization and ReLU activation operations are performed to further adjust and refine the frequency domain features. A33: The second frequency domain specific operation layer is used to suppress the high-frequency noise in the input frequency domain field data after Fourier transformation using a Gaussian low-pass filter; A34: The third depth-wise separable convolutional layer has 64 input channels and 64 output channels, and uses a 3x3 convolution kernel for processing, with a stride of 1 to maintain spatial resolution, and same padding to keep the input and output sizes consistent. After the third depth-wise separable convolutional layer, batch normalization and ReLU activation operations are performed to provide efficient feature extraction while maintaining computational efficiency. A35: Output convolution layer, the input channel of this layer is 64, the output channel is 2, and it uses a convolution kernel of size 3x3 for processing, with a stride of 1 to maintain spatial resolution, and same padding is used to keep the input and output sizes consistent.

9. The multi-angle registration model based on Fourier-Net-SAR according to claim 8, characterized in that: The principle formula of the IDFT layer is: Where H and W represent the height and width of the image, respectively, φ represents the full-resolution spatial displacement field, and [F(φ)]k,l is the representation in the frequency domain.

10. The multi-angle registration model based on Fourier-Net-SAR according to claim 9, characterized in that: The composition structure of the displacement field optimization layer includes: A41: Displacement field optimization input layer, used to input the spatial displacement field output after IDFT layer processing. The input channel of the displacement field optimization input layer is 2; A42: The first convolution layer of the displacement field optimization has 2 input channels and 64 output channels. It uses a 3x3 convolution kernel with a stride of 1 to maintain spatial resolution and same padding to keep the input and output sizes consistent. Batch normalization and ReLU activation operations are performed after the first convolution layer of the displacement field optimization to extract high-level features of the displacement field. A43: Displacement field optimization The second displacement field smoothing operation layer is used to smooth the displacement field using a bilateral filter to reduce discontinuities and unnatural deformations; A44: Displacement field optimized output convolution layer. The input channel of the displacement field optimized output convolution layer is 64, the output channel is 2, and a convolution kernel of size 3x3 is used for processing. The stride is 1 to maintain the spatial resolution, and same padding is used to keep the input and output sizes consistent.

Images