A multi-angle registration model based on Fourier-Net-SAR

Abstract

The present invention relates to the field of image processing, and in particular to a multi-angle registration model based on Fourier-Net-SAR. The technical problem is that a method based on SIFI features, a method based on deep neural networks, a MU-Net multi-scale framework method, a shape context method, a method for directly matching multi-angle SAR images, and a method based on semantic distribution positions are used to solve the registration problem of multi-angle SAR images. In the case of sparse surface data and complex scenes, the registration performance and robustness are insufficient to meet the needs. The technical solution is a multi-angle registration model based on Fourier-Net-SAR, comprising an SMA feature extraction network, a displacement field estimation layer, a DFT layer, a frequency domain optimization layer, an IDFT layer, and a displacement field optimization layer. The present invention sets an SMA feature extraction network and adopts multiple technical modules, which cover preprocessing strategies, STN-Attention, and multi-scale feature fusion technologies. These technical modules are combined to provide a comprehensive and efficient solution for SAR image registration.

Description

Technical Field

[0001] The present invention relates to the field of image registration processing, and in particular to a multi-angle registration model based on Fourier-Net-SAR. Background Art

[0002] SAR image registration is a classic problem and technical challenge in the field of SAR image processing. It aims to achieve accurate spatial alignment of two or more images. Specifically, for two images in a set of image datasets, it maps one image to the other by finding a spatial transformation method so that the corresponding spatial positions in the two images correspond one to one to achieve information fusion. This is because these images may come from different devices, different times or different perspectives. Therefore, image registration must be performed before many image analysis tasks (such as image fusion, change detection, target recognition and agricultural monitoring). Therefore, SAR image registration plays a vital and indispensable role in image analysis. Only by accurately aligning images acquired at different times or different angles can subsequent analysis and application be carried out. However, due to the different imaging times and ground observation positions of SAR images, there may be differences in radiation information and geometric distortion in the same scene, which further increases the complexity of SAR image registration. With the rapid development of deep neural network technology, deep learning-based image registration has become a hot topic. Compared with traditional methods, SAR registration methods based on deep learning can better capture image features and construct more refined spatial mapping relationships between images. Through the feature representation of deep learning, these registration methods can mine more abstract and discriminative features from SAR images, making the common features between SAR images more obvious, thereby achieving more accurate image matching. This deep learning-based method not only performs well in cases with limited samples or complex scenes, but also shows excellent performance in large-scale data and diverse scenes. Methods based on SIFI features, deep neural networks, MU-Net multi-scale framework, shape context, direct matching of multi-angle SAR images, and semantic distribution position-based methods are used to solve the registration problem of multi-angle SAR images. However, when the surface data is scarce and the scenes are complex, the registration performance and robustness are not enough to meet the needs. Summary of the Invention

[0003] In order to overcome the problem of multi-angle SAR image registration solved by using SIFI feature-based methods, deep neural network-based methods, MU-Net multi-scale framework methods, shape context methods, direct matching of multi-angle SAR images and semantic distribution position-based methods, the registration performance and robustness are not enough to meet the needs when the surface data is scarce and the scenes are complex.

[0004] The technical solution of the present invention is: a multi-angle registration model based on Fourier-Net-SAR, comprising:

[0005] A11: SMA feature extraction network, used to process the input SAR image and extract features from the input SAR image. When extracting features from the input SAR image, the characteristics of the input SAR image, such as the influence of noise and angle, are considered. These include:

[0006] A preprocessing module is used to preprocess the input SAR image;

[0007] The STN-Attention module is used to assist in extracting image features and reduce changes caused by different angles;

[0008] Multi-scale feature extraction module, which is used to fuse features extracted at different scales through depth-wise separable convolution to obtain full-resolution feature representation;

[0009] A12: Displacement field estimation layer, which is used to estimate the displacement field by adding a set of convolutional layers after feature extraction of the input SAR image. The displacement field is estimated directly from the extracted image features, including:

[0010] A21: The first convolution layer has 1024 input channels and uses a 3x3 convolution kernel with a padding value of 1 to keep the spatial size of the feature map unchanged. The number of output channels is set to 512 to reduce the feature dimension while retaining key information.

[0011] A22: ReLU activation function, applied after the first convolution layer to introduce nonlinearity to capture complex displacement patterns;

[0012] A23: Additional convolutional layers, which process features through three additional convolutional layers. The number of output channels of the second convolution layer is reduced to 256, the number of output channels of the third convolution layer is reduced to 128, and the fourth layer is further reduced to 64. These layers continue to reduce dimensionality and refine features in preparation for displacement field estimation;

[0013] A24: Final displacement field output layer. The final displacement field output layer is a convolutional layer with two output channels. The two channels at each pixel position represent the displacement in the x and y directions respectively. This layer uses a 3x3 convolution kernel and a padding value of 1 to ensure that the output displacement field has the same spatial size as the input image.

[0014] A13: DFT layer, used to further process the displacement field in the frequency domain to remove high-frequency noise in the displacement field and make the displacement field smoother;

[0015] A14: Frequency domain optimization layer, used to further optimize the frequency domain field. It can be differentiated when running the entire process to facilitate effective backpropagation, including:

[0016] A31: Frequency domain optimization input layer, used to input the frequency domain field processed by the DFT layer. The input channel of the input layer is 2, and the input channels of the two frequency domain optimization input layers correspond to the real part and imaginary part of the frequency domain data respectively.

[0017] A32: The first point-by-point convolution layer has 2 input channels and 64 output channels, and uses a 1x1 convolution kernel for processing. The stride of the first point-by-point convolution layer is 1 to maintain spatial resolution, and same padding is used to keep the input and output sizes consistent. After the first point-by-point convolution layer, batch normalization and ReLU activation operations are performed to further adjust and refine the frequency domain features.

[0018] A33: The second frequency domain specific operation layer is used to suppress the high-frequency noise in the input frequency domain field data after Fourier transformation using a Gaussian low-pass filter;

[0019] A34: The third depth-wise separable convolution layer has 64 input channels and 64 output channels. It uses a 3x3 convolution kernel for processing, with a stride of 1 to maintain spatial resolution and same padding to keep the input and output sizes consistent. Batch normalization and ReLU activation operations are performed after the third depth-wise separable convolution layer to provide efficient feature extraction while maintaining computational efficiency.

[0020] A35: Output convolution layer, which has 64 input channels and 2 output channels, and uses a 3x3 convolution kernel for processing, with a stride of 1 to maintain spatial resolution, and uses same padding to keep the input and output sizes consistent;

[0021] A15: IDFT layer, used to convert the frequency domain field processed by the frequency domain optimization layer into a displacement field;

[0022] A16: Displacement field optimization layer, used to refine the displacement field to prevent unnatural image deformation and output the final displacement field, including:

[0023] A41: Displacement field optimization input layer, used to input the spatial displacement field output after IDFT layer processing. The input channel of the displacement field optimization input layer is 2;

[0024] A42: Displacement field optimization of the first convolutional layer. This layer has 2 input channels and 64 output channels, and uses a 3x3 convolution kernel for processing. The stride is 1 to maintain spatial resolution, and same padding is used to keep the input and output sizes consistent. Batch normalization and ReLU activation operations are performed after the first convolutional layer of displacement field optimization to extract high-level features of the displacement field.

[0025] A43: Displacement field optimization The second displacement field smoothing operation layer is used to smooth the displacement field using a bilateral filter to reduce discontinuities and unnatural deformations.

[0026] A44: Displacement field optimized output convolution layer. The input channel of the displacement field optimized output convolution layer is 64, the output channel is 2, and a convolution kernel of size 3x3 is used for processing. The stride is 1 to maintain the spatial resolution, and the same padding is used to keep the input and output sizes consistent.

[0027] Preferably, the Fourier-Net-SAR model is mainly used to obtain the displacement field of the image. The image to be registered is transformed through the obtained displacement field to finally obtain the registration result. The core part of this model is the SMA feature extraction network, which uses the powerful feature extraction capability of the SMA feature extraction network to complete the SAR image registration task.

[0028] Preferably, when the preprocessing module preprocesses the input SAR image, the source image and the target image are first adjusted to an appropriate resolution, then the convolution layer is used to reduce the initial noise, and finally the Lee filter is used to further suppress the noise.

[0029] Preferably, during the forward propagation of the network, the feature map first passes through the feature extraction layer and then is input into the displacement field estimation layer; the output of the displacement field estimation layer is a full-resolution displacement field, represented as a displacement vector for each pixel position; the displacement field estimation layer enables the network to accurately predict the displacement of each pixel in the image, which is crucial for achieving high-precision image registration; by using a series of convolutional layers, the network can extract the necessary displacement information from complex image features while maintaining computational efficiency.

[0030] Preferably, the Gaussian low-pass filter smoothes the high-frequency components in its unique way in the frequency domain while retaining the low-frequency components, thereby denoising and enhancing the frequency domain field; the mathematical definition of the Gaussian low-pass filter is as follows:

[0031]

[0032] Where u and v represent the horizontal and vertical frequency coordinates in the frequency domain, respectively. σ is the standard deviation of the Gaussian kernel and is set to 2, which controls the bandwidth of the filter. When implementing a Gaussian low-pass filter, we first need to construct a Gaussian kernel of the same size as the input frequency domain field. This kernel is generated according to the above formula, with its center aligned to the center frequency of the frequency domain field. Then, we multiply this Gaussian kernel with the input frequency domain field to implement frequency domain filtering:

[0033] F filtered (u,v)=F(u,v)·G(u,v)

[0034] Among them, F(u,v) represents the frequency domain field of the input, and Ffiltered(u,v) is the frequency domain field after filtering; the key to this frequency domain filtering process is to maintain the differentiability of the operation so that backpropagation can be performed during model training; in this way, the Gaussian low-pass filter not only effectively reduces the high-frequency noise in the frequency domain in the frequency domain optimization layer, but also, due to its differentiability, can be integrated with the deep learning model to achieve end-to-end training and optimization; it should be noted that the 64-channel output after point-by-point convolution means that for each pixel position, there are now 64 different eigenvalues, so when performing frequency domain operations (Gaussian low-pass filtering), these operations need to be performed independently on each channel.

[0035] Preferably, the spatial displacement field optimization layer is designed to refine and smooth the displacement field after IDFT, reducing unnatural deformations that may occur during the image registration process; by adding convolution layers and smoothing operations, this layer helps to enhance the continuity and naturalness of the displacement field, thereby improving the quality of the final registered image and outputting the final spatial displacement field. Finally, the image to be registered uses the displacement field to complete the registration task; at the same time, in the training stage, gradient regularization is achieved by calculating the gradient of the displacement field (that is, the difference between the displacement of each pixel point and the displacement of its neighborhood) and summing these gradients to ensure that the displacement field is smooth, which helps to further avoid unnatural image deformations, and the result image obtained by the image to be registered after the displacement field will also perform similarity calculation with the reference image to generate a Loss value, and backpropagate back to the encoder part for parameter optimization operations.

[0036] Preferably, the STN-Attention module includes a localization network, a grid generator, and a sampler. The localization network includes a convolutional layer, a pooling layer, a non-local attention module, and a ReLU activation function. When the STN-Attention module is working, it includes the following steps:

[0037] S11: input the preprocessed image into the localization network;

[0038] S12: The localization network calculates the spatial transformation parameter θ through convolutional layers, pooling layers, non-local attention modules, and ReLU activation functions;

[0039] S13: The grid generator generates a sampling grid using the transformation parameters θ and the size of the input image;

[0040] S14: The sampler uses a grid to spatially sample the input image X, generating a feature map Y that integrates different angles.

[0041] Preferably, a non-local attention block is specifically introduced in the localization network to capture the long-range dependencies of the input feature map. The inclusion of the non-local attention block in the localization network is a significant enhancement because it allows the network to consider global context information when determining the spatial transformation parameters; the input is a pre-processed feature map and the output is a feature map modulated by the non-local attention.

[0042] Preferably, the non-local attention module includes three convolutional layers and one output convolutional layer. When the non-local attention module is working, it includes the following steps:

[0043] S21: Use three convolutional layers to generate three feature maps: θ(X), φ(X), and g(X);

[0044] S22: Calculate attention weight:

[0045] S23: Apply softmax normalization:

[0046] S24: Calculate the output feature map: Y i =∑ j α ij g(X j );

[0047] S25: Process Y with the output convolutional layer to produce the final attention modulation features.

[0048] Preferably, the SMA feature extraction network includes the following steps when processing the input SAR image and extracting the features of the input SAR image:

[0049] S11: Preprocessing the input SAR image to ensure efficient processing of images with different resolutions, wherein the preprocessing method includes noise reduction and resolution adjustment;

[0050] S12: Use the spatial transformation network to process the pre-processed image to reduce the image changes caused by different observation angles;

[0051] S13: Used to extract and fuse multi-scale features of images that have passed through the spatial transformation network to enhance the expressiveness of features.

[0052] Preferably, the SMA feature extraction network processes the input SAR image and extracts the features of the input SAR image, thereby processing the SAR image to generate a full-resolution feature map, which provides key information for the subsequent image registration step. Through such a processing flow, the unique properties of the SAR image are fully considered, thereby enhancing the performance of the model in the multi-angle image registration task.

[0053] As a preferred option, the expression formula for multi-scale feature fusion is:

[0054] F=Concat(X,U(Xlow),D(Xhigh))

[0055] Among them, X is the feature map processed by the STN-Attention module, F is the feature map that integrates features of different scales, U and D represent upsampling and downsampling operations respectively, and Xlow and Xhigh are the low-scale and high-scale feature maps respectively.

[0056] Preferably, the definition formula of the frequency domain field in the DFT layer is: Bφ=F(G(SMA(IM,IF))); wherein IM represents the image to be registered, IF represents the reference image, F represents the DFT layer, φ represents the full-resolution spatial displacement field, G represents the displacement field estimation layer, SMA represents the SMA feature extraction network, and Bφ represents the frequency domain field.

[0057] Preferably, the positioning formula of the full-resolution spatial displacement field in the DFT layer is:

[0058]

[0059] where the size of the full-resolution spatial displacement field is H×W, i∈[0,H-1] and j∈[0,W-1] are discrete indices in the spatial domain, while k∈[0,H-1] and l∈[0,W-1] are discrete indices in the frequency domain.

[0060] Preferably, the displacement field can be further processed in the frequency domain, for example, only retaining signals within a specific frequency range; this is very effective in removing high-frequency noise in the displacement field, especially when processing high-noise data such as SAR images. By removing high-frequency components, the displacement field can be made smoother, thereby reducing artifacts and unnatural deformations that appear in the registered image; and the Fourier transform helps to handle nonlinear and complex deformation patterns, which may be difficult to achieve in the direct spatial domain; therefore, in order to bridge the domain gap between real-valued spatial images and the frequency domain without increasing complexity, a discrete Fourier transform (DFT) layer is a simple and effective method that can generate the frequency domain without requiring the network itself to process complex values.

[0061] Preferably, the frequency domain optimization layer optimizes the frequency domain field using the convolution and filtering operations of the convolutional neural network; wherein, when optimizing the frequency domain field using the convolution and filtering operations of the convolutional neural network, the high-frequency noise features in the frequency domain are particularly infused and the low-frequency components representing the main structural information of the image are retained.

[0062] Preferably, the frequency domain optimization layer optimizes the frequency domain field through the convolution and filtering operations of the CNN, paying special attention to the high-frequency noise characteristics in the frequency domain and retaining the low-frequency components, which represent the main structural information of the image; retaining this information during the registration process helps to improve the registration accuracy, especially in the case of complex structures such as SAR images, which may undergo diverse changes due to various reasons (such as different imaging angles). Processing the displacement field in the frequency domain can improve the robustness of the registration process, enabling it to adapt to these changes.

[0063] As a preference, the principle formula of the IDFT layer is:

[0064]

[0065] Where H and W represent the height and width of the image, respectively, φ represents the full-resolution spatial displacement field, and [F(φ)]k,l is the representation in the frequency domain.

[0066] Beneficial effects of the present invention:

[0067] 1. Compared with existing technologies that use SIFI feature-based methods, deep neural network-based methods, MU-Net multi-scale framework methods, shape context methods, direct matching of multi-angle SAR images, and semantic distribution location-based methods to solve the multi-angle SAR image registration problem, the registration performance and robustness are insufficient when surface data is scarce and in complex scenes. This registration method uses an SMA feature extraction network and adopts multiple technical modules, including preprocessing strategies, STN-Attention, and multi-scale feature fusion technology. These technical modules combine to provide a comprehensive and efficient solution for SAR image registration.

[0068] 2. The Fourier-Net-SAR model mainly obtains the displacement field of the image. The image to be registered is transformed through the obtained displacement field to finally obtain the registration result. The core part of this model is the SMA feature extraction network, which uses the powerful feature extraction capability of the SMA feature extraction network to complete the SAR image registration task;

[0069] 3. During the network's forward propagation, the feature map first passes through the feature extraction layer and is then input into the displacement field estimation layer. The output of the displacement field estimation layer is a full-resolution displacement field, represented as a displacement vector for each pixel position. The displacement field estimation layer enables the network to accurately predict the displacement of each pixel in the image, which is crucial for achieving high-precision image registration. By using a series of convolutional layers, the network is able to extract the necessary displacement information from complex image features while maintaining computational efficiency.

[0070] 4. The displacement field can be further processed in the frequency domain, for example, only retaining signals within a specific frequency range; this is very effective in removing high-frequency noise in the displacement field, especially when processing high-noise data such as SAR images. By removing high-frequency components, the displacement field can be made smoother, thereby reducing artifacts and unnatural deformations in the registered image; and the Fourier transform helps to deal with nonlinear and complex deformation patterns, which may be difficult to achieve in the direct spatial domain; therefore, in order to bridge the domain gap between real-valued spatial images and the frequency domain without increasing complexity, a discrete Fourier transform (DFT) layer is a simple and effective method that can generate the frequency domain without requiring the network itself to process complex values. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 Shown is a schematic diagram of the construction of the multi-angle registration model based on Fourier-Net-SAR of the present invention;

[0072] Figure 2Shown is a schematic diagram of the structure of the SMA feature extraction network in the multi-angle registration model based on Fourier-Net-SAR of the present invention;

[0073] Figure 3 Shown is a schematic diagram of the workflow of the STN-Attention module in the multi-angle registration model based on Fourier-Net-SAR of the present invention. DETAILED DESCRIPTION

[0074] The present invention will be further described below with reference to the accompanying drawings and examples.

[0075] See also Figure 1 The present invention provides an embodiment: a multi-angle registration model based on Fourier-Net-SAR, comprising:

[0076] A11: SMA feature extraction network, used to process the input SAR image and extract features of the input SAR image. When extracting features from the input SAR image, the characteristics of the input SAR image, such as the influence of noise and angle, are taken into account.

[0077] A12: Displacement field estimation layer, which is used to estimate the displacement field by adding a set of convolutional layers after feature extraction of the input SAR image, directly estimating the displacement field from the extracted image features;

[0078] A13: DFT layer, used to further process the displacement field in the frequency domain to remove high-frequency noise in the displacement field and make the displacement field smoother;

[0079] A14: Frequency domain optimization layer, used to further optimize the frequency domain field while making the entire process differentiable for effective backpropagation.

[0080] A15: IDFT layer, used to convert the frequency domain field processed by the frequency domain optimization layer into a displacement field;

[0081] A16: Displacement field optimization layer, used to refine the displacement field to prevent unnatural image deformation and output the final displacement field.

[0082] Preferably, the Fourier-Net-SAR model is mainly used to obtain the displacement field of the image. The image to be registered is transformed through the obtained displacement field to finally obtain the registration result. The core part of this model is the SMA feature extraction network, which uses the powerful feature extraction capability of the SMA feature extraction network to complete the SAR image registration task.

[0083] See also Figure 2-3 In this embodiment, the SMA feature extraction network includes:

[0084] A preprocessing module is used to preprocess the input SAR image;

[0085] The STN-Attention module is used to assist in extracting image features and reduce changes caused by different angles;

[0086] The multi-scale feature extraction module is used to fuse features extracted at different scales through depth-wise separable convolution to obtain full-resolution feature representation.

[0087] Preferably, when the preprocessing module preprocesses the input SAR image, the source image and the target image are first adjusted to an appropriate resolution, then the convolution layer is used to reduce the initial noise, and finally the Lee filter is used to further suppress the noise.

[0088] Preferably, the STN-Attention module includes a localization network, a grid generator, and a sampler. The localization network includes a convolutional layer, a pooling layer, a non-local attention module, and a ReLU activation function. When the STN-Attention module is working, it includes the following steps:

[0089] S11: input the preprocessed image into the localization network;

[0090] S12: The localization network calculates the spatial transformation parameter θ through convolutional layers, pooling layers, non-local attention modules, and ReLU activation functions;

[0091] S13: The grid generator generates a sampling grid using the transformation parameters θ and the size of the input image;

[0092] S14: The sampler uses a grid to spatially sample the input image X, generating a feature map Y that integrates different angles.

[0093] Preferably, a non-local attention block is specifically introduced in the localization network to capture the long-range dependencies of the input feature map. The inclusion of the non-local attention block in the localization network is a significant enhancement because it allows the network to consider global context information when determining the spatial transformation parameters; the input is a pre-processed feature map and the output is a feature map modulated by the non-local attention.

[0094] Preferably, the non-local attention module includes three convolutional layers and one output convolutional layer. When the non-local attention module is working, it includes the following steps:

[0095] S21: Use three convolutional layers to generate three feature maps: (θ(X)), (φ(X)), and (g(X));

[0096] S22: Calculate attention weight:

[0097] S23: Apply softmax normalization:

[0098] S24: Calculate the output feature map: (Y i =∑ j α ij g(X j ));

[0099] S25: Process (Y) with the output convolutional layer to produce the final attention modulation features.

[0100] Preferably, the SMA feature extraction network includes the following steps when processing the input SAR image and extracting the features of the input SAR image:

[0101] S11: Preprocessing the input SAR image to ensure efficient processing of images with different resolutions, wherein the preprocessing method includes noise reduction and resolution adjustment;

[0102] S12: Use the spatial transformation network to process the pre-processed image to reduce the image changes caused by different observation angles;

[0103] S13: Used to extract and fuse multi-scale features of images that have passed through the spatial transformation network to enhance the expressiveness of features.

[0104] Preferably, the SMA feature extraction network processes the input SAR image and extracts the features of the input SAR image, thereby processing the SAR image to generate a full-resolution feature map, which provides key information for the subsequent image registration step. Through such a processing flow, the unique properties of the SAR image are fully considered, thereby enhancing the performance of the model in the multi-angle image registration task.

[0105] As a preferred option, the expression formula for multi-scale feature fusion is:

[0106] F=Concat(X,U(Xlow),D(Xhigh))

[0107] Among them, X is the feature map processed by the STN-Attention module, F is the feature map that integrates features of different scales, U and D represent upsampling and downsampling operations respectively, and Xlow and Xhigh are the low-scale and high-scale feature maps respectively.

[0108] Preferably, the displacement field estimation layer includes:

[0109] A21: The first convolution layer has 1024 input channels and uses a 3x3 convolution kernel with a padding value of 1 to keep the spatial size of the feature map unchanged. The number of output channels is set to 512 to reduce the feature dimension while retaining key information.

[0110] A22: ReLU activation function, applied after the first convolution layer to introduce nonlinearity to capture complex displacement patterns;

[0111] A23: Additional convolutional layers, which process features through three additional convolutional layers. The number of output channels of the second convolution layer is reduced to 256, the number of output channels of the third convolution layer is reduced to 128, and the fourth layer is further reduced to 64. These layers continue to reduce dimensionality and refine features in preparation for displacement field estimation;

[0112] A24: Final displacement field output layer. The final displacement field output layer is a convolutional layer with two output channels. The two channels at each pixel position represent the displacement in the x and y directions respectively. This layer uses a 3x3 convolution kernel and a padding value of 1 to ensure that the output displacement field has the same spatial size as the input image.

[0113] Preferably, during the forward propagation of the network, the feature map first passes through the feature extraction layer and then is input into the displacement field estimation layer; the output of the displacement field estimation layer is a full-resolution displacement field, represented as a displacement vector for each pixel position; the displacement field estimation layer enables the network to accurately predict the displacement of each pixel in the image, which is crucial for achieving high-precision image registration; by using a series of convolutional layers, the network can extract the necessary displacement information from complex image features while maintaining computational efficiency.

[0114] Preferably, the definition formula of the frequency domain field in the DFT layer is: Bφ=F(G(SMA(IM,IF))); wherein IM represents the image to be registered, IF represents the reference image, F represents the DFT layer, φ represents the full-resolution spatial displacement field, G represents the displacement field estimation layer, SMA represents the SMA feature extraction network, and Bφ represents the frequency domain field.

[0115] Preferably, the positioning formula of the full-resolution spatial displacement field in the DFT layer is:

[0116]

[0117] where the size of the full-resolution spatial displacement field is H×W, i∈[0,H-1] and j∈[0,W-1] are discrete indices in the spatial domain, while k∈[0,H-1] and l∈[0,W-1] are discrete indices in the frequency domain.

[0118] Preferably, the displacement field can be further processed in the frequency domain, for example, only retaining signals within a specific frequency range; this is very effective in removing high-frequency noise in the displacement field, especially when processing high-noise data such as SAR images. By removing high-frequency components, the displacement field can be made smoother, thereby reducing artifacts and unnatural deformations that appear in the registered image; and the Fourier transform helps to handle nonlinear and complex deformation patterns, which may be difficult to achieve in the direct spatial domain; therefore, in order to bridge the domain gap between real-valued spatial images and the frequency domain without increasing complexity, a discrete Fourier transform (DFT) layer is a simple and effective method that can generate the frequency domain without requiring the network itself to process complex values.

[0119] Preferably, the frequency domain optimization layer optimizes the frequency domain field using the convolution and filtering operations of the convolutional neural network; wherein, when optimizing the frequency domain field using the convolution and filtering operations of the convolutional neural network, the high-frequency noise features in the frequency domain are particularly infused and the low-frequency components representing the main structural information of the image are retained.

[0120] Preferably, the frequency domain optimization layer optimizes the frequency domain field through the convolution and filtering operations of the CNN, paying special attention to the high-frequency noise characteristics in the frequency domain and retaining the low-frequency components, which represent the main structural information of the image; retaining this information during the registration process helps to improve the registration accuracy, especially in the case of complex structures such as SAR images, which may undergo diverse changes due to various reasons (such as different imaging angles). Processing the displacement field in the frequency domain can improve the robustness of the registration process, enabling it to adapt to these changes.

[0121] Preferably, the frequency domain optimization layer comprises:

[0122] A31: Frequency domain optimization input layer, used to input the frequency domain field processed by the DFT layer. The input channel of the input layer is 2, and the input channels of the two frequency domain optimization input layers correspond to the real part and imaginary part of the frequency domain data respectively.

[0123] A32: The first point-by-point convolution layer has 2 input channels and 64 output channels, and uses a 1x1 convolution kernel for processing. The stride of the first point-by-point convolution layer is 1 to maintain spatial resolution, and same padding is used to keep the input and output sizes consistent. After the first point-by-point convolution layer, batch normalization and ReLU activation operations are performed to further adjust and refine the frequency domain features.

[0124] A33: The second frequency domain specific operation layer is used to suppress the high-frequency noise in the input frequency domain field data after Fourier transformation using a Gaussian low-pass filter;

[0125] A34: The third depth-wise separable convolution layer has 64 input channels and 64 output channels. It uses a 3x3 convolution kernel for processing, with a stride of 1 to maintain spatial resolution and same padding to keep the input and output sizes consistent. Batch normalization and ReLU activation operations are performed after the third depth-wise separable convolution layer to provide efficient feature extraction while maintaining computational efficiency.

[0126] A35: Output convolution layer, the input channel of this layer is 64, the output channel is 2, and it uses a convolution kernel of size 3x3 for processing, with a stride of 1 to maintain spatial resolution, and same padding is used to keep the input and output sizes consistent.

[0127] Preferably, the Gaussian low-pass filter smoothes the high-frequency components in its unique way in the frequency domain while retaining the low-frequency components, thereby denoising and enhancing the frequency domain field; the mathematical definition of the Gaussian low-pass filter is as follows:

[0128]

[0129] Where u and v represent the horizontal and vertical frequency coordinates in the frequency domain, respectively. σ is the standard deviation of the Gaussian kernel and is set to 2, which controls the bandwidth of the filter. When implementing a Gaussian low-pass filter, we first need to construct a Gaussian kernel of the same size as the input frequency domain field. This kernel is generated according to the above formula, with its center aligned to the center frequency of the frequency domain field. Then, we multiply this Gaussian kernel with the input frequency domain field to implement frequency domain filtering:

[0130] F filtered (u,v)=F(u,v)·G(u,v)

[0131] Among them, F(u,v) represents the frequency domain field of the input, and Ffiltered(u,v) is the frequency domain field after filtering; the key to this frequency domain filtering process is to maintain the differentiability of the operation so that backpropagation can be performed during model training; in this way, the Gaussian low-pass filter not only effectively reduces the high-frequency noise in the frequency domain in the frequency domain optimization layer, but also, due to its differentiability, can be integrated with the deep learning model to achieve end-to-end training and optimization; it should be noted that the 64-channel output after point-by-point convolution means that for each pixel position, there are now 64 different eigenvalues, so when performing frequency domain operations (Gaussian low-pass filtering), these operations need to be performed independently on each channel.

[0132] As a preference, the principle formula of the IDFT layer is:

[0133]

[0134] Where H and W represent the height and width of the image, respectively, φ represents the full-resolution spatial displacement field, and [F(φ)]k,l is the representation in the frequency domain.

[0135] Preferably, the displacement field optimization layer comprises:

[0136] A41: Displacement field optimization input layer, used to input the spatial displacement field output after IDFT layer processing. The input channel of the displacement field optimization input layer is 2;

[0137] A42: Displacement field optimization of the first convolutional layer. This layer has 2 input channels and 64 output channels, and uses a 3x3 convolution kernel for processing. The stride is 1 to maintain spatial resolution, and same padding is used to keep the input and output sizes consistent. Batch normalization and ReLU activation operations are performed after the first convolutional layer of displacement field optimization to extract high-level features of the displacement field.

[0138] A43: Displacement field optimization The second displacement field smoothing operation layer is used to smooth the displacement field using a bilateral filter to reduce discontinuities and unnatural deformations.

[0139] A44: Displacement field optimized output convolution layer. The input channel of the displacement field optimized output convolution layer is 64, the output channel is 2, and a convolution kernel of size 3x3 is used for processing. The stride is 1 to maintain the spatial resolution, and the same padding is used to keep the input and output sizes consistent.

[0140] Preferably, the design of the spatial displacement field optimization layer is intended to refine and smooth the displacement field after IDFT, reducing unnatural deformations that may occur during the image registration process. By adding convolutional layers and smoothing operations, this layer helps to enhance the continuity and naturalness of the displacement field, thereby improving the quality of the final registered image and outputting the final spatial displacement field. Finally, the image to be registered uses the displacement field to complete the registration task. At the same time, in the training phase, gradient regularization is achieved by calculating the gradient of the displacement field (that is, the difference between the displacement of each pixel and the displacement of its neighborhood) and summing these gradients to ensure that the displacement field is smooth, which helps to further avoid unnatural image deformations, and the result image obtained by the displacement field of the image to be registered will also perform similarity calculations with the reference image to generate Loss values, and backpropagate back to the encoder part for parameter optimization operations.

[0141] During the work, the processing of the raw SAR image includes resolution adjustment and noise suppression to reduce the interference of noise on subsequent feature extraction. The specific operation is as follows: the input raw SAR image is subjected to a series of processing steps, including convolutional layers for initial noise reduction and Lee filters for further noise reduction, and the final output is the image after noise reduction and Lee filtering. The purpose of this process is to reduce the noise level in the image while maintaining the same image resolution, so as to improve the accuracy of subsequent feature extraction.

[0142] Next, the processed image passes through the STN-Attention module, whose main function is to assist in extracting image features and reduce the changes caused by different angles. In STN-Attention, non-local attention blocks are also introduced to better capture long-range dependencies in the image, thereby enhancing the performance of STN-Attention when handling complex geometric transformations. The key to this step is to use STN-Attention to normalize and align the images, so that the network can better understand the spatial relationship between images and reduce deformation caused by angle changes.

[0143] After STN-Attention processing, deep SAR feature extraction is performed to further extract useful information from the image. This process includes multi-scale feature fusion such as multi-layer convolution, pooling structure, and up- and down-sampling. The core of multi-scale feature fusion is to fuse features extracted at different scales through depthwise separable convolution to obtain full-resolution feature representation. This fusion method ensures that the network can obtain information at different scales, thereby improving the diversity and expression ability of features.

[0144] Through the above steps, the entire image processing pipeline includes steps such as resolution adjustment, noise suppression, STN-Attention-assisted feature extraction, and multi-scale feature fusion, which aims to address the specific challenges of SAR images to achieve better registration performance.

[0145] The embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the scope of knowledge of those skilled in the art without departing from the spirit of the present invention.

Claims

1. A multi-angle registration model based on Fourier-Net-SAR; characterized by: Includes: A11: SMA feature extraction network, used to process the input SAR image and extract features from the input SAR image. When extracting features from the input SAR image, the characteristics of the input SAR image, such as the influence of noise and angle, are considered. These include: A preprocessing module is used to preprocess the input SAR image; The STN-Attention module is used to assist in extracting image features and reduce changes caused by different angles; Multi-scale feature extraction module, which is used to fuse features extracted at different scales through depth-wise separable convolution to obtain full-resolution feature representation; A12: Displacement field estimation layer, which is used to estimate the displacement field by adding a set of convolutional layers after feature extraction of the input SAR image. The displacement field is estimated directly from the extracted image features, including: A21: The first convolution layer has 1024 input channels and uses a 3x3 convolution kernel with a padding value of 1 to keep the spatial size of the feature map unchanged. The number of output channels is set to 512 to reduce the feature dimension while retaining key information. A22: ReLU activation function, applied after the first convolution layer to introduce nonlinearity to capture complex displacement patterns; A23: Additional convolutional layer, through three additional convolutional layers to process features, the second convolution layer reduces the number of output channels to 256, the third convolution layer reduces the number of output channels to 128, and the fourth layer further reduces it to 64. These layers continue to reduce the dimension and refine the features to prepare for displacement field estimation; A24: Final displacement field output layer. The final displacement field output layer is a convolutional layer with two output channels. The two channels at each pixel position represent the displacement in the x and y directions respectively. This layer uses a 3x3 convolution kernel and a padding value of 1 to ensure that the output displacement field has the same spatial size as the input image. A13: DFT layer, used to further process the displacement field in the frequency domain to remove high-frequency noise in the displacement field and make the displacement field smoother; A14: Frequency domain optimization layer, used to further optimize the frequency domain field and perform differentiation while running the entire process for effective backpropagation, including: A31: Frequency domain optimization input layer, used to input the frequency domain field processed by the DFT layer. The input channel of the input layer is 2, and the input channels of the two frequency domain optimization input layers correspond to the real part and imaginary part of the frequency domain data respectively. A32: The first point-by-point convolution layer has 2 input channels and 64 output channels, and uses a 1x1 convolution kernel for processing. The stride of the first point-by-point convolution layer is 1 to maintain spatial resolution, and same padding is used to keep the input and output sizes consistent. After the first point-by-point convolution layer, batch normalization and ReLU activation operations are performed to further adjust and refine the frequency domain features. A33: The second frequency domain specific operation layer is used to suppress the high-frequency noise in the input frequency domain field data after Fourier transformation using a Gaussian low-pass filter; A34: The third depth-wise separable convolution layer has 64 input channels and 64 output channels. It uses a 3x3 convolution kernel for processing, with a stride of 1 to maintain spatial resolution and same padding to keep the input and output sizes consistent. Batch normalization and ReLU activation operations are performed after the third depth-wise separable convolution layer to provide efficient feature extraction while maintaining computational efficiency. A35: Output convolution layer, which has 64 input channels and 2 output channels, and uses a 3x3 convolution kernel for processing, with a stride of 1 to maintain spatial resolution, and uses same padding to keep the input and output sizes consistent; A15: IDFT layer, used to convert the frequency domain field processed by the frequency domain optimization layer into a displacement field; A16: Displacement field optimization layer, used to refine the displacement field to prevent unnatural image deformation and output the final displacement field, including: A41: Displacement field optimization input layer, used to input the spatial displacement field output after IDFT layer processing. The input channel of the displacement field optimization input layer is 2; A42: Displacement field optimization of the first convolutional layer. This layer has 2 input channels and 64 output channels, and uses a 3x3 convolution kernel for processing. The stride is 1 to maintain spatial resolution, and same padding is used to keep the input and output sizes consistent. Batch normalization and ReLU activation operations are performed after the first convolutional layer of displacement field optimization to extract high-level features of the displacement field. A43: Displacement field optimization The second displacement field smoothing operation layer is used to smooth the displacement field using a bilateral filter to reduce discontinuities and unnatural deformations. A44: Displacement field optimized output convolution layer. The input channel of the displacement field optimized output convolution layer is 64, the output channel is 2, and a convolution kernel of size 3x3 is used for processing. The stride is 1 to maintain the spatial resolution, and the same padding is used to keep the input and output sizes consistent.

2. The multi-angle registration model based on Fourier-Net-SAR according to claim 1, characterized in that: The STN-Attention module consists of a localization network, a grid generator, and a sampler. The localization network includes a convolutional layer, a pooling layer, a non-local attention module, and a ReLU activation function. When the STN-Attention module is working, it includes the following steps: S11: input the preprocessed image into the localization network; S12: The localization network calculates the spatial transformation parameter θ through convolutional layers, pooling layers, non-local attention modules, and ReLU activation functions; S13: The grid generator generates a sampling grid using the transformation parameters θ and the size of the input image; S14: The sampler uses a grid to spatially sample the input image X, generating a feature map Y that integrates different angles.

3. The multi-angle registration model based on Fourier-Net-SAR according to claim 2, characterized in that: The non-local attention module consists of three convolutional layers and one output convolutional layer. When the non-local attention module works, it includes the following steps: S21: Use three convolutional layers to generate three feature maps: θ(X), φ(X), and g(X); S22: Calculate attention weight: S23: Apply softmax normalization: S24: Calculate the output feature map: Y i =Σ j α ij g(X j ); S25: Process Y with the output convolutional layer to produce the final attention modulation features.

4. The multi-angle registration model based on Fourier-Net-SAR according to claim 3, characterized in that: The definition formula of the frequency domain field in the DFT layer is: Bφ=F(G(SMA(IM,IF))); where IM represents the image to be registered, IF represents the reference image, F represents the DFT layer, φ represents the full-resolution spatial displacement field, G represents the displacement field estimation layer, SMA represents the SMA feature extraction network, and Bφ represents the frequency domain field. The positioning formula of the full-resolution spatial displacement field in the DFT layer is: where the size of the full-resolution spatial displacement field is H×W, i∈[0,H-1] and j∈[0,W-1] are discrete indices in the spatial domain, while k∈[0,H-1] and l∈[0,W-1] are discrete indices in the frequency domain.

5. The multi-angle registration model based on Fourier-Net-SAR according to claim 4, characterized in that: The frequency domain optimization layer uses the convolution and filtering operations of the convolutional neural network to optimize the frequency domain field; among them, when using the convolution and filtering operations of the convolutional neural network to optimize the frequency domain field, it especially infuses the high-frequency noise features in the frequency domain and retains the low-frequency components representing the main structural information of the image.

6. The multi-angle registration model based on Fourier-Net-SAR according to claim 5, characterized in that: The principle formula of the IDFT layer is: Where H and W represent the height and width of the image, respectively, φ represents the full-resolution spatial displacement field, and [F(φ)]k,l is the representation in the frequency domain.

Images