Optical and SAR image registration method, device and equipment based on position perception

Abstract

The present application relates to a method, apparatus, and device for optical and SAR image registration based on position perception. The method utilizes a heterogeneous image registration network to perform image registration on an optical reference image and a SAR image. In the network, the two input images are superimposed in the channel direction, and then a multi-scale feature extraction unit fused with coordinate attention is used to obtain a deep feature expression containing position information. To further improve the positioning capability of the features, the multi-scale feature fusion unit is used to effectively aggregate the deep feature maps at different levels to obtain representative features that have both deep semantic representation capabilities and precise positioning information. Finally, a fully connected layer is used to map geometric transformation parameters, and a spatial transformation module is used to perform an affine transformation on the image to be registered to obtain the registered image. This method can improve the accuracy of heterogeneous image registration while also improving the real-time performance of the registration.

Description

Technical Field

[0001] The present application relates to the technical field of heterogeneous image registration, and in particular to a method, apparatus and device for optical and SAR image registration based on position perception. Background Art

[0002] Optical images and Synthetic Aperture Radar (SAR) images are two primary data sources for Earth observation. The information they convey about ground features is highly complementary. Combining the characteristics of optical and SAR images for fusion analysis holds great promise for applications in change detection, ground feature classification, and target recognition. Image registration is a crucial technical prerequisite for achieving this fusion analysis. Image registration technology is widely used in tasks such as missile-borne scene matching guidance, aircraft positioning and navigation, and ground feature change detection. Achieving high-precision, real-time image registration is a current research priority in this field. Due to their distinct imaging mechanisms, in addition to differences in geometric structure, rotation, and resolution between homologous images, optical and SAR images also exhibit significant nonlinear grayscale distortion, rendering homologous image registration methods largely ineffective.

[0003] In recent years, numerous effective methods have been proposed for heterogeneous remote sensing image registration, primarily categorized as traditional methods and deep learning-based methods. However, both methods suffer from issues such as insufficient generalization across image scenes and poor real-time performance. Summary of the Invention

[0004] Based on this, it is necessary to provide a method, device and equipment for optical and SAR image registration based on position perception that can solve at least one of the above technical problems.

[0005] A method for optical and SAR image registration based on position perception, the method comprising:

[0006] Acquire a registration dataset, wherein the registration dataset includes a plurality of pre-registered optical image and SAR image pairs;

[0007] Performing a random affine transformation on the SAR image in the registration dataset to obtain a SAR training image having an affine transformation relationship with the original SAR image and a true value label of the transformation parameter, and forming a training data pair with the SAR training image and the corresponding optical image;

[0008] Inputting the training data pair into a heterogeneous image registration network, wherein the training data are superimposed in the channel direction to obtain a channel superimposed image, extracting feature images of different scales from the channel superimposed image through a multi-scale feature extraction unit that fuses coordinate attention, fusing the feature images of different scales with multi-scale local information and global information through a feature fusion unit to obtain a feature fusion image, performing global adaptive average pooling on the feature fusion image, and obtaining training transformation parameters through a parameter regression unit;

[0009] The SAR training image is affine transformed by a spatial transformation unit according to training transformation parameters to obtain a training registered image;

[0010] A loss function is constructed based on the training registered image, the training transformation parameters, the optical image in the training data pair, and the true value labels of the corresponding transformation parameters;

[0011] Training the heterogeneous image registration network according to the calculation result of the loss function until the loss function converges, thereby obtaining a trained heterogeneous image registration network;

[0012] A SAR image to be registered and an optical reference image are obtained, the SAR image to be registered and the optical reference image are input into a trained heterogeneous image registration network to obtain transformation parameters, and an affine transformation is performed on the SAR image to be registered according to the transformation parameters by a spatial transformation unit to obtain a registered SAR image.

[0013] In one embodiment, the multi-scale feature extraction unit fused with coordinate attention includes three feature extraction layers fused with coordinate attention connected in sequence, namely a first feature extraction layer, a second feature extraction layer, and a third feature extraction layer;

[0014] Inputting the channel superposition image into the first feature extraction layer to extract a shallow feature image;

[0015] Inputting the shallow feature image into the second feature extraction layer to extract a middle feature image;

[0016] The middle-layer feature image is input into the third feature extraction layer to extract a deep-layer feature image.

[0017] In one embodiment, each feature extraction layer in the multi-scale feature extraction unit integrating coordinate attention includes a Res2Net module and a coordinate attention mechanism module;

[0018] The feature images of different scales input to each feature extraction layer are extracted through the Res2Net module to obtain high-level semantic features;

[0019] The high-level semantic features are then used to extract deeper feature images through the coordinate attention mechanism module.

[0020] In one embodiment, in the feature fusion unit:

[0021] Performing different processing on the shallow feature image, the middle feature image, and the deep feature image respectively, and transforming the feature images of different scales to have the same spatial size as the shallow feature image and the same number of channels as the deep feature image;

[0022] The feature images after spatial and channel transformation are added, and the addition results are sequentially passed through the coordinate attention mechanism module and the fourth feature extraction layer for deep feature fusion to obtain the feature fusion image.

[0023] In one embodiment, the performing different processing on the shallow feature image, the middle feature image, and the deep feature image respectively includes:

[0024] Expanding the number of channels of the shallow feature image using 1×1 convolution;

[0025] After expanding the number of channels of the middle-layer feature image using 1×1 convolution, a two-fold upsampling process is performed;

[0026] After processing the deep feature image using global adaptive average pooling, it is expanded using 1×1 convolution logarithmically.

[0027] In one embodiment, constructing a loss function based on the training registered image, the training transformation parameters, the optical image in the training data pair, and the corresponding transformation parameter true value labels includes:

[0028] constructing a similarity loss function based on a similarity measure between the training registered image and the optical image in the training data pair;

[0029] Constructing a four-corner shift loss function based on the training transformation parameters and the corresponding transformation parameter true value labels;

[0030] A total loss function is obtained according to the similarity loss function and the four-corner offset loss function.

[0031] In one embodiment, when obtaining the trained heterogeneous image registration network, the spatial transformation unit is embedded into the network;

[0032] After the SAR image to be registered and the optical reference image are input into the trained heterogeneous image registration network, a registered SAR image registered based on the optical reference image is obtained.

[0033] A device for optical and SAR image registration based on position perception, comprising:

[0034] A registration data set acquisition module, configured to acquire a registration data set, wherein the registration data set includes a plurality of pre-aligned optical image and SAR image pairs;

[0035] a training data pair construction module, configured to perform a random affine transformation on the SAR image in the registration dataset to obtain a SAR training image having an affine transformation relationship with the original SAR image and a true value label of the transformation parameter, and to form a training data pair with the SAR training image and the corresponding optical image;

[0036] a training transformation parameter obtaining module, configured to input the training data pair into a heterogeneous image registration network, wherein the training data is superimposed in the channel direction to obtain a channel superimposed image, wherein the channel superimposed image is extracted with a multi-scale feature extraction unit that fuses coordinate attention to extract feature images of different scales, and then a feature fusion unit fuses the feature images of different scales with multi-scale local information and global information to obtain a feature fusion image, and then a parameter regression unit obtains training transformation parameters after performing global adaptive average pooling on the feature fusion image;

[0037] A module for obtaining a registered training image, configured to perform an affine transformation on the SAR training image through a spatial transformation unit according to training transformation parameters to obtain a registered training image;

[0038] A loss function construction module is used to construct a loss function based on the training registered image, the training transformation parameters, the optical image in the training data pair, and the true value labels of the corresponding transformation parameters;

[0039] A heterogeneous image registration network training module is used to train the heterogeneous image registration network according to the calculation result of the loss function until the loss function converges, thereby obtaining a trained heterogeneous image registration network;

[0040] The heterogeneous image registration module is used to obtain a SAR image to be registered and an optical reference image, input the SAR image to be registered and the optical reference image into a trained heterogeneous image registration network, obtain transformation parameters, and perform an affine transformation on the SAR image to be registered according to the transformation parameters through a spatial transformation unit to obtain a registered SAR image.

[0041] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0042] Acquire a registration dataset, wherein the registration dataset includes a plurality of pre-registered optical image and SAR image pairs;

[0043] Performing a random affine transformation on the SAR image in the registration dataset to obtain a SAR training image having an affine transformation relationship with the original SAR image and a true value label of the transformation parameter, and forming a training data pair with the SAR training image and the corresponding optical image;

[0044] Inputting the training data pair into a heterogeneous image registration network, wherein the training data are superimposed in the channel direction to obtain a channel superimposed image, extracting feature images of different scales from the channel superimposed image through a multi-scale feature extraction unit that fuses coordinate attention, fusing the feature images of different scales with multi-scale local information and global information through a feature fusion unit to obtain a feature fusion image, performing global adaptive average pooling on the feature fusion image, and obtaining training transformation parameters through a parameter regression unit;

[0045] The SAR training image is affine transformed by a spatial transformation unit according to training transformation parameters to obtain a training registered image;

[0046] A loss function is constructed based on the training registered image, the training transformation parameters, the optical image in the training data pair, and the true value labels of the corresponding transformation parameters;

[0047] Training the heterogeneous image registration network according to the calculation result of the loss function until the loss function converges, thereby obtaining a trained heterogeneous image registration network;

[0048] A SAR image to be registered and an optical reference image are obtained, the SAR image to be registered and the optical reference image are input into a trained heterogeneous image registration network to obtain transformation parameters, and an affine transformation is performed on the SAR image to be registered according to the transformation parameters by a spatial transformation unit to obtain a registered SAR image.

[0049] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0050] Acquire a registration dataset, wherein the registration dataset includes a plurality of pre-registered optical image and SAR image pairs;

[0051] Performing a random affine transformation on the SAR image in the registration dataset to obtain a SAR training image having an affine transformation relationship with the original SAR image and a true value label of the transformation parameter, and forming a training data pair with the SAR training image and the corresponding optical image;

[0052] Inputting the training data pair into a heterogeneous image registration network, wherein the training data are superimposed in the channel direction to obtain a channel superimposed image, extracting feature images of different scales from the channel superimposed image through a multi-scale feature extraction unit that fuses coordinate attention, fusing the feature images of different scales with multi-scale local information and global information through a feature fusion unit to obtain a feature fusion image, performing global adaptive average pooling on the feature fusion image, and obtaining training transformation parameters through a parameter regression unit;

[0053] The SAR training image is affine transformed by a spatial transformation unit according to training transformation parameters to obtain a training registered image;

[0054] A loss function is constructed based on the training registered image, the training transformation parameters, the optical image in the training data pair, and the true value labels of the corresponding transformation parameters;

[0055] Training the heterogeneous image registration network according to the calculation result of the loss function until the loss function converges, thereby obtaining a trained heterogeneous image registration network;

[0056] A SAR image to be registered and an optical reference image are obtained, the SAR image to be registered and the optical reference image are input into a trained heterogeneous image registration network to obtain transformation parameters, and an affine transformation is performed on the SAR image to be registered according to the transformation parameters by a spatial transformation unit to obtain a registered SAR image.

[0057] The aforementioned location-aware optical and SAR image registration method, apparatus, and device utilize a heterogeneous image registration network to register an optical reference image and a SAR image. In this network, the two input images are superimposed in the channel direction, and then a multi-scale feature extraction unit fused with coordinate attention is used to obtain a deep feature representation containing location information. To further improve the localization capability of features, the multi-scale feature fusion unit effectively aggregates deep feature maps at different levels to obtain representative features that have both deep semantic representation capabilities and precise positioning information. Finally, a fully connected layer is used to map geometric transformation parameters, and a spatial transformation module is used to perform an affine transformation on the image to be registered to obtain the registered image. This method can improve the accuracy of heterogeneous image registration while also enhancing the real-time performance of the registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 1 is a flow chart of a method for optical and SAR image registration based on position awareness in one embodiment;

[0059] Figure 2A schematic block diagram of a heterogeneous image registration network training process in one embodiment;

[0060] Figure 3 A schematic diagram of the structure of a multi-scale feature extraction unit integrating coordinate attention in one embodiment;

[0061] Figure 4 Schematic diagram of the structure of a feature fusion unit in one embodiment;

[0062] Figure 5 is a structural block diagram of an optical and SAR image registration device based on position perception in one embodiment;

[0063] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0065] In the existing technology, for the registration problem of optical and SAR images with significant geometric and radiometric differences, traditional methods and current registration algorithms based on deep learning mostly rely on feature point detection and matching. However, due to the different scene areas, it is easy to have many mismatched points or insufficient valid homonymous points, resulting in registration failure. In this method, Figure 1 As shown, a method for optical and SAR image registration based on position perception is provided, comprising the following steps:

[0066] Step S100, obtaining a registration dataset, where the registration dataset includes a plurality of pre-registered optical image and SAR image pairs;

[0067] Step S110, performing a random affine transformation on the SAR image in the registration data set to obtain a SAR training image having an affine transformation relationship with the original SAR image and a true value label of the transformation parameter, and forming a training data pair with the SAR training image and the corresponding optical image;

[0068] Step S120: Input the training data pair into a heterogeneous image registration network. In the heterogeneous image registration network, the training data are superimposed in the channel direction to obtain a channel superimposed image. The channel superimposed image is extracted by a multi-scale feature extraction unit that fuses coordinate attention to extract feature images of different scales. The feature fusion unit then fuses the feature images of different scales with multi-scale local information and global information to obtain a feature fusion image. The feature fusion image is then subjected to global adaptive average pooling and then a parameter regression unit is used to obtain training transformation parameters.

[0069] Step S130, performing affine transformation on the SAR training image through a spatial transformation unit according to the training transformation parameters to obtain a training registered image;

[0070] Step S140, constructing a loss function based on the training registered image, the training transformation parameters, the optical image in the training data pair, and the true value labels of the corresponding transformation parameters;

[0071] Step S150, training the heterogeneous image registration network according to the calculation result of the loss function until the loss function converges, thereby obtaining a trained heterogeneous image registration network;

[0072] Step S160: Acquire a SAR image to be registered and an optical reference image, input the SAR image to be registered and the optical reference image into a trained heterogeneous image registration network, obtain transformation parameters, and perform an affine transformation on the SAR image to be registered according to the transformation parameters using a spatial transformation unit to obtain a registered SAR image.

[0073] In this embodiment, an integrated registration framework with enhanced position awareness is proposed. This registration framework is the heterogeneous image registration network described in the above method. Within this framework, the input image pairs are superimposed in the channel direction. A deep feature representation containing position information is obtained through a multi-scale feature extraction layer that fuses coordinate attention. To further improve the localization capability of the features, a multi-scale feature fusion layer (MSFF) is used to effectively aggregate the deep feature maps of different levels to obtain representative features that have both deep semantic representation capabilities and precise positioning information. Finally, a fully connected layer is used to map the geometric transformation parameters, and a spatial transformer (ST) module is used to perform an affine transformation on the image to be registered to obtain the registered image. This registration framework automatically learns the geometric relationship between the image pairs to be registered without feature point detection and directly outputs the registration results. This eliminates the need for iterative post-processing and the reliance on feature points, making it applicable to a variety of different image scene areas.

[0074] In this embodiment, steps S100 to S150 are a process of training a heterogeneous image registration network, and step S160 is a process of performing image registration using the trained heterogeneous image registration network.

[0075] Steps S100 and S110 are the steps for preparing training data. The registration data consists of pre-matched optical and SAR image pairs, so they cannot be used directly as training data. In this method, the SAR images in the matched pairs are subjected to random affine transformations to obtain training SAR images with transformed image angles. These training SAR images are then combined with the corresponding optical images to form training data pairs, generating multiple training data sets to train the neural network. The random affine transformation of the SAR images also generates true value labels for the transformation parameters, which are used in the subsequent calculation of the loss function.

[0076] In step S120, the two images in the training data pair are simultaneously input into the heterogeneous image registration network for training. The processing process of the training data pair in the network is as follows: Figure 2 shown.

[0077] In this embodiment, the heterogeneous image registration network includes a multi-scale feature extraction unit and a feature fusion unit that are connected in sequence and fuse coordinate attention.

[0078] In this embodiment, the SAR training image and the optical image in the training data pair are first superimposed in the channel direction to obtain a channel superimposed image, and then the channel superimposed image is input into the multi-scale feature extraction unit fused with coordinate attention for feature extraction.

[0079] In this embodiment, the multi-scale feature extraction unit fused with coordinate attention includes three sequentially connected feature extraction layers fused with coordinate attention: a first feature extraction layer, a second feature extraction layer, and a third feature extraction layer. The channel stacked image is input into the first feature extraction layer to extract a shallow feature image. The shallow feature image is input into the second feature extraction layer to extract a mid-level feature image. The mid-level feature image is input into the third feature extraction layer to extract a deep feature image.

[0080] In this embodiment, each feature extraction layer in the multi-scale feature extraction unit integrated with coordinate attention includes a Res2Net module and a coordinate attention mechanism (CA) module. The Res2Net module extracts high-level semantic features from the input feature images of different scales, which are then used by the coordinate attention mechanism to extract feature images sensitive to position information.

[0081] Specifically, the deep semantic information extracted by the convolutional network can, to a certain extent, adapt to the radiation differences between optical and SAR images, and the appropriate expansion of the feature receptive field is more conducive to the representation of the deep semantic features of heterogeneous images. However, as the number of convolutional network layers increases, the position information that feature localization relies on is lost. Therefore, in order to extract semantic features with strong generalization ability and position information, this method uses the Res2Net network structure with multi-scale feature representation capabilities to extract the deep semantic information of the input image pair. By integrating the coordinate attention mechanism (CA), the feature position information is explicitly modeled and accurately modeled to improve feature localization capabilities.

[0082] Res2Net is a variant of ResNet that achieves deep feature extraction by cascading multiple Res2Net modules, resulting in a pyramid-like approach to multi-scale feature extraction. The Res2Net module is a reconstruction of the bottleneck block used in the standard ResNet architecture. By constructing hierarchical feature groups within a single residual block and performing residual-like connections, it replaces the conventional 3×3 convolution kernel group, expanding the receptive field of the network layer.

[0083] like Figure 3 As shown on the left, specifically, after the input feature map undergoes 1×1 convolution, it is evenly divided into n feature subsets according to the number of channels, that is, each subset has the same spatial size and the number of channels is 1 / n. Except for x1, each x1 has a corresponding 3×3 convolution group H t , will pass through H t The output feature is recorded as y t , then the residual connection within the block can be expressed as: feature subset x t With the output y of the previous group t-1 Add, after convolution group H t Then we get the output y t , the corresponding mathematical expression is:

[0084]

[0085] After the above process is repeated several times, the output feature maps {y1,y2,...y n} are concatenated and then go through 1×1 convolution again to achieve full fusion of feature information. Due to the combination effect, y t With ratio x t Larger receptive field, increased equivalent receptive field, and fine-grained multi-scale features are generated after re-stitching.

[0086] After the above-mentioned Res2Net module is used for feature extraction, relatively abstract high-level semantic features can be obtained. However, due to operations such as convolution and pooling, generally speaking, the feature information contained in this semantic feature is not very sensitive to position, and the feature positioning accuracy is poor. However, unlike tasks such as classification and scene recognition, this method has a high demand for the precise location information of the features. Therefore, in order to drive the network to further extract important information that is conducive to the mapping of the input image to the spatial relationship and explore the spatial position relationship between features, the coordinate attention mechanism (CA) is introduced to improve the Res2Net module. The specific structure is as follows: Figure 3 As shown, its structure is as Figure 3 Shown on the right.

[0087] Specifically, in the CA module, by performing one-dimensional global adaptive pooling on the input feature x in the X and Y directions, the horizontal and vertical input features are aggregated into two independent direction-aware feature maps. This allows the long-range dependency along one direction to be captured while retaining the precise position information along the other direction. This process is called coordinate information embedding and can be understood as the feature encoding of channel relationships and long-range spatial dependencies. The corresponding mathematical expression is:

[0088]

[0089] In formula (2), are the outputs of the cth channel with height h and width w respectively. In order to effectively utilize the representation information obtained in the information embedding process, it is necessary to transform the above two direction perception feature maps to generate a coordinate attention map.

[0090] Furthermore, firstly transform the feature map z h 、z w Perform cascade and use 1×1 convolution transformation K1 to obtain the intermediate feature map f, that is:

[0091] f=δ(K1([z h ,z w ])) (3)

[0092] In formula (3), δ represents the nonlinear activation unit (ReLu), which is divided into and By unifying the number of channels through 1×1 convolution to make it the same as the number of input channels, the attention weights are obtained as follows:

[0093]

[0094] In formula (4), σ is the activation function (Sigmoid). h and gw As the attention weight, multiplied by the input feature x, the final output is obtained:

[0095]

[0096] By combining the multi-scale feature extraction capabilities of the Res2Net module and the position perception capabilities of coordinate attention, the feature extraction module can automatically learn the spatial location information of features between images, helping the network to more accurately capture the geometric transformation relationships between images.

[0097] Generally speaking, deep convolutional neural networks follow the approach of gradually reducing the size of the feature map space in their architecture design to obtain high-level semantic features for prediction. These features have a large receptive field and rich semantic information, but have low resolution and ignore the positioning information of image edges, corners, and other auxiliary image registration. If shallow feature prediction is adopted, although the resolution of the feature map is high, there is also more interference information in it. Especially for optical and SAR images with significant nonlinear grayscale differences, the invariant semantic information of the shallow features that can be used to register the corresponding heterogeneous images is very limited. At the same time, the receptive field of shallow features is small, which will lead to the loss of global information. Therefore, in this method, the output feature maps F1, F2, and F3 of Res2Netlayer1_CA (first feature extraction layer), Res2Net layer2_CA (second feature extraction layer), and Res2Net layer3_CA (third feature extraction layer) are combined into a feature pyramid and input into the multi-scale feature fusion module MSFF to aggregate the multi-scale local information and global information of the hierarchy. While expressing the deep semantic information of the heterogeneous image well, it captures a certain degree of position information and improves the registration accuracy. The specific structure is as follows: Figure 4 shown.

[0098] In the feature fusion unit, the shallow, mid-level, and deep feature images are processed differently, transforming the feature images of different scales to the same spatial size and the same number of channels as the shallow feature images. The feature images after spatial and channel transformations are added together, and the result of the addition is passed through the coordinate attention mechanism module and the fourth feature extraction layer for deep feature fusion to produce the feature fusion image.

[0099] Furthermore, the shallow feature map F1 has a larger spatial size and fewer channels, and contains shallow feature information that is beneficial for positioning, while the deep feature map F3 has a smaller spatial size and more channels, and contains discriminative semantic information. Therefore, in order to prevent feature information loss and resolution reduction during fusion, this paper performs different transformations on the feature maps F1, F2, and F3 to obtain feature maps with the same spatial size as F1 and the same number of channels as the deep feature map F3.

[0100] Specifically, the deep feature map F3 feature map uses 1×1 convolution to expand the number of channels to obtain the feature map The middle layer feature map F2 uses 1×1 convolution to expand the number of channels and then performs two times upsampling to obtain the feature map The deep feature map F3 first uses global adaptive average pooling to obtain global feature information, and then performs a 1×1 convolution transformation and expands it to obtain a feature map. After obtaining the above feature map, the most direct approach is to The gain of combining them and simply adding them together is very limited, or even worse, because there is a semantic gap between the feature maps of different scale levels, and the semantic information in them may interfere with each other. The resulting features after addition are input into the coordinate attention module, and then into the improved Res2Net layer4_CA (that is, the fourth feature extraction layer), so as to further deeply fuse the aforementioned hierarchical features. Among them, the structure of the fourth feature extraction layer is similar to the structure of each feature extraction layer in the multi-scale feature extraction unit that fuses the coordinate attention. Since the Res2Net module used for feature extraction in this method has a fine-grained multi-scale feature representation capability, accurate location information is embedded after fusing the coordinate attention, and the MSFF aggregates local and global information by fusing multi-scale feature maps of different levels, thereby ensuring the resolution of the output features and alleviating the loss of shallow positioning information. Through the combination of the two, the feature recognition ability and position perception ability are improved, providing an effective basis for the regression mapping of the spatial transformation relationship. The feature map output by the MSFF is subjected to global adaptive average pooling and then sent to the parameter regression network to obtain the final spatial transformation parameters. In step S130, the SAR training image is affine transformed according to the obtained training transformation parameters using the spatial transformation module to obtain the registered training image.

[0101] In step S140, constructing a loss function based on the training registered image, the training transformation parameters, the optical image in the training data pair, and the corresponding transformation parameter true value labels includes: constructing a similarity loss function based on a similarity measure between the training registered image and the optical image in the training data pair; constructing a four-corner shift loss function based on the training transformation parameters and the corresponding transformation parameter true value labels; and finally, obtaining a total loss function based on the similarity loss function and the four-corner shift loss function.

[0102] Specifically, when constructing the four-corner offset loss function based on the training transformation parameters and the corresponding transformation parameter true value labels, where the transformation parameter true value labels are obtained by performing random affine transformation on the SAR images in the registration dataset, the process is as follows: Assuming that the optical reference image is I R , the corresponding registered SAR image is I T , after random transformation Θ T→S The SAR image after S , where Θ R→S =Θ T→S , then the reference image I R and the image to be registered I S The spatial position relationship is:

[0103]

[0104] In formula (6), (u,v)∈I R ,(u′,v′)∈I S is the coordinate of the pixel in the corresponding image. When iteratively updating the network parameters, since the transformation parameter Θ is a mixture of rotation, translation and scale components, and these components have different dimensions, if we directly minimize the true value of the affine parameter Transformation parameters predicted by the network It is difficult to balance the rotation, scale and translation components and the network is difficult to converge.

[0105] Therefore, in this method, the affine parameters are parameterized by four points. Let the image I R The corner point is C R ={(x i ,y i ),i=1,2,3,4}, after affine transformation Θ R→S Then get the corresponding point coordinates C S ={(x i ′,y i ′), i=1,2,3,4}, the displacement C of the four corner points before and after the image transformation can be calculated 4pt :

[0106]

[0107] In formula (7), where Δx i =x i ′-x i .

[0108] Therefore, according to the affine parameters and Calculate separately and And construct the four corner point offset loss Loss based on the mean square error MSE 4pt , the four corner point offset loss formula is expressed as:

[0109]

[0110] The network is trained using the four-corner offset loss function shown in formula (8), which allows the network to converge to a better value and has a higher accuracy for natural image registration. However, since the target data of the registration task in this method is optical and SAR images with significant nonlinear radiation differences, in addition to the loss function Loss 4pt In addition, in this method, a loss function based on the similarity measurement between heterogeneous images is constructed. NMI To help the registration network converge better on the target data and improve the registration accuracy. When calculating the similarity measure for heterogeneous images to complete the registration task, mutual information MI can overcome the radiation difference between images and achieve better results. However, since MI is greatly affected by the size of the overlapping area of ​​the two images, the registration result cannot be guaranteed to be correct when the MI value reaches the maximum. In order to make the objective function more accurately reflect the relationship between the MI value and the registration effect, the normalized mutual information (NMI) is used as the similarity measure between images in this method, and its corresponding mathematical expression is:

[0111]

[0112] In formula (9), p A (a), p B (b) The marginal probability distribution of images A and B, respectively, p AB (a, b) is the joint probability distribution of images A and B. In addition, in order to further improve the robustness of the NMI value in reflecting the registration effect, the registration direction of the image is taken into account and the input image pair (I R ,I S ) to construct the loss function Loss NMI ,Right now:

[0113]

[0114] In formula (10), F θ is the image to be registered I S Towards optical image I R The transformation function of the registration, is the inverse transformation function. When registering images based on the transformation parameters output by the network, in this method, a differentiable spatial transformation module ST is used so that the gradient can be back-propagated.

[0115] The total loss function is the above four corner point offset loss Loss4pt Similarity loss between images NMI The sum is:

[0116] Loss total =Loss 4pt +Loss NMI (11)

[0117] In this embodiment, after constructing the total loss function, it is calculated and the parameters of the heterogeneous image registration network are adjusted based on the calculation results. Another pair of training data is then input into the heterogeneous image registration network, and the total loss function is recalculated based on the output results. This is done until the calculation results of the loss function converge, indicating that the source image registration network has also converged, resulting in a trained heterogeneous image registration network.

[0118] In this embodiment, after obtaining a trained heterogeneous image registration network, a spatial transformation unit can be embedded into the network. Thus, when using it to register SAR images, simply input the SAR image to be registered and the optical reference image into the trained heterogeneous image registration network to obtain a registered SAR image based on the optical reference image.

[0119] In the aforementioned position-aware optical and SAR image registration methods, an integrated registration network with enhanced position awareness is proposed. Compared to traditional point-based registration algorithms, this method eliminates the reliance on feature point detection and matching by directly regressing transformation parameters using deep features extracted from the network. This method demonstrates robustness across various image scene regions, particularly those with repetitive and weak textures, where traditional methods often fail. When using deep networks to nonlinearly map the spatial relationships between optical and SAR images, methods that solely regress transformation parameters using high-level features or simply concatenated multi-scale features often produce poor registration results. However, this method utilizes fine-grained multi-scale feature extraction combined with coordinate attention to embed position information. This method effectively fuses shallow features containing localization information with deep features containing high-level semantic information, improving feature localization accuracy and significantly enhancing registration performance. Experimental results demonstrate that this method achieves significantly higher registration accuracy than four other representative algorithms on publicly available high-resolution datasets. This method also maintains high registration speed while achieving improved accuracy, laying a solid foundation for optical and SAR image registration and localization tasks. This method can be extended to other types of remote sensing image registration and is applicable to a variety of scene areas.

[0120] It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0121] In one embodiment, Figure 5 As shown, a position-aware optical and SAR image registration device is provided, comprising: a registration data set acquisition module 200, a training data pair construction module 210, a training transformation parameter acquisition module 220, a training registered image acquisition module 230, a loss function construction module 240, a heterogeneous image registration network training module 250, and a heterogeneous image registration module 260, wherein:

[0122] A registration data set acquisition module 200 is configured to acquire a registration data set, wherein the registration data set includes a plurality of pre-aligned optical image and SAR image pairs;

[0123] A training data pair construction module 210 is configured to perform a random affine transformation on the SAR image in the registration dataset to obtain a SAR training image having an affine transformation relationship with the original SAR image and a true value label of the transformation parameter, and to form a training data pair with the SAR training image and the corresponding optical image;

[0124] A training transformation parameter obtaining module 220 is used to input the training data pair into a heterogeneous image registration network. In the heterogeneous image registration network, the training data is superimposed in the channel direction to obtain a channel superimposed image. The channel superimposed image is extracted with a multi-scale feature extraction unit that fuses coordinate attention to extract feature images of different scales. The feature fusion unit then fuses the feature images of different scales with multi-scale local information and global information to obtain a feature fusion image. The feature fusion image is then subjected to global adaptive average pooling and then obtained through a parameter regression unit. Training transformation parameters are obtained;

[0125] A training registered image obtaining module 230 is configured to perform an affine transformation on the SAR training image through a spatial transformation unit according to training transformation parameters to obtain a training registered image;

[0126] A loss function construction module 240 is configured to construct a loss function based on the training registered image, the training transformation parameters, the optical image in the training data pair, and the true value labels of the corresponding transformation parameters;

[0127] A heterogeneous image registration network training module 250 is used to train the heterogeneous image registration network according to the calculation result of the loss function until the loss function converges, thereby obtaining a trained heterogeneous image registration network;

[0128] The heterogeneous image registration module 260 is configured to obtain a SAR image to be registered and an optical reference image, input the SAR image to be registered and the optical reference image into a trained heterogeneous image registration network, obtain transformation parameters, and perform an affine transformation on the SAR image to be registered according to the transformation parameters using a spatial transformation unit to obtain a registered SAR image.

[0129] Regarding the specific limitations of the position-aware optical and SAR image registration device, please refer to the limitations of the position-aware optical and SAR image registration method above and will not be repeated here. Each module in the position-aware optical and SAR image registration device described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the modules can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so that the processor can call and execute the operations corresponding to each of the modules.

[0130] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for optical and SAR image registration based on position perception is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0131] Those skilled in the art will understand that Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0132] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0133] Acquire a registration dataset, wherein the registration dataset includes a plurality of pre-registered optical image and SAR image pairs;

[0134] Performing a random affine transformation on the SAR image in the registration dataset to obtain a SAR training image having an affine transformation relationship with the original SAR image and a true value label of the transformation parameter, and forming a training data pair with the SAR training image and the corresponding optical image;

[0135] Inputting the training data pair into a heterogeneous image registration network, wherein the training data are superimposed in the channel direction to obtain a channel superimposed image, extracting feature images of different scales from the channel superimposed image through a multi-scale feature extraction unit that fuses coordinate attention, fusing the feature images of different scales with multi-scale local information and global information through a feature fusion unit to obtain a feature fusion image, performing global adaptive average pooling on the feature fusion image, and obtaining training transformation parameters through a parameter regression unit;

[0136] The SAR training image is affine transformed by a spatial transformation unit according to training transformation parameters to obtain a training registered image;

[0137] A loss function is constructed based on the training registered image, the training transformation parameters, the optical image in the training data pair, and the true value labels of the corresponding transformation parameters;

[0138] Training the heterogeneous image registration network according to the calculation result of the loss function until the loss function converges, thereby obtaining a trained heterogeneous image registration network;

[0139] A SAR image to be registered and an optical reference image are obtained, the SAR image to be registered and the optical reference image are input into a trained heterogeneous image registration network to obtain transformation parameters, and an affine transformation is performed on the SAR image to be registered according to the transformation parameters by a spatial transformation unit to obtain a registered SAR image.

[0140] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0141] Acquire a registration dataset, wherein the registration dataset includes a plurality of pre-registered optical image and SAR image pairs;

[0142] Performing a random affine transformation on the SAR image in the registration dataset to obtain a SAR training image having an affine transformation relationship with the original SAR image and a true value label of the transformation parameter, and forming a training data pair with the SAR training image and the corresponding optical image;

[0143] Inputting the training data pair into a heterogeneous image registration network, wherein the training data are superimposed in the channel direction to obtain a channel superimposed image, extracting feature images of different scales from the channel superimposed image through a multi-scale feature extraction unit that fuses coordinate attention, fusing the feature images of different scales with multi-scale local information and global information through a feature fusion unit to obtain a feature fusion image, performing global adaptive average pooling on the feature fusion image, and obtaining training transformation parameters through a parameter regression unit;

[0144] The SAR training image is affine transformed by a spatial transformation unit according to training transformation parameters to obtain a training registered image;

[0145] A loss function is constructed based on the training registered image, the training transformation parameters, the optical image in the training data pair, and the true value labels of the corresponding transformation parameters;

[0146] Training the heterogeneous image registration network according to the calculation result of the loss function until the loss function converges, thereby obtaining a trained heterogeneous image registration network;

[0147] A SAR image to be registered and an optical reference image are obtained, the SAR image to be registered and the optical reference image are input into a trained heterogeneous image registration network to obtain transformation parameters, and an affine transformation is performed on the SAR image to be registered according to the transformation parameters by a spatial transformation unit to obtain a registered SAR image.

[0148] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0149] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0150] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for optical and SAR image registration based on position perception, characterized in that: The method comprises: Acquire a registration dataset, wherein the registration dataset includes a plurality of pre-registered optical image and SAR image pairs; Performing a random affine transformation on the SAR image in the registration dataset to obtain a SAR training image having an affine transformation relationship with the original SAR image and a true value label of the transformation parameter, and forming a training data pair with the SAR training image and the corresponding optical image; Inputting the training data pair into a heterogeneous image registration network, wherein the training data are superimposed in the channel direction to obtain a channel superimposed image, extracting feature images of different scales from the channel superimposed image through a multi-scale feature extraction unit that fuses coordinate attention, fusing the feature images of different scales with multi-scale local information and global information through a feature fusion unit to obtain a feature fusion image, performing global adaptive average pooling on the feature fusion image, and obtaining training transformation parameters through a parameter regression unit; The SAR training image is affine transformed by a spatial transformation unit according to training transformation parameters to obtain a training registered image; A loss function is constructed based on the training registered image, the training transformation parameters, the optical image in the training data pair, and the true value labels of the corresponding transformation parameters; Training the heterogeneous image registration network according to the calculation result of the loss function until the loss function converges, thereby obtaining a trained heterogeneous image registration network; A SAR image to be registered and an optical reference image are obtained, the SAR image to be registered and the optical reference image are input into a trained heterogeneous image registration network to obtain transformation parameters, and an affine transformation is performed on the SAR image to be registered according to the transformation parameters by a spatial transformation unit to obtain a registered SAR image.

2. The optical and SAR image registration method according to claim 1, characterized in that: The multi-scale feature extraction unit fused with coordinate attention includes three feature extraction layers fused with coordinate attention connected in sequence, namely a first feature extraction layer, a second feature extraction layer and a third feature extraction layer; Inputting the channel superposition image into the first feature extraction layer to extract a shallow feature image; Inputting the shallow feature image into the second feature extraction layer to extract a middle feature image; The middle-layer feature image is input into the third feature extraction layer to extract a deep-layer feature image.

3. The optical and SAR image registration method according to claim 2, characterized in that: Each feature extraction layer in the multi-scale feature extraction unit fused with coordinate attention includes a Res2Net module and a coordinate attention mechanism module; The feature images of different scales input to each feature extraction layer are extracted through the Res2Net module to obtain high-level semantic features; The high-level semantic features are then used to extract feature images of the focus position information through the coordinate attention mechanism module.

4. The optical and SAR image registration method according to claim 3, characterized in that: In the feature fusion unit: Performing different processing on the shallow feature image, the middle feature image, and the deep feature image respectively, and transforming the feature images of different scales to have the same spatial size as the shallow feature image and the same number of channels as the deep feature image; The feature images after spatial and channel transformation are added, and the addition results are sequentially passed through the coordinate attention mechanism module and the fourth feature extraction layer for deep feature fusion to obtain the feature fusion image.

5. The optical and SAR image registration method according to claim 4, characterized in that: The performing different processing on the shallow feature image, the middle feature image and the deep feature image respectively includes: use 1×1 Convolution expands the number of channels of the shallow feature image; use 1×1 After the convolution expands the number of channels of the middle-layer feature image, it performs a two-fold upsampling process; After processing the deep feature image using global adaptive average pooling, 1×1 Convolution transformation and expansion.

6. The optical and SAR image registration method according to any one of claims 1 to 5, characterized in that: The loss function is constructed based on the training registered image, the training transformation parameters, the optical image in the training data pair, and the corresponding transformation parameter true value labels, including: constructing a similarity loss function based on a similarity measure between the training registered image and the optical image in the training data pair; Constructing a four-corner shift loss function based on the training transformation parameters and the corresponding transformation parameter true value labels; A total loss function is obtained according to the similarity loss function and the four-corner offset loss function.

7. The optical and SAR image registration method according to claim 6, characterized in that: When obtaining the trained heterogeneous image registration network, embedding the spatial transformation unit into the network; After the SAR image to be registered and the optical reference image are input into the trained heterogeneous image registration network, a registered SAR image registered based on the optical reference image is obtained.

8. An optical and SAR image registration device based on position perception, characterized in that: The device comprises: A registration data set acquisition module, configured to acquire a registration data set, wherein the registration data set includes a plurality of pre-aligned optical image and SAR image pairs; a training data pair construction module, configured to perform a random affine transformation on the SAR image in the registration dataset to obtain a SAR training image having an affine transformation relationship with the SAR image and a true value label of the transformation parameter, and to form a training data pair with the SAR training image and the corresponding optical image; a training transformation parameter obtaining module, configured to input the training data pair into a heterogeneous image registration network, wherein the training data is superimposed in the channel direction to obtain a channel superimposed image, wherein the channel superimposed image is extracted with a multi-scale feature extraction unit that fuses coordinate attention to extract feature images of different scales, and then a feature fusion unit fuses the feature images of different scales with multi-scale local information and global information to obtain a feature fusion image, and then a parameter regression unit obtains training transformation parameters after performing global adaptive average pooling on the feature fusion image; A module for obtaining a registered training image, configured to perform an affine transformation on the SAR training image through a spatial transformation unit according to training transformation parameters to obtain a registered training image; A loss function construction module is used to construct a loss function based on the training registered image, the training transformation parameters, the optical image in the training data pair, and the true value labels of the corresponding transformation parameters; A heterogeneous image registration network training module is used to train the heterogeneous image registration network according to the calculation result of the loss function until the loss function converges, thereby obtaining a trained heterogeneous image registration network; The heterogeneous image registration module is used to obtain a SAR image to be registered and an optical reference image, input the SAR image to be registered and the optical reference image into a trained heterogeneous image registration network, obtain transformation parameters, and perform an affine transformation on the SAR image to be registered according to the transformation parameters through a spatial transformation unit to obtain a registered SAR image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to claim 7 is implemented.

Images