Optical image and synthetic aperture radar image registration method, device, electronic device, storage medium and computer program product

Through phase consistency transformation and amplitude response image operation, combined with the improved LoFTR model and asymmetry consistent sampling algorithm, the problem of nonlinear radiation distortion differences in optical images and synthetic aperture radar image registration is solved, achieving efficient and robust Image registration effect.

CN118735975BActive Publication Date: 2024-11-12INST OF AUTOMATION CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411228063.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-03
Publication Date
2024-11-12
Estimated Expiration
2044-09-03

AI Technical Summary

Technical Problem

The prior art is difficult to achieve effective registration between optical images and synthetic aperture radar images, mainly due to the large differences in nonlinear radiation distortion, resulting in poor registration results.

Method used

Phase consistency transformation and amplitude response image operation are adopted, and image registration is improved through an early exit mechanism and an adaptive window attention mechanism to improve matching accuracy and robustness.

Benefits of technology

The matching accuracy and speed between optical images and synthetic aperture radar images are improved, and the robustness and stability of nonlinear radiation distortion is achieved, and better registration results are obtained.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118735975B_ABST
    Figure CN118735975B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a registration method, device, electronic device, storage medium and computer program product for optical images and synthetic aperture radar images, including: obtaining an optical image to be registered and a synthetic aperture radar image to be registered; performing phase consistency transformation and amplitude response image generation operations on the optical image to be registered and the synthetic aperture radar image to be registered respectively; obtaining an initial matching point set based on an improved LoFTR model; iteratively screening the initial matching point set using a progressive consistent sampling algorithm; and obtaining a target registration relationship based on a target matching point set. In this way, due to the use of phase consistency transformation, amplitude response image generation operations, an improved LoFTR model and iterative screening, the matching accuracy and matching speed between the optical image and the synthetic aperture radar image can be improved, and robust matching can be achieved. That is, the present disclosure has good robustness and stability to nonlinear radiation distortion in matching optical images with synthetic aperture radar images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more particularly, to a registration method, device, electronic device, storage medium and computer program product for optical image and synthetic aperture radar image. Background Art

[0002] Remote sensing image matching refers to extracting reliable matching point pairs from images with overlapping areas, and multi-source image registration usually includes image registration of different sensors, image registration of different platforms, image registration of different bands, and image registration of different resolutions, etc. Compared with single-source images, multi-source images contain complementary information. For example, Synthetic Aperture Radar (SAR) images are mainly obtained by transmitting radar signals and collecting the backscattered signals reaching the sensor. They have all-weather ground observation capabilities, but their texture information is not very clear; while optical images have rich color and texture information, but they are greatly affected by factors such as lighting and weather. In order to effectively use this information, it is necessary to align optical images and SAR images.

[0003] However, the image structural features of heterogeneous images vary greatly, for example, there are large differences in image intensity, geometric structure, texture, etc., which makes it difficult to learn appropriate similar features from them. For example, the current registration method is very sensitive to the difference in nonlinear radiation distortion between optical images and SAR images, and it is difficult to obtain a good registration result. Summary of the invention

[0004] The present disclosure provides an optical image and a synthetic aperture radar image registration method, device, electronic device, storage medium and computer program product to at least solve the problem in the above-mentioned related technologies that it is difficult to obtain a relatively good registration result between the optical image and the SAR image.

[0005] According to a first aspect of an embodiment of the present disclosure, a registration method for an optical image and a synthetic aperture radar image is provided, comprising: acquiring an optical image to be registered and a synthetic aperture radar image to be registered; performing phase consistency transformation and amplitude response image generation operations on the optical image to be registered and the synthetic aperture radar image to be registered, respectively, to obtain an optical amplitude response image corresponding to the optical image to be registered and a synthetic aperture radar amplitude response image corresponding to the synthetic aperture radar image to be registered; acquiring an initial matching point set between the optical amplitude response image and the synthetic aperture radar amplitude response image based on an improved LoFTR model, wherein the improved LoFTR model is a model obtained by adding an early exit function and an adaptive window function to an original LoFTR model; iteratively screening the initial matching point set using a progressive consistent sampling algorithm to obtain a target matching point set; and obtaining a target registration relationship based on the target matching point set.

[0006] Optionally, the operations of performing phase consistency transformation and generating amplitude response images respectively on the optical image to be registered and the synthetic aperture radar image to be registered include: performing convolution calculations on a plurality of Log-Gabor wavelet filters and a two-dimensional image respectively to obtain a response component corresponding to each Log-Gabor wavelet filter in the plurality of Log-Gabor wavelet filters, wherein the two-dimensional image includes the optical image to be registered and the synthetic aperture radar image to be registered, and each Log-Gabor wavelet filter corresponds to its own scale and direction; based on the response component, obtaining an amplitude component and phase component; based on the amplitude component and the phase component, obtaining the sum of the amplitude components at multiple scales in any direction of multiple directions for each pixel point in the two-dimensional image, and generating an amplitude component sum sequence of the pixel point arranged in sequence in the multiple directions based on the amplitude component sum, wherein the multiple directions are directions corresponding to the multiple Log-Gabor wavelet filters respectively, and the multiple scales are scales corresponding to the multiple Log-Gabor wavelet filters respectively; based on the maximum value of the amplitude component sum sequence corresponding to each pixel point, obtaining the optical amplitude response image and the synthetic aperture radar amplitude response image.

[0007] Optionally, the acquiring the initial matching point set between the optical amplitude response image and the synthetic aperture radar amplitude response image based on the improved LoFTR model includes: using a feature extraction network included in the improved LoFTR model to perform feature extraction on the optical amplitude response image and the synthetic aperture radar amplitude response image to obtain an optical coarse matching feature map and an optical fine matching feature map corresponding to the optical amplitude response image, and obtaining a synthetic aperture radar coarse matching feature map and a synthetic aperture radar fine matching feature map corresponding to the synthetic aperture radar amplitude response image; using the LoFTR module included in the improved LoFTR model to process the optical coarse matching feature map and the synthetic aperture radar coarse matching feature map to obtain a processing result; based on the processing result, obtaining a confidence matrix; based on the confidence matrix, obtaining a coarse matching prediction result; based on the coarse matching prediction result, the optical fine matching feature map and the synthetic aperture radar fine matching feature map, acquiring the initial matching point set.

[0008] Optionally, the LoFTR module includes multiple Transformer layers, and the image registration method further includes: for each Transformer layer in the multiple Transformer layers, determining whether the difference between the output result of each Transformer layer and the final output result after passing through the multiple Transformer layers is less than a preset difference threshold; if the difference between the output result of each Transformer layer and the final output result is less than the preset difference threshold, terminating the processing process of the multiple Transformer layers.

[0009] Optionally, the LoFTR module includes multiple Transformer layers, and the LoFTR module included in the improved LoFTR model is used to process the optical coarse matching feature map and the synthetic aperture radar coarse matching feature map to obtain a processing result, including: for some Transformer layers among the multiple Transformer layers, determining relevant information of feature points input into the part of the Transformer layers, wherein the feature points are feature points included in the optical coarse matching feature map and the synthetic aperture radar coarse matching feature map; based on the relevant information of the feature points, determining a target window corresponding to the part of the Transformer layers, wherein the target window is a window for attention calculation; for each feature point in the optical coarse matching feature map and the synthetic aperture radar coarse matching feature map, processing the feature points centered on the feature point and located in the target window to obtain the processing result.

[0010] Optionally, the initial matching point set is iteratively screened using a progressive consistent sampling algorithm to obtain a target matching point set, including: for each iteration process, the following operations are performed: k matching points are randomly selected from the initial matching point set; a model is fitted using the k matching points, and the fitted model is applied to the initial matching point set to obtain a fitting error for each matching point in the initial matching point set; a consistency metric is calculated based on the fitting error of each matching point in the initial matching point set, wherein the smaller the fitting error of each matching point, the higher the consistency metric; when the consistency metric is greater than or equal to a preset threshold, the k matching points selected in this iteration are used as the target matching point set; otherwise, the next iteration of this iteration is entered until the number of iterations reaches a preset number.

[0011] Optionally, the acquiring the optical image to be registered and the synthetic aperture radar image to be registered includes: selecting an original image pair under the same scene from a data set, wherein the original image pair includes an original optical image and an original synthetic aperture radar image; performing histogram equalization processing and Lee filtering processing on the original synthetic aperture radar image to obtain a processed synthetic aperture radar image; using the original optical image as the optical image to be registered, and using the processed synthetic aperture radar image as the synthetic aperture radar image to be registered.

[0012] According to a second aspect of an embodiment of the present disclosure, a registration device for an optical image and a synthetic aperture radar image is provided, comprising: an image acquisition module to be registered, configured to acquire an optical image to be registered and a synthetic aperture radar image to be registered; an amplitude response image acquisition module, configured to perform phase consistency transformation and amplitude response image generation operations on the optical image to be registered and the synthetic aperture radar image to be registered, respectively, to obtain an optical amplitude response image corresponding to the optical image to be registered and a synthetic aperture radar amplitude response image corresponding to the synthetic aperture radar image to be registered; an initial matching point set acquisition module, configured to acquire an initial matching point set between the optical amplitude response image and the synthetic aperture radar amplitude response image based on an improved LoFTR model, wherein the improved LoFTR model is a model obtained by adding an early exit function and an adaptive window function to the original LoFTR model; an iterative screening module, configured to perform iterative screening on the initial matching point set using a progressive consistent sampling algorithm to obtain a target matching point set; and a registration relationship acquisition module, configured to acquire a target registration relationship based on the target matching point set.

[0013] Optionally, the amplitude response image acquisition module is configured to: perform convolution calculations on multiple Log-Gabor wavelet filters and a two-dimensional image respectively to obtain a response component corresponding to each Log-Gabor wavelet filter in the multiple Log-Gabor wavelet filters, wherein the two-dimensional image includes the optical image to be registered and the synthetic aperture radar image to be registered, and each Log-Gabor wavelet filter corresponds to its own scale and direction; based on the response component, obtain an amplitude component and a phase component; based on the amplitude component and the phase component, obtain the sum of amplitude components at multiple scales in any direction of multiple directions for each pixel point in the two-dimensional image, and based on the amplitude component sum, generate an amplitude component sum sequence of the pixel point arranged in sequence in the multiple directions, wherein the multiple directions are the directions corresponding to the multiple Log-Gabor wavelet filters, and the multiple scales are the scales corresponding to the multiple Log-Gabor wavelet filters; based on the maximum value of the amplitude component sum sequence corresponding to each pixel point, obtain the optical amplitude response image and the synthetic aperture radar amplitude response image.

[0014] Optionally, the initial matching point set acquisition module is configured to: use the feature extraction network included in the improved LoFTR model to perform feature extraction on the optical amplitude response image and the synthetic aperture radar amplitude response image to obtain an optical coarse matching feature map and an optical fine matching feature map corresponding to the optical amplitude response image, and obtain a synthetic aperture radar coarse matching feature map and a synthetic aperture radar fine matching feature map corresponding to the synthetic aperture radar amplitude response image; use the LoFTR module included in the improved LoFTR model to process the optical coarse matching feature map and the synthetic aperture radar coarse matching feature map to obtain a processing result; based on the processing result, obtain a confidence matrix; based on the confidence matrix, obtain a coarse matching prediction result; based on the coarse matching prediction result, the optical fine matching feature map and the synthetic aperture radar fine matching feature map, obtain the initial matching point set.

[0015] Optionally, the LoFTR module includes multiple Transformer layers, and the image registration device also includes: a determination module, configured to determine, for each Transformer layer in the multiple Transformer layers, whether the difference between the output result of each Transformer layer and the final output result after passing through the multiple Transformer layers is less than a preset difference threshold; a termination module, configured to terminate the processing process of the multiple Transformer layers when the difference between the output result of each Transformer layer and the final output result is less than the preset difference threshold.

[0016] Optionally, the LoFTR module includes multiple Transformer layers, and the initial matching point set acquisition module is configured to: determine, for some Transformer layers among the multiple Transformer layers, relevant information of feature points input into the part of the Transformer layers, wherein the feature points are feature points contained in the optical coarse matching feature map and the synthetic aperture radar coarse matching feature map; determine a target window corresponding to the part of the Transformer layers based on the relevant information of the feature points, wherein the target window is a window for attention calculation; for each feature point in the optical coarse matching feature map and the synthetic aperture radar coarse matching feature map, process the feature points centered on the feature point and located in the target window to obtain the processing result.

[0017] Optionally, the iterative screening module is configured to: for each iteration process, perform the following operations: randomly select k matching points from the initial matching point set; fit a model using the k matching points, and apply the fitted model to the initial matching point set to obtain the fitting error of each matching point in the initial matching point set; calculate a consistency metric based on the fitting error of each matching point in the initial matching point set, wherein the smaller the fitting error of each matching point, the higher the consistency metric; when the consistency metric is greater than or equal to a preset threshold, use the k matching points selected in this iteration as the target matching point set; otherwise, enter the next iteration of this iteration until the number of iterations reaches a preset number.

[0018] Optionally, the image acquisition module for registration is configured to: select an original image pair of the same scene from a data set, wherein the original image pair includes an original optical image and an original synthetic aperture radar image; perform histogram equalization processing and Lee filtering processing on the original synthetic aperture radar image to obtain a processed synthetic aperture radar image; use the original optical image as the optical image to be registered, and use the processed synthetic aperture radar image as the synthetic aperture radar image to be registered.

[0019] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the optical image and synthetic aperture radar image registration method according to the present disclosure.

[0020] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method for aligning optical images and synthetic aperture radar images according to the present disclosure.

[0021] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the registration method for an optical image and a synthetic aperture radar image according to the present disclosure.

[0022] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0023] In the present disclosure, the image to be registered is subjected to phase consistency transformation and amplitude response image generation operations, and an improved LoFTR model and asymptotically consistent sampling algorithm are used for image registration, which can improve the matching accuracy and matching speed between the optical image and the synthetic aperture radar image, and achieve robust matching. That is, the present disclosure has good robustness and stability to the nonlinear radiation distortion of the matching of the optical image and the synthetic aperture radar image, and can efficiently obtain good registration results.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0026] Figure 1 is a flow chart illustrating a registration method of an optical image and a synthetic aperture radar image according to an exemplary embodiment of the present disclosure;

[0027] Figure 2 is a schematic diagram showing a process of acquiring an initial matching point set by using an optical amplitude response image and a synthetic aperture radar amplitude response image according to an exemplary embodiment of the present disclosure;

[0028] Figure 3 is a specific implementation flow chart showing registration of an optical image and a SAR image according to an exemplary embodiment of the present disclosure;

[0029] Figure 4 is a schematic diagram showing a comparison between the time consumed for running a test set in the related art and the time consumed for running a test set in the present disclosure;

[0030] Figure 5is a schematic diagram showing an example of an original SAR image and an original visible light image according to an exemplary embodiment of the present disclosure;

[0031] Figure 6 is a schematic diagram showing an example of a SAR image and an original visible light image after histogram equalization and Lee filtering according to an exemplary embodiment of the present disclosure;

[0032] Figure 7 is a schematic diagram showing an example of a SAR image that has undergone phase-consistency transformation and an amplitude response image generation operation and a visible light image that has undergone phase-consistency transformation and an amplitude response image generation operation according to an exemplary embodiment of the present disclosure;

[0033] Figure 8 is a schematic diagram showing an example of a registration result of a visible light image that has been inversely transformed after registration and an experimental image according to an exemplary embodiment of the present disclosure;

[0034] Fig. 9 is a schematic diagram showing another example of an original SAR image and an original visible light image according to an exemplary embodiment of the present disclosure;

[0035] Fig.10 is a schematic diagram showing another example of a SAR image and an original visible light image after histogram equalization and Lee filtering according to an exemplary embodiment of the present disclosure;

[0036] Fig.11 is a schematic diagram showing another example of a SAR image subjected to phase-consistency transformation and amplitude response image generation operations and a visible light image subjected to phase-consistency transformation and amplitude response image generation operations according to an exemplary embodiment of the present disclosure;

[0037] Fig.12 is a schematic diagram showing another example of a registration result of a visible light image that has been inversely transformed after registration and an experimental image according to an exemplary embodiment of the present disclosure;

[0038] Fig.13 is a block diagram showing an apparatus for registering an optical image and a synthetic aperture radar image according to an exemplary embodiment of the present disclosure;

[0039] Fig.14 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0042] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three types of parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three types of parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0043] In the related technology, the methods of image registration are generally divided into three types, namely feature-based methods, region-based methods and deep learning-based methods. The feature-based method mainly performs feature detection first, which is usually divided into point feature detection, line feature detection and surface feature detection. Then, the hand-made descriptors are used to find matching points, and the local correspondence between images is established based on the matching points. The more typical methods include scale-invariant feature transform (SIFT), accelerated robust feature algorithm (SURF) and ORB algorithm; the region-based method usually uses the grayscale information on the preset template window as a benchmark to define a similarity measure to calculate the similarity between windows. Traditional similarity measures include sum of squared differences (SSD) and normalized cross correlation (NCC) algorithm; the deep learning-based method can extract more abstract and advanced features, and is more robust to speckle noise and image appearance changes. In addition, deep convolutional networks are learnable and trainable, which gives them a great advantage in the field of image registration.

[0044] However, as mentioned above, the image structural features of heterogeneous images vary greatly, for example, there are large differences in image appearance, structure, etc., which makes it difficult to learn appropriate similar features from them. For example, the current registration method is very sensitive to the difference in nonlinear radiation distortion between optical images and SAR images, making it difficult to obtain a good registration result.

[0045] In order to solve the above-mentioned problems existing in the related art, the registration method, device, electronic device, storage medium and computer program product of optical image and synthetic aperture radar image provided by the present disclosure perform phase consistency transformation and generate amplitude response image operation on the image to be registered, and adopt the improved LoFTR model and progressive consistent sampling algorithm for image registration, which can improve the matching accuracy and matching speed between the optical image and the synthetic aperture radar image, and realize robust matching. That is, the present disclosure has good robustness and stability to the nonlinear radiation distortion of matching the optical image and the synthetic aperture radar image, and can efficiently obtain good registration results.

[0046] Figure 1 is a flow chart illustrating a registration method of an optical image and a synthetic aperture radar image according to an exemplary embodiment of the present disclosure.

[0047] Reference Figure 1 In step 101, an optical image to be registered and a synthetic aperture radar image to be registered may be acquired.

[0048] According to an exemplary embodiment of the present disclosure, an original image pair of the same scene may be selected from a data set, wherein the original image pair may include an original optical image and an original synthetic aperture radar image. Then, the original synthetic aperture radar image may be subjected to histogram equalization processing and Lee filtering processing to obtain a processed synthetic aperture radar image. Next, the original optical image may be used as an optical image to be registered, and the processed synthetic aperture radar image may be used as a synthetic aperture radar image to be registered.

[0049] In step 102, phase consistency transformation and amplitude response image generation operations can be performed on the optical image to be registered and the synthetic aperture radar image to be registered, respectively, so as to obtain an optical amplitude response image corresponding to the optical image to be registered and a synthetic aperture radar amplitude response image corresponding to the synthetic aperture radar image to be registered.

[0050] According to an exemplary embodiment of the present disclosure, multiple Log-Gabor wavelet filters can be convolved with a two-dimensional image to obtain a response component corresponding to each Log-Gabor wavelet filter in the multiple Log-Gabor wavelet filters, wherein the two-dimensional image can include an optical image to be registered and a synthetic aperture radar image to be registered, and each Log-Gabor wavelet filter can correspond to its own scale and direction. Next, based on the above response components, an amplitude component and a phase component can be obtained.

[0051] Then, based on the amplitude component and the phase component, the sum of the amplitude components at multiple scales in any direction of multiple directions for each pixel in the two-dimensional image can be obtained, and based on the sum of the amplitude components, a sequence of the amplitude components and the sum of the pixel points arranged in sequence in multiple directions can be generated, wherein the multiple directions can be directions corresponding to multiple Log-Gabor wavelet filters, and the multiple scales can be scales corresponding to multiple Log-Gabor wavelet filters.

[0052] Next, based on the maximum value of the amplitude component and sequence corresponding to each pixel, the optical amplitude response image and the synthetic aperture radar amplitude response image can be obtained. Exemplarily, the maximum value of the amplitude component and sequence corresponding to each pixel can be used as the value of the corresponding pixel in the amplitude response image to obtain the optical amplitude response image and the synthetic aperture radar amplitude response image.

[0053] Specifically, in the present disclosure, multiple 2D Log-Gabor filters of different scales and directions can be used to obtain a phase consistency PC map. The function of the 2D Log-Gabor filter can be defined as follows:

[0054]

[0055] in, can be expressed in log polar coordinates, s represents the scale of the 2D Log-Gabor filter, o represents the direction of the 2D Log-Gabor filter, is the center frequency of the 2D Log-Gabor filter, yes bandwidth, yes bandwidth.

[0056] In the spatial domain, the 2D Log-Gabor filter can be expressed as:

[0057]

[0058] in, represents the real part of the even-symmetric Log-Gabor filter, Represents the imaginary part of an odd-symmetric Log-Gabor filter.

[0059] Convolution of two-dimensional image signals with even-symmetric and odd-symmetric Log-Gabor wavelet filters of any scale and any direction The response component can be obtained and Then, the corresponding amplitude component can be obtained in scale s and direction o through the obtained response component and phase component , the formula can be specifically as follows:

[0060]

[0061] For each direction o, the Log-Gabor amplitude component and , the amplitude components and sequence of the Log-Gabor in different directions can be obtained by arranging them in order . is the number of directions, Represents the different channels of a sequence of Log-Gabor amplitude components and.

[0062] For each pixel point contained in the optical image to be registered and the synthetic aperture radar image to be registered, the corresponding amplitude component and sequence of each pixel point can be obtained. , and then we can get the sequence of amplitude components and corresponding to each pixel The maximum value of . Then, based on the maximum value of the sequence of the sum of the amplitude components corresponding to each pixel point contained in the optical image to be registered, an amplitude response image corresponding to the optical image to be registered can be generated; and, based on the maximum value of the sequence of the sum of the amplitude components corresponding to each pixel point contained in the synthetic aperture radar image to be registered, an amplitude response image corresponding to the synthetic aperture radar image to be registered can be generated. In this way, by performing the phase consistency transformation and the operation of generating the amplitude response image, it is possible to solve the problem that the image registration is sensitive to nonlinear radiation distortion, that is, the phase consistency transformation and the operation of generating the amplitude response image provided by the present disclosure have good robustness and stability to the nonlinear radiation distortion of matching the optical image and the synthetic aperture radar image, which can help to obtain better registration results.

[0063] In step 103, an initial matching point set between the optical amplitude response image and the synthetic aperture radar amplitude response image can be obtained based on an improved (Detector-Free Local Feature Matching with Transformers, LoFTR) model, wherein the improved LoFTR model can be a model obtained by adding an early exit function and an adaptive window function to the original LoFTR model. That is, in the present disclosure, the Transformer early exit mechanism and the adaptive window attention mechanism can be used to improve the original LoFTR model to obtain an improved LoFTR model.

[0064] According to an exemplary embodiment of the present disclosure, the feature extraction network included in the improved LoFTR model can be used to perform feature extraction on the above-mentioned optical amplitude response image and synthetic aperture radar amplitude response image to obtain an optical coarse matching feature map and an optical fine matching feature map corresponding to the optical amplitude response image, and a synthetic aperture radar coarse matching feature map and a synthetic aperture radar fine matching feature map corresponding to the synthetic aperture radar amplitude response image can be obtained.

[0065] Then, the LoFTR module included in the improved LoFTR model can be used to process the optical coarse matching feature map and the synthetic aperture radar coarse matching feature map to obtain the processing result. Next, the confidence matrix can be obtained based on the processing result. Then, we can use the confidence matrix Next, an initial matching point set can be obtained based on the coarse matching prediction result, the optical fine matching feature map, and the synthetic aperture radar fine matching feature map.

[0066] Specifically, an image pair consisting of an optical amplitude response image and a synthetic aperture radar amplitude response image can be input into the improved LoFTR model, and a feature extraction network included in the improved LoFTR model can be used to extract features from the input image pair to obtain an optical coarse matching feature map and an optical fine matching feature map corresponding to the optical amplitude response image, and a synthetic aperture radar coarse matching feature map and a synthetic aperture radar fine matching feature map corresponding to the synthetic aperture radar amplitude response image can be obtained. The above-mentioned "feature extraction network" can be, but is not limited to: a convolutional neural network ResNet + a feature pyramid network (Feature Pyramid Network, FPN).

[0067] Next, the above optical coarse matching feature map and synthetic aperture radar coarse matching feature map can be converted into a one-dimensional vector, and the position code can be added, and then it can be input into the LoFTR module contained in the improved LoFTR model for processing. The LoFTR module can include multiple Transformer layers, and these Transformer layers can be multiple continuous self-attention layers and cross-attention layers. Then, the processing results output by the LoFTR module can be matched using a differentiable matching layer to obtain a confidence matrix Next, we can use the confidence matrix , select the confidence level above the threshold The matching points can be further processed by the Mutual Nearest Neighbor (MNN) criterion to obtain the coarse matching prediction results.

[0068] Then, for each selected coarse-level matching prediction, a local window of size w × w can be cropped from the optical fine matching feature map and the synthetic aperture radar fine matching feature map. Each cropped local window continues to refine the coarse-level matching prediction result, i.e., the coarse-level matching relationship, to the sub-pixel level through a layer of self-attention and a layer of cross-attention as the final matching prediction result.

[0069] Among them, the loss function of the model can be as follows:

[0070]

[0071] in, It refers to the coarse matching error, which can be expressed by the following formula:

[0072]

[0073] in, is the confidence matrix, is the ground truth label, i and j refer to the two matching pixels in the coarse matching prediction, i is the pixel on the optical coarse matching feature map, and j is the pixel on the synthetic aperture radar coarse matching feature map.

[0074] It refers to the fine matching error, which can be specifically expressed by the following formula:

[0075]

[0076] in, It refers to the point mapped by the above pixel point i on the optical precise matching feature map. Yes The corresponding points obtained by the above model on the synthetic aperture radar precise matching feature map are: is the ground truth label, is also the ground truth, is the standard deviation.

[0077] It refers to the confidence error, which can be specifically expressed by the following formula:

[0078]

[0079] Among them, CE represents the cross entropy loss function, and its formula can be specifically:

[0080]

[0081] Refers to each pixel on the optical image The corresponding prediction confidence is It refers to the prediction confidence corresponding to each pixel point j on the SAR image. c 0 refers to the similarity between the current optical image feature point matching result and the final optical image feature point matching result. c 1 refers to the similarity between the current SAR image feature point matching result and the final SAR image feature point matching result.

[0082] Figure 2 FIG. 1 is a schematic diagram showing a process of obtaining an initial matching point set using an optical amplitude response image and a synthetic aperture radar amplitude response image according to an exemplary embodiment of the present disclosure. Figure 2 ,This process can include five parts in total, namely “full image feature extraction”, “LoFTR module processing”, “coarse level matching acquisition”, “small LoFTR module processing” and “fine level matching acquisition”.

[0083] Specifically, for the "full-image feature extraction" part, the improved LoFTR model can use the standard convolutional neural network ResNet (CNN) and combine it with the feature pyramid network (FPN) to extract the optical amplitude response image. and synthetic aperture radar amplitude response image Multi-level features are extracted from the image. These features can be divided into a coarse level and a fine level. The "coarse level" can refer to the optical coarse matching feature map and the synthetic aperture radar coarse matching feature map, and the coarse level can correspond to 1 / 8 of the original image size; the "fine level" can refer to the optical fine matching feature map and the synthetic aperture radar fine matching feature map, and the fine level can correspond to 1 / 2 of the original image size.

[0084] For the "LoFTR module processing" part, the coarse-level feature map obtained above can be input into the LoFTR module included in the improved LoFTR model, that is, the obtained optical coarse matching feature map and synthetic aperture radar coarse matching feature map can be input into the LoFTR module. As mentioned above, the LoFTR module can include multiple Transformer layers, and these Transformer layers can be multiple consecutive self-attention layers and cross-attention layers. The self-attention layer can be used to process the same set of features, that is, the self-attention layer can be used to process features from the same image; and the cross-attention layer can be used to process features from different sources, that is, the cross-attention layer can be used to process features from different images.

[0085] In the LoFTR module, these two types of layers, i.e. these two attention mechanisms, can be used alternately multiple times. The self-attention layer, i.e. the self-attention mechanism, can learn the relationship between each feature and all other features in the same image; while the cross-attention layer, i.e. the cross-attention mechanism, can effectively match features from different sources, i.e. the cross-attention mechanism can effectively match features from different images.

[0086] It should be noted that in the LoFTR module, the Transformer layer early exit mechanism can be used, that is, the overall confidence of feature point matching after each layer can be detected. When the confidence reaches a certain threshold, the early exit mechanism can be triggered to avoid processing redundant Transformer layers.

[0087] For the "coarse-level matching acquisition" part, the score matrix S between the transformed features can be calculated through two different differentiable matching layers for the processing results output by the LoFTR module. The score matrix S can be obtained by calculating the similarity between the two transformed feature vectors. Then, the confidence matrix P is obtained by performing a softmax regression operation on the score matrix S. C Next, we can use the confidence matrix , select the confidence level above the threshold The matching points can be further processed by the mutual nearest neighbor (MNN) criterion to filter out possible abnormal coarse matches, and then the coarse-level matching prediction results can be obtained.

[0088] Regarding the “small LoFTR module processing” part, after obtaining the coarse-level matching prediction results, it is necessary to refine the coarse-level matching prediction results from coarse to fine to adapt them to the original image resolution. Specifically, for each coarse matching prediction result, first, its position can be located in the fine-level feature map, that is, its position can be located in the optical fine matching feature map and the synthetic aperture radar fine matching feature map. Then, local windows of size w × w can be cropped in the optical fine matching feature map and the synthetic aperture radar fine matching feature map, respectively. Next, a smaller LoFTR module can be used to process the features in each local window obtained above. After the transformation, two transformed local feature maps A and B can be obtained. is a preset value, exemplary, The value of can be 1.

[0089] For the "fine-level matching acquisition" part, the center vector of the local feature map A obtained above can be correlated with all the vectors in the local feature map B to generate a heat map. Then, by performing expectation calculation on the probability distribution, we can get The final position with sub-pixel accuracy on the image can be obtained, that is, a fine-level matching result such as the initial matching point set can be obtained. It can be seen that the processing process of the improved LoFTR model is an image registration process from coarse matching to fine matching, which is crucial for obtaining high-quality registration relationships on featureless areas with low texture or repeated patterns, and makes the calculation more lightweight based on the original model.

[0090] According to an exemplary embodiment of the present disclosure, the above-mentioned LoFTR module may include multiple Transformer layers, and for some of the multiple Transformer layers, relevant information of feature points of the input part of the Transformer layers may be determined, wherein the feature points may be feature points contained in the optical coarse matching feature map and the synthetic aperture radar coarse matching feature map. Then, based on the relevant information of the feature points, the target window corresponding to the part of the Transformer layers may be determined, wherein the target window is a window for attention calculation. Next, for each feature point in the optical coarse matching feature map and the synthetic aperture radar coarse matching feature map, the feature points centered on the feature point and located in the target window may be processed to obtain a processing result.

[0091] That is, in the present disclosure, an adaptive window attention mechanism can be added to the LoFTR module. For example, in the first two Transformer layers, the traditional linear attention mechanism can be kept unchanged, while in the subsequent Transformer layers, the adaptive window attention mechanism can be adopted.

[0092] The adaptive window attention mechanism can automatically adjust the window size according to the characteristics of the input sequence of the Transformer layer, thereby limiting the range of queries and keys, and calculating the attention weight within the window. Furthermore, the window size can be obtained by learning for a linear layer. The linear layer can transform a tensor of all 1s to map it to a real number to represent the window size. Then, this real number can be restricted and rounded to obtain the final window size. The introduction of the adaptive window attention mechanism can make the model more efficient, flexible, and adaptable, and can better handle complex input data. Moreover, compared with the traditional linear attention mechanism, the adaptive window attention mechanism can not only reduce the amount of calculation, but also dynamically adjust the window size according to the information learned in each layer, which can significantly reduce the computational overhead while maintaining the model performance.

[0093] According to an exemplary embodiment of the present disclosure, as described above, the LoFTR module may include multiple Transformer layers. For each of the multiple Transformer layers, it may be determined whether the gap between the output result of each Transformer layer and the final output result after passing through multiple Transformer layers is less than a preset gap threshold, that is, it may be determined based on the overall confidence of the feature point matching whether the output result of each Transformer layer and the final output result after passing through multiple Transformer layers can be considered consistent. When the gap between the output result of each Transformer layer and the final output result is less than the preset gap threshold, the processing process of the multiple Transformer layers may be terminated to achieve early exit.

[0094] That is, in the present disclosure, a Transformer layer early exit mechanism can be added to the LoFTR module, and the Transformer layer early exit mechanism can be implemented by performing confidence detection on the output results of each Transformer layer. Specifically, when the output result of each Transformer layer is almost consistent with the final output result after all Transformer layers, the iteration of the Transformer layer can be ended in advance, and then the precise matching stage can be directly entered. That is, the overall confidence of the feature point matching after each Transformer layer can be detected. When the confidence reaches a certain threshold, the iteration of the Transformer layer can be triggered to exit early. In this way, it is possible to avoid processing redundant Transformer layers, that is, it is possible to reduce the iteration of meaningless Transformer layers, thereby saving computing resources and achieving a certain degree of lightweight computing.

[0095] In step 104, a progressive consistent sampling algorithm may be used to iteratively screen the initial matching point set to obtain a target matching point set.

[0096] According to an exemplary embodiment of the present disclosure, for each iteration process, the following operations may be performed:

[0097] K matching points can be randomly selected from the above-mentioned initial matching point set. Then, the model can be fitted using the randomly selected k matching points, and the fitted model can be applied to the above-mentioned initial matching point set to obtain the fitting error of each matching point in the initial matching point set. Next, the consistency metric can be calculated based on the fitting error of each matching point in the initial matching point set. The smaller the fitting error of each matching point, the higher the consistency metric. When the consistency metric is greater than or equal to a preset threshold, the k matching points selected in this iteration can be used as the target matching point set, that is, the k matching points selected in this iteration can be used as the final matching point set. Otherwise, that is, when the consistency metric is less than the preset threshold, the next iteration of this iteration can be entered until the number of iterations reaches the preset number of times.

[0098] It should be noted that, in addition to "the number of iterations reaches a preset number", the condition for terminating the iteration may also be other conditions, for example, "the convergence criterion is satisfied" and so on.

[0099] Thus, in the present disclosure, by iteratively refining the initial matching point set through the Progressive Sampling Consensus (PROSAC) method, erroneous matching points in the initial matching point set can be eliminated, thereby ensuring the accuracy of the matching points and preventing the interference of erroneous matching points on the final registration relationship.

[0100] In step 105, a target registration relationship may be acquired based on the target matching point set. Exemplarily, the target registration relationship may be a transformation matrix, and the transformation matrix is ​​used to align the optical image to be registered with the synthetic aperture radar image to be registered.

[0101] Figure 3 FIG. 1 is a flowchart showing a specific implementation of registering an optical image and a SAR image according to an exemplary embodiment of the present disclosure. Figure 3 ,The registration process can include six parts, namely “obtaining the original image pair”, “image preprocessing”, “phase consistency transformation and generating amplitude response image operation”, “obtaining the initial matching point set”, “iterative refinement”, and “obtaining the registration relationship”.

[0102] Among them, for the part of "obtaining original image pairs", as mentioned above, the original image pairs can be selected from the SEN1-2 dataset, wherein the original image pairs can include original optical images and original synthetic aperture radar images. The SEN1-2 dataset refers to a large open dataset containing Sentinel-1 SAR and Sentinel-2 optical patch pairs.

[0103] As for the "image preprocessing" part, as mentioned above, the original optical image and the original synthetic aperture radar image can be subjected to random affine transformation respectively to obtain the transformed optical image and the transformed synthetic aperture radar image. In addition, the synthetic aperture radar image subjected to random affine transformation can be subjected to histogram equalization and Lee filtering to obtain the processed synthetic aperture radar image. Then, the transformed optical image can be used as the optical image to be registered, and the processed synthetic aperture radar image can be used as the synthetic aperture radar image to be registered. In this way, the optical image to be registered and the synthetic aperture radar image to be registered can be obtained.

[0104] Regarding the part of "phase consistency transformation and generation of amplitude response image", as mentioned above, 2D Log-Gabor filters of different scales and directions can be used to perform phase consistency transformation and generate amplitude response image operations on the optical image to be registered and the synthetic aperture radar image to be registered, thereby obtaining the optical amplitude response image and the synthetic aperture radar amplitude response image. The specific transformation process has been described in detail in the previous article and will not be repeated here.

[0105] Regarding the part of "obtaining the initial matching point set", as mentioned above, the improved LoFTR model can be used to perform coarse matching first to obtain a coarse matching prediction result. Then, a fine matching can be performed based on the coarse matching prediction result to obtain the initial matching point set. The specific process of obtaining the initial matching point set has been described in detail in the previous article and will not be repeated here.

[0106] Regarding the “iterative refinement” part, as mentioned above, the Progressive Consistent Sampling (PROSAC) method can be used to iteratively refine the initial matching point set to remove the erroneous matching points in the initial matching point set, and then the final target matching point set can be obtained. The specific process of “iterative refinement” has been described in detail in the previous article and will not be repeated here.

[0107] As for the part of "obtaining the registration relationship", as mentioned above, the target registration relationship can be obtained based on the target matching point set. Exemplarily, the target registration relationship can be a transformation matrix.

[0108] Figure 4It is a schematic diagram showing a comparison between the time consumed in running a test set in the related art and the time consumed in running a test set in the present disclosure. Figure 4 (a) is a schematic diagram of the time consumed in running the test set in the related art, namely, Figure 4 (a) is a schematic diagram of the time taken by the original LoFTR model to run the test set; Figure 4 (b) is a schematic diagram of the time consumed in running the test set in the present disclosure, that is, Figure 4 (b) is a schematic diagram of the time taken by the improved LoFTR model to run the test set. It can be seen that compared with the related art, the matching accuracy and matching efficiency of the optical image and synthetic aperture radar image registration method provided by the present disclosure are better than the related art.

[0109] Furthermore, the effect of using the registration method of the optical image and the synthetic aperture radar image disclosed in the present invention to perform registration is shown in the figure below: Figure 5~Figure 12 shown.

[0110] Figure 5 FIG. 4 is a schematic diagram showing an example of an original SAR image and an original visible light image according to an exemplary embodiment of the present disclosure. Figure 5 (a) is the original SAR image. Figure 5 (b) in the figure is the original visible light image, that is, the original optical image.

[0111] Figure 6 is a schematic diagram showing an example of a SAR image after histogram equalization and Lee filtering and an original visible light image according to an exemplary embodiment of the present disclosure. Figure 6 (a) in the figure is a SAR image after histogram equalization and Lee filtering. Figure 6 (b) in the figure is the original visible light image.

[0112] Figure 7 1 is a schematic diagram showing an example of a SAR image subjected to phase congruency transformation and amplitude response image generation operations and a visible light image subjected to phase congruency transformation and amplitude response image generation operations according to an exemplary embodiment of the present disclosure. Figure 7 (a) is a SAR image after phase consistency transformation and amplitude response image generation. Figure 7 (b) in the figure is a visible light image after phase consistency transformation and amplitude response image generation.

[0113] Figure 8 is a schematic diagram showing an example of a registration result of a visible light image that has been inversely transformed after registration and an experimental image according to an exemplary embodiment of the present disclosure. Figure 8 (a) in the figure is the visible light image after inverse transformation after registration. Figure 8 (b) in the figure is the registration result of the experimental image.

[0114] Fig. 9 FIG. 4 is a schematic diagram showing another example of a raw SAR image and a raw visible light image according to an exemplary embodiment of the present disclosure. Fig. 9 (a) is the original SAR image. Fig. 9 (b) in the figure is the original visible light image.

[0115] Fig.10 is a schematic diagram showing another example of a SAR image and an original visible light image after histogram equalization and Lee filtering according to an exemplary embodiment of the present disclosure. Fig.10 (a) in the figure is a SAR image after histogram equalization and Lee filtering. Fig.10 (b) in the figure is the original visible light image.

[0116] Fig.11 1 is a schematic diagram showing another example of a SAR image subjected to phase congruency transformation and an amplitude response image generation operation and a visible light image subjected to phase congruency transformation and an amplitude response image generation operation according to an exemplary embodiment of the present disclosure. Fig.11 (a) is a SAR image after phase consistency transformation and amplitude response image generation. Fig.11 (b) in the figure is a visible light image after phase consistency transformation and amplitude response image generation.

[0117] Fig.12 is a schematic diagram showing another example of the registration result of a visible light image that has been inversely transformed after registration and an experimental image according to an exemplary embodiment of the present disclosure. Fig.12 (a) in the figure is the visible light image after inverse transformation after registration. Fig.12 (b) in the figure is the registration result of the experimental image.

[0118] It can be intuitively seen from the above effect diagram that the registration method for optical images and synthetic aperture radar images provided by the present disclosure extracts evenly distributed matching points for all image pairs and achieves better image registration. This means that the registration method for optical images and synthetic aperture radar images provided by the present disclosure has strong robustness to various geometric distortions including translation, rotation and affine, so that it can be applied to various remote sensing images including SAR images, drone images and high-resolution satellite images.

[0119] The method for lightweight optical and SAR image registration based on the improved LoFTR model provided by the present disclosure can reduce the amount of calculation of the original LoFTR model and improve the matching efficiency by adopting the Transformer layer early exit mechanism and the adaptive window attention mechanism. In addition, by adopting the progressive consistent sampling method (PROSAC) for iterative refinement, the matching accuracy can be improved and robust matching can be achieved; by adopting phase consistency transformation and generating amplitude response image operations, the nonlinear radiation distortion between multi-source images can be better eliminated. That is, the present disclosure has good robustness and stability to the nonlinear radiation distortion of matching optical images with synthetic aperture radar images, which can ensure that better registration results are obtained faster.

[0120] Fig.13 is a block diagram illustrating an apparatus for registering an optical image and a synthetic aperture radar image according to an exemplary embodiment of the present disclosure.

[0121] Reference Fig.13 The optical image and synthetic aperture radar image registration device 1300 may include a to-be-registered image acquisition module 1301, an amplitude response image acquisition module 1302, an initial matching point set acquisition module 1303, an iterative screening module 1304 and a registration relationship acquisition module 1305.

[0122] The image acquisition module 1301 to be registered can acquire an optical image to be registered and a synthetic aperture radar image to be registered.

[0123] According to an exemplary embodiment of the present disclosure, the image acquisition module 1301 to be registered can select an original image pair under the same scene from a data set, wherein the original image pair can include an original optical image and an original synthetic aperture radar image. Then, the image acquisition module 1301 to be registered can perform histogram equalization processing and Lee filtering processing on the original synthetic aperture radar image to obtain a processed synthetic aperture radar image. Next, the image acquisition module 1301 to be registered can use the above original optical image as the optical image to be registered, and can use the above processed synthetic aperture radar image as the synthetic aperture radar image to be registered.

[0124] The amplitude response image acquisition module 1302 can perform phase consistency transformation and amplitude response image generation operations on the optical image to be registered and the synthetic aperture radar image to be registered, respectively, and then obtain the optical amplitude response image corresponding to the optical image to be registered and the synthetic aperture radar amplitude response image corresponding to the synthetic aperture radar image to be registered.

[0125] According to an exemplary embodiment of the present disclosure, the amplitude response image acquisition module 1302 can perform convolution calculations on multiple Log-Gabor wavelet filters and a two-dimensional image respectively to obtain a response component corresponding to each Log-Gabor wavelet filter in the multiple Log-Gabor wavelet filters, wherein the two-dimensional image can include an optical image to be registered and a synthetic aperture radar image to be registered, and each Log-Gabor wavelet filter can correspond to its own scale and direction. Next, the amplitude response image acquisition module 1302 can obtain an amplitude component and a phase component based on the above response components.

[0126] Then, the amplitude response image acquisition module 1302 can obtain the sum of the amplitude components at multiple scales in any direction of multiple directions for each pixel point in the two-dimensional image based on the amplitude component and the phase component, and can generate a sequence of the amplitude components and the amplitude components of the pixel point arranged in sequence in multiple directions based on the amplitude component, wherein the multiple directions can be directions corresponding to multiple Log-Gabor wavelet filters, and the multiple scales can be scales corresponding to multiple Log-Gabor wavelet filters.

[0127] Next, the amplitude response image acquisition module 1302 can acquire the optical amplitude response image and the synthetic aperture radar amplitude response image based on the maximum value of the amplitude component and sequence corresponding to each pixel. Exemplarily, the maximum value of the amplitude component and sequence corresponding to each pixel can be used as the value of the corresponding pixel in the amplitude response image to obtain the optical amplitude response image and the synthetic aperture radar amplitude response image.

[0128] The initial matching point set acquisition module 1303 can acquire an initial matching point set between the optical amplitude response image and the synthetic aperture radar amplitude response image based on the improved LoFTR model, wherein the improved LoFTR model can be a model obtained by adding an early exit function and an adaptive window function to the original LoFTR model.

[0129] According to an exemplary embodiment of the present disclosure, the initial matching point set acquisition module 1303 can use the feature extraction network included in the improved LoFTR model to perform feature extraction on the above-mentioned optical amplitude response image and synthetic aperture radar amplitude response image, and obtain the optical coarse matching feature map and the optical fine matching feature map corresponding to the optical amplitude response image, and can obtain the synthetic aperture radar coarse matching feature map and the synthetic aperture radar fine matching feature map corresponding to the synthetic aperture radar amplitude response image.

[0130] Then, the initial matching point set acquisition module 1303 can use the LoFTR module included in the improved LoFTR model to process the optical rough matching feature map and the synthetic aperture radar rough matching feature map to obtain a processing result. Next, the initial matching point set acquisition module 1303 can obtain the confidence matrix based on the processing result. Then, the initial matching point set acquisition module 1303 can be based on the confidence matrix Next, an initial matching point set can be obtained based on the coarse matching prediction result, the optical fine matching feature map, and the synthetic aperture radar fine matching feature map.

[0131] According to an exemplary embodiment of the present disclosure, the above-mentioned LoFTR module may include multiple Transformer layers. For some of the multiple Transformer layers, the initial matching point set acquisition module 1303 may determine the relevant information of the feature points of the input part of the Transformer layer, wherein the feature points may be feature points contained in the optical coarse matching feature map and the synthetic aperture radar coarse matching feature map. Then, the initial matching point set acquisition module 1303 may determine the target window corresponding to the part of the Transformer layer based on the relevant information of the feature points, wherein the target window is a window for attention calculation. Next, for each feature point in the optical coarse matching feature map and the synthetic aperture radar coarse matching feature map, the initial matching point set acquisition module 1303 may process the feature points centered on the feature point and located in the target window to obtain a processing result.

[0132] According to an exemplary embodiment of the present disclosure, the optical image and synthetic aperture radar image registration apparatus 1300 may further include a determination module and a termination module.

[0133] As mentioned above, the LoFTR module may include multiple Transformer layers. For each of the multiple Transformer layers, the determination module may determine whether the gap between the output result of each Transformer layer and the final output result after passing through multiple Transformer layers is less than a preset gap threshold, that is, it may determine whether the output result of each Transformer layer and the final output result after passing through multiple Transformer layers can be considered consistent based on the overall confidence of the feature point matching. When the gap between the output result of each Transformer layer and the final output result is less than the preset gap threshold, the termination module may terminate the processing of the multiple Transformer layers to achieve early exit.

[0134] The iterative screening module 1304 may use a progressive consistent sampling algorithm to perform iterative screening on the initial matching point set to obtain a target matching point set.

[0135] According to an exemplary embodiment of the present disclosure, for each iteration process, the iteration screening module 1304 may perform the following operations:

[0136] K matching points can be randomly selected from the above-mentioned initial matching point set. Then, the model can be fitted using the randomly selected k matching points, and the fitted model can be applied to the above-mentioned initial matching point set to obtain the fitting error of each matching point in the initial matching point set. Next, the consistency metric can be calculated based on the fitting error of each matching point in the initial matching point set. The smaller the fitting error of each matching point, the higher the consistency metric. When the consistency metric is greater than or equal to a preset threshold, the k matching points selected in this iteration can be used as the target matching point set, that is, the k matching points selected in this iteration can be used as the final matching point set. Otherwise, that is, when the consistency metric is less than the preset threshold, the next iteration of this iteration can be entered until the number of iterations reaches the preset number of times.

[0137] It should be noted that, in addition to "the number of iterations reaches a preset number", the condition for terminating the iteration may also be other conditions, for example, "the convergence criterion is satisfied" and so on.

[0138] Thus, in the present disclosure, by iteratively refining the initial matching point set through the Progressive Sampling Consensus (PROSAC) method, erroneous matching points in the initial matching point set can be eliminated, thereby ensuring the accuracy of the matching points and preventing the interference of erroneous matching points on the final registration relationship.

[0139] The registration relationship acquisition module 1305 may acquire the target registration relationship based on the target matching point set. Exemplarily, the target registration relationship may be a transformation matrix, which is used to align the optical image to be registered with the synthetic aperture radar image to be registered.

[0140] Fig.14 is a block diagram illustrating an electronic device 1400 according to an exemplary embodiment of the present disclosure.

[0141] Reference Fig.14 The electronic device 1400 includes at least one memory 1401 and at least one processor 1402, wherein the at least one memory 1401 stores instructions, and when the instructions are executed by the at least one processor 1402, the registration method of the optical image and the synthetic aperture radar image according to the exemplary embodiment of the present disclosure is executed.

[0142] As an example, the electronic device 1400 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instructions. Here, the electronic device 1400 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above instructions (or instruction sets) individually or in combination. The electronic device 1400 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0143] In the electronic device 1400, the processor 1402 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0144] The processor 1402 may execute instructions or codes stored in the memory 1401, wherein the memory 1401 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0145] The memory 1401 may be integrated with the processor 1402, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 1401 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 1401 and the processor 1402 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 1402 can read files stored in the memory.

[0146] In addition, the electronic device 1400 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 1400 may be connected to each other via a bus and / or a network.

[0147] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the above-mentioned registration method of the optical image and the synthetic aperture radar image. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0148] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, including a computer program, and when the computer program is executed by a processor, the registration method of the optical image and the synthetic aperture radar image according to the present disclosure is implemented.

[0149] According to the optical image and synthetic aperture radar image registration method, device, electronic device, storage medium and computer program product disclosed in the present invention, the image to be registered is subjected to phase consistency transformation and amplitude response image generation operations, and an improved LoFTR model and asymptotically consistent sampling algorithm are used for image registration, which can improve the matching accuracy and matching speed between the optical image and the synthetic aperture radar image, and achieve robust matching. That is, the present invention has good robustness and stability to the nonlinear radiation distortion of the matching of the optical image and the synthetic aperture radar image, and can efficiently obtain good registration results.

[0150] According to an exemplary embodiment of the present disclosure, by performing phase consistency transformation and generating amplitude response image operations, it is possible to solve the problem that image registration is sensitive to nonlinear radiation distortion. That is, the phase consistency transformation and amplitude response image generation operations provided by the present disclosure have good robustness and stability to the nonlinear radiation distortion of matching optical images with synthetic aperture radar images, and better registration results can be obtained.

[0151] According to an exemplary embodiment of the present disclosure, the introduction of the adaptive window attention mechanism can make the model more efficient, flexible, and adaptable, and can better process complex input data. Moreover, compared with the traditional linear attention mechanism, the adaptive window attention mechanism can not only reduce the amount of calculation, but also dynamically adjust the window size according to the information learned in each layer, which can significantly reduce the computational overhead while maintaining the model performance.

[0152] According to an exemplary embodiment of the present disclosure, a Transformer layer early exit mechanism may be added to the LoFTR module, so as to avoid processing redundant Transformer layers, that is, to reduce iterations of meaningless Transformer layers, thereby saving computing resources.

[0153] According to an exemplary embodiment of the present disclosure, the initial matching point set can be iteratively refined by the Progressive Consistent Sampling Method (PROSAC), and erroneous matching points in the initial matching point set can be eliminated, thereby ensuring the accuracy of the matching points and preventing the erroneous matching points from interfering with the final registration relationship.

[0154] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0155] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A registration method for an optical image and a synthetic aperture radar image, characterized in that: include: Acquire an optical image to be registered and a synthetic aperture radar image to be registered; Performing phase consistency transformation and amplitude response image generation operations on the optical image to be registered and the synthetic aperture radar image to be registered respectively, to obtain an optical amplitude response image corresponding to the optical image to be registered and a synthetic aperture radar amplitude response image corresponding to the synthetic aperture radar image to be registered; Acquiring an initial matching point set between the optical amplitude response image and the synthetic aperture radar amplitude response image based on an improved LoFTR model, wherein the improved LoFTR model is a model obtained by adding an early exit function and an adaptive window function to the original LoFTR model; Iteratively screening the initial matching point set using a progressive consistent sampling algorithm to obtain a target matching point set; Acquire a target registration relationship based on the target matching point set; The operations of performing phase consistency transformation and generating amplitude response images on the optical image to be registered and the synthetic aperture radar image to be registered respectively include: Performing convolution calculations on a plurality of Log-Gabor wavelet filters and a two-dimensional image respectively to obtain a response component corresponding to each of the plurality of Log-Gabor wavelet filters, wherein the two-dimensional image includes the optical image to be registered and the synthetic aperture radar image to be registered, and each of the Log-Gabor wavelet filters has a respective scale and direction; Based on the response component, obtaining an amplitude component and a phase component; Based on the amplitude component and the phase component, the sum of the amplitude components of each pixel point in the two-dimensional image at multiple scales in any direction of multiple directions is obtained, and based on the sum of the amplitude components, a sequence of the sum of the amplitude components of the pixel point arranged in sequence in the multiple directions is generated, wherein the multiple directions are directions corresponding to the multiple Log-Gabor wavelet filters respectively, and the multiple scales are scales corresponding to the multiple Log-Gabor wavelet filters respectively; Based on the amplitude component corresponding to each pixel point and the maximum value of the sequence, acquiring the optical amplitude response image and the synthetic aperture radar amplitude response image; Wherein, the acquiring of an initial matching point set between the optical amplitude response image and the synthetic aperture radar amplitude response image based on the improved LoFTR model includes: Using the feature extraction network included in the improved LoFTR model to perform feature extraction on the optical amplitude response image and the synthetic aperture radar amplitude response image, an optical coarse matching feature map and an optical fine matching feature map corresponding to the optical amplitude response image are obtained, and a synthetic aperture radar coarse matching feature map and a synthetic aperture radar fine matching feature map corresponding to the synthetic aperture radar amplitude response image are obtained; Processing the optical coarse matching feature map and the synthetic aperture radar coarse matching feature map using the LoFTR module included in the improved LoFTR model to obtain a processing result; Based on the processing results, obtaining a confidence matrix; Based on the confidence matrix, obtaining a coarse-level matching prediction result; The initial matching point set is acquired based on the coarse-level matching prediction result, the optical precise matching feature map and the synthetic aperture radar precise matching feature map.

2. The registration method for an optical image and a synthetic aperture radar image according to claim 1, characterized in that: The LoFTR module includes a plurality of Transformer layers, and the registration method of the optical image and the synthetic aperture radar image further includes: For each of the plurality of Transformer layers, determining whether a difference between an output result of each Transformer layer and a final output result after passing through the plurality of Transformer layers is less than a preset difference threshold; When the gap between the output result of each Transformer layer and the final output result is smaller than the preset gap threshold, the processing process of the multiple Transformer layers is terminated.

3. The registration method for an optical image and a synthetic aperture radar image according to claim 1, characterized in that: The LoFTR module includes a plurality of Transformer layers, and the LoFTR module included in the improved LoFTR model is used to process the optical coarse matching feature map and the synthetic aperture radar coarse matching feature map to obtain a processing result, including: For some of the Transformer layers among the multiple Transformer layers, determining relevant information of feature points input into the some of the Transformer layers, wherein the feature points are feature points included in the optical coarse matching feature map and the synthetic aperture radar coarse matching feature map; Based on the relevant information of the feature points, determining a target window corresponding to the part of the Transformer layer, wherein the target window is a window for performing attention calculation; For each feature point in the optical coarse matching feature map and the synthetic aperture radar coarse matching feature map, a feature point centered on the feature point and located within the target window is processed to obtain the processing result.

4. The registration method for an optical image and a synthetic aperture radar image according to claim 1, characterized in that: The step of iteratively screening the initial matching point set using a progressive consistent sampling algorithm to obtain a target matching point set includes: For each iteration, do the following: Randomly select k matching points from the initial matching point set; Fitting a model using the k matching points, and applying the fitted model to the initial matching point set to obtain a fitting error for each matching point in the initial matching point set; Calculating a consistency metric based on the fitting error of each matching point in the initial matching point set, wherein the smaller the fitting error of each matching point is, the higher the consistency metric is; When the consistency measure is greater than or equal to a preset threshold, the k matching points selected in this iteration are used as the target matching point set; Otherwise, enter the next iteration of this iteration until the number of iterations reaches the preset number.

5. The registration method of an optical image and a synthetic aperture radar image according to claim 1, characterized in that: The step of acquiring the optical image to be registered and the synthetic aperture radar image to be registered comprises: Selecting an original image pair of the same scene from the data set, wherein the original image pair includes an original optical image and an original synthetic aperture radar image; Performing histogram equalization processing and Lee filtering processing on the original synthetic aperture radar image to obtain a processed synthetic aperture radar image; The original optical image is used as the optical image to be registered, and the processed synthetic aperture radar image is used as the synthetic aperture radar image to be registered.

6. A registration device for optical images and synthetic aperture radar images, characterized in that: include: An image acquisition module to be registered is configured to acquire an optical image to be registered and a synthetic aperture radar image to be registered; an amplitude response image acquisition module, configured to perform phase consistency transformation and amplitude response image generation operations on the optical image to be registered and the synthetic aperture radar image to be registered, respectively, to obtain an optical amplitude response image corresponding to the optical image to be registered and a synthetic aperture radar amplitude response image corresponding to the synthetic aperture radar image to be registered; an initial matching point set acquisition module, configured to acquire an initial matching point set between the optical amplitude response image and the synthetic aperture radar amplitude response image based on an improved LoFTR model, wherein the improved LoFTR model is a model obtained by adding an early exit function and an adaptive window function to the original LoFTR model; An iterative screening module is configured to perform iterative screening on the initial matching point set using a progressive consistent sampling algorithm to obtain a target matching point set; A registration relationship acquisition module, configured to acquire a target registration relationship based on the target matching point set; Wherein, the amplitude response image acquisition module is configured as follows: Performing convolution calculations on a plurality of Log-Gabor wavelet filters and a two-dimensional image respectively to obtain a response component corresponding to each of the plurality of Log-Gabor wavelet filters, wherein the two-dimensional image includes the optical image to be registered and the synthetic aperture radar image to be registered, and each of the Log-Gabor wavelet filters has a respective scale and direction; Based on the response component, obtaining an amplitude component and a phase component; Based on the amplitude component and the phase component, the sum of the amplitude components of each pixel point in the two-dimensional image at multiple scales in any direction of multiple directions is obtained, and based on the sum of the amplitude components, a sequence of the sum of the amplitude components of the pixel point arranged in sequence in the multiple directions is generated, wherein the multiple directions are directions corresponding to the multiple Log-Gabor wavelet filters respectively, and the multiple scales are scales corresponding to the multiple Log-Gabor wavelet filters respectively; Based on the amplitude component corresponding to each pixel point and the maximum value of the sequence, acquiring the optical amplitude response image and the synthetic aperture radar amplitude response image; Wherein, the initial matching point set acquisition module is configured as follows: Using the feature extraction network included in the improved LoFTR model to perform feature extraction on the optical amplitude response image and the synthetic aperture radar amplitude response image, an optical coarse matching feature map and an optical fine matching feature map corresponding to the optical amplitude response image are obtained, and a synthetic aperture radar coarse matching feature map and a synthetic aperture radar fine matching feature map corresponding to the synthetic aperture radar amplitude response image are obtained; Processing the optical coarse matching feature map and the synthetic aperture radar coarse matching feature map using the LoFTR module included in the improved LoFTR model to obtain a processing result; Based on the processing results, obtaining a confidence matrix; Based on the confidence matrix, obtaining a coarse-level matching prediction result; The initial matching point set is acquired based on the coarse-level matching prediction result, the optical precise matching feature map and the synthetic aperture radar precise matching feature map.

7. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the registration method for an optical image and a synthetic aperture radar image as claimed in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the registration method for an optical image and a synthetic aperture radar image as claimed in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the optical image and synthetic aperture radar image registration method according to any one of claims 1 to 5 is implemented.