Multimodal remote sensing image registration method and system, terminal device and storage medium

Abstract

The application discloses a kind of multi-modal remote sensing image registration method, system, terminal equipment and storage medium, based on the structure similarity between multi-modal remote sensing image pair, local feature descriptor (edge feature) is extracted using pre-trained edge detection network, and feature matching is carried out using traditional template matching method, accurate and efficient registration under multiple modal images is realized.The application obtains higher matching accuracy and better robustness in multi-source image matching.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image registration, and in particular to a multi-modal remote sensing image registration method and system, a terminal device, and a storage medium. BACKGROUND

[0002] With the rapid development of remote sensing technology, ground observation images from various sensors such as visible light and synthetic aperture radar are becoming increasingly abundant. Due to the influence of image size, cloud cover, imaging quality and other factors, it is often difficult to obtain global information of the target area by detecting the target area of a single type of image in a complex environment. Multi-modal image information collected by different platforms and sensors is complementary. Multi-modal image registration can provide the ability to jointly use multiple information, thereby obtaining greater data volume and shorter re-visit time. Therefore, multi-modal remote sensing image registration has attracted widespread attention in recent years. Multi-modal remote sensing image registration is the process of identifying homonymous points in two or more images obtained from different sensors, different perspectives, or at different times. It is also a bottom-up task and a preprocessing process for many remote sensing image analysis such as image super-resolution and image fusion. However, due to the existence of a large amount of nonlinear radiation distortion and geometric deformation between multi-modal images, high-precision registration between multi-modal images is still a challenging task. Similar to traditional image matching, multi-modal image registration can be implemented by traditional methods and deep learning methods. Traditional methods can be mainly divided into feature-based and region-based methods. The most core challenge of region-based methods is how to design and use appropriate similarity measures to drive the iterative process, so as to accurately estimate the geometric transformation. A direct solution is to use or modify commonly used indicators under information theory, and another method is to indirectly use similarity measures by simplifying the unified domain, such as fast Fourier transform (FFT), structural information extraction and using descriptors to map image intensity to high-dimensional space, etc. In the prior art, based on the structural properties of images, a phase consistency descriptor with illumination and contrast invariance is proposed, and the model is extended to a new image registration method, which uses the Euclidean distance between multi-scale phase controllers (MS-PC) descriptors as a similarity measure to realize the corresponding relationship. Experimental results show that MS-PC has strong robustness to the radiation difference between images, and the method is superior to the quantitative accuracy and connected point number of two commonly used methods (SIFT and SAR-SIFT). However, due to the high resolution of remote sensing images and strong noise caused by the atmosphere, using similarity measures to guide optimization will bring a lot of computational burden. In addition, remote sensing images usually have significant geometric differences, such as large rotation, scaling, deformation and small overlap area, which make the solution space complex and difficult to optimize.Therefore, feature-based methods are more widely used for image registration, which try to estimate the geometric transformation between images by identifying matching features, which can be points, lines, surfaces, etc., but among them the features must be salient and stable, such as Harris corner points and scale-invariant feature transform (SIFT), the traditional SIFT algorithm often has some problems in the sensitivity of feature distribution and descriptor to saliency radiometric differences--especially in multi-source remote sensing imaging, considering the structural similarity between images can be well preserved and used for image registration of different modes, the prior art proposes a fast and robust multi-modal matching framework. Specifically, a dense description of the image is first generated based on existing local descriptors such as HOG and LSS. Then the 3D-FFT and the directional gradient are used to determine and match the similarity measure in the frequency domain, and the feature points extracted from the Harris corner points are matched by a template matching scheme. In the iterative mismatch removal process in the three polynomial model estimation and consistency judgment, the final performance of the entire large-size image pair is verified under the piecewise linear transformation model of local affine estimation. Experimental results show that the matching performance of this framework is better than that of existing matching methods. However, the method is not ideal for the case where there is a large geometric deviation between the two images.

[0003] With the development of artificial intelligence technology, deep convolutional neural networks as a kind of advanced feature extractor are introduced into the field of image registration. By using the nonlinear operation and hierarchical structure of CNN, the image information is continuously learned from low to high level, and complex high-level image features are obtained. A representation method is proposed, and high-level feature matching using more abstract semantic information. The prior art proposes a multi-temporal remote sensing image registration method based on CNN features, which learns multi-scale feature descriptors and gradually increases the selection of initial images to improve matching performance. The multi-scale feature descriptors are generated by a pre-trained VGG network, and a TPS model is integrated to explain the non-rigid transformation. Pairing is performed under the GMM and EM framework. However, the deep learning method in multi-modal remote sensing image registration is not as rich as in computer vision, mainly because on the one hand there is no public multi-modal remote sensing image matching dataset available for training and testing, on the other hand the complexity of remote sensing images themselves has characteristics such as high resolution, mixed noise and large geometric variance, making it more difficult to design a more effective deep registration framework, so the prior art cannot achieve accurate and efficient image registration. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a multi-modal remote sensing image registration method, system, terminal device and storage medium to realize accurate and efficient image registration in view of the shortcomings of the prior art.

[0005] To solve the above technical problems, the technical scheme adopted by the present application is: a multi-modal remote sensing image registration method, comprising the following steps:

[0006] S1, extracting edge features of different scales of the reference image and the remote sensing image;

[0007] S2, selecting edge features of the maximum scale of the reference image and edge features of the maximum scale of the remote sensing image respectively, and calculating the offset between the selected edge features;

[0008] S3, connecting the edge features of different scales of the reference image and the remote sensing image respectively to obtain a reference image feature map and a remote sensing image feature map;

[0009] S4, determining the search area of the remote sensing image feature map using the offset;

[0010] S5, selecting multiple corner points on the search area of the reference image feature map and the remote sensing image feature map respectively, and realizing the registration of the remote sensing image by using the pixel-by-pixel matching method.

[0011] The present application calculates the offset between the reference image and the remote sensing image based on the edge features of the maximum scale, selects the search area of the remote sensing image feature map based on the offset, and then combines the corner point detection process to perform image registration in the search area, which greatly reduces the registration calculation amount and improves the efficiency of image registration. At the same time, the method of the present application does not involve a complex deep registration framework, and the implementation process is simple. Experiments show that the method of the present application obtains high matching accuracy and good robustness in multi-source image matching.

[0012] In step S1, the edge features of different scales of the reference image and the remote sensing image are extracted by using a pre-trained edge detection network. The present application does not need to involve a complex deep registration framework, further simplifying the image registration process and improving the image registration efficiency. At the same time, the results of the edge detection network are used for coarse matching (determining the offset) and then for accurate matching, further improving the accuracy of image registration.

[0013] In the present application, the pre-trained edge detection network uses a convolutional neural network.

[0014] In the present application, in order to ensure the image registration accuracy while considering the image registration efficiency, in step S1, edge features of three scales of the reference image and the remote sensing image are extracted respectively.

[0015] In step S2 of the present application, the calculation formula of the offset (δx, δy) is:

[0016]

[0017] Where (x, y) is the reference image feature map F ref_3The position coordinates of the pixel on the graph, (x+δ) x ,y+δ y F represents the feature map of the remote sensing image. sen_3 The position coordinates of the pixels on the screen.

[0018] The above offset calculation process is simple and easy to achieve coarse registration between the reference image and the remote sensing image (i.e., the image to be registered).

[0019] In step S5, the specific implementation process of registering remote sensing images using the pixel-by-pixel matching method includes: determining matching windows centered on each corner point of the reference image on the feature map of the reference image; using each matching window as a template, sliding within the search area to find the point on the remote sensing image feature map with the highest similarity to the corner point in each matching window; the point with the highest similarity is the point registered with the corner point in the corresponding matching window.

[0020] This invention employs a template matching pixel-by-pixel matching method to achieve image registration. Since this fine registration process is performed within the search area, it ensures both registration accuracy and efficiency, thus achieving efficient and high-precision image registration.

[0021] In this invention, in order to further improve image registration efficiency and speed up the entire calculation process, before determining the matching window, the reference image feature map and the remote sensing image feature map are converted into frequency domain feature maps through three-dimensional fast Fourier transform. At this time, based on the frequency domain feature map, the remote sensing image is registered using a pixel-by-pixel matching method.

[0022] As an inventive concept, the present invention also provides a multimodal remote sensing image registration system, comprising:

[0023] The edge feature extraction module is used to extract edge features at different scales from the reference image and the remote sensing image;

[0024] The offset calculation module is used to select the edge features at the largest scale of the reference image and the edge features at the largest scale of the remote sensing image, and calculate the offset between the selected edge features.

[0025] The first feature map generation module is used to connect edge features of different scales in the reference image to obtain a feature map of the reference image.

[0026] The second feature map generation module is used to connect edge features of different scales in the remote sensing image to obtain the remote sensing image feature map.

[0027] The search region determination module is used to determine the search region of the remote sensing image feature map using the offset.

[0028] The registration module is used to select multiple corner points in the search areas of the reference image feature map and the remote sensing image feature map, and to register the remote sensing image using a pixel-by-pixel matching method.

[0029] As an inventive concept, the present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory; characterized in that the processor executes the computer program to implement the steps of the multimodal image registration method described above.

[0030] As an inventive concept, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon; when the computer program / instructions are executed by a processor, they implement the steps of the multimodal image registration method described above.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] 1. This invention performs coarse and fine registration on images, greatly improving the accuracy of image registration.

[0033] 2. Based on the structural similarity between multimodal remote sensing image pairs, this invention uses a pre-trained edge detection network to extract local feature descriptors (edge ​​features) and uses traditional template matching methods for feature matching, achieving accurate and efficient registration under multiple modal images.

[0034] 3. This invention achieves high matching accuracy and good robustness in multi-source image matching. Experiments show that, compared with both classical methods and deep learning methods, the method of this invention has higher registration accuracy and faster registration speed. This invention provides a good foundation for high-precision and robust matching of multi-source remote sensing images. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the method in Embodiment 1 of the present invention;

[0036] Figure 2 This is a schematic diagram of the edge detection results in Embodiment 1 of the present invention;

[0037] Figure 3 This is a schematic diagram of the pixel-by-pixel matching process in Embodiment 1 of the present invention;

[0038] Figures 4(a) to 4(d)The multi-modal image samples in the MMD data set constructed in Embodiment 1 of the present application; Fig. 4(a) GF: (23°30'34"N, 113°09'35"E), LDS8: (23°33'23"N, 113°06'46"E); Fig. 4(b) GF: (23°30'23"N, 113°23'34"E), LDS8: (23°31'50"N, 113°25'02"E); Fig. 4(c) GF: (22°16'07"N, 112°36'46"E), LDS8: (22°18'05"N, 112°36'06"E); Fig. 4(d) GF: (22°46'15"N, 114°18'37"E), LDS8: (22°48'54"N, 114°19'31"E); wherein, GF represents high-resolution satellite data, LDS8 represents Landsat8 satellite data, and the data in the brackets represent longitude and latitude;

[0039] Fig. 5(a) is a laser radar depth optical picture in Embodiment 1 of the present application; Fig. 5(b) is a laser radar depth optical picture in the DFM method; Fig. 5(c) is a visible infrared picture in Embodiment 1 of the present application; Fig. 5(d) is a visible SAR picture in Embodiment 1 of the present application; Fig. 5(e) is an infrared optical picture in Embodiment 1 of the present application; Fig. 5(f) is an infrared optical picture in the SURT method; Fig. 5(g) is an infrared optical picture in the ORB method; and Fig. 5(h) is an infrared optical picture in the SIFT method. DETAILED DESCRIPTION

[0040] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0041] In the embodiment description of the present application, the terms "first", "second" and other similar terms are not intended to imply any order, quantity and importance, but are merely used to distinguish different elements. In this document, the terms "one", "a" and other similar terms are not intended to mean that there is only one of the described things, but that the description is only directed to one of the described things, which can have one or more. In the embodiment description of the present application, the terms "include", "contain" and other similar terms are intended to mean logical interrelation, and cannot be regarded as indicating spatial structural relation. For example, "A includes B" is intended to mean that B logically belongs to A, and does not mean that B is located inside A in space. In addition, the meaning of the terms "include", "contain" and other similar terms should be regarded as open, rather than closed. For example, "A includes B" is intended to mean that B belongs to A, but B does not necessarily constitute all of A, and A can also include C, D, E and other elements.

[0042] Embodiment 1

[0043] The principle of the method of Embodiment 1 of the present application is shown in Figure 1 This embodiment is based on the principle that there is a modal gap between edges in multi-modal images, but the modal gap is usually consistent Figure 2 In addition, after edge extraction, coarse scale features usually only contain abstract information, while fine scale edge features can provide more fine-grained edge details. Inspired by this principle, this embodiment proposes to use edge features as modal robust descriptors for image matching, and to perform progressive matching from coarse to fine.

[0044] Specifically, a pre-trained edge detection network (i.e., RCF (Liu Y, Cheng M M, Hu X, et al. Richer convolutional features for edge detection [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017:3000-3009.)) is used as the descriptor network to extract edge features of three different scales (scales) for the reference image and the image to be registered. In this embodiment, two pre-trained edge detection networks are used, one for extracting edge features of the reference image and one for extracting edge features of the image to be registered. In the fine matching stage, the edge features of three different scales (i.e., Fref 3, Fref, Fsen 3, Fsen) are concatenated as local feature descriptors, which are used to preserve detailed information and overall structure information. After extracting the edge features, the generated edge features are fed to the subsequent coarse matching stage and feature matching (fine matching) stage. The pre-trained network used in this embodiment is a convolutional neural network, and the convolutional features in CNN (Convolutional Neural Network) gradually become coarse with the increase of the receptive field (Liu Y, Cheng M M, Hu X, et al. Richer convolutional features for edge detection [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017:3000-3009.).

[0045] Figure 2 Edge features extracted from two images of different modalities are described. From Figure 2 As can be seen from the first column, although the appearance difference is large, the multi-modal images share consistent edges at different scales (edge consistency, see Figure 2 the second and third columns).

[0046] Given the coarsest scale edge features (Fref 3 f—3 extracted from the reference image, i.e., the largest scale edge features, and the largest scale edge features extracted from the image to be registered, the two largest scale edge features are used for coarse registration. Template matching is performed using the largest scale edge features in the reference image and the largest scale edge features in the image to be registered to calculate the offset (δx, δy) between the reference image and the image to be registered:

[0047]

[0048] where (x, y) is the position coordinate of the pixel point on the reference image feature map F ref_3 x y sen_3 where (x, y) is the position coordinate of the pixel point on the reference image feature map F

[0049] Compared with global matching, the method of the embodiment has stronger robustness when there is a large displacement deviation between multi-modal remote sensing images. The superiority of the matching accuracy and the calculation time of the embodiment will be discussed later.

[0050] After the coarse matching stage, the edge features extracted at coarse and fine scales, i.e. all the edge features extracted as mentioned above, will be used for more fine-grained feature matching. That is, the edge features of different scales corresponding to the reference image and the remote sensing image are connected respectively to obtain the reference image feature map and the remote sensing image feature map. First, the search area of the remote sensing image feature map is determined by using the offset (δx, δy) calculated as mentioned above. Then, the corner points on the reference image feature map are obtained by using the Harris corner point detection method (Harris, Chris, Mike Stephens, et al. 1988. “A combined corner and edge detector.” Alvey Vision Conference 15(50): 10-5244), and the unstable corner points are filtered out by using the non-maximum suppression method (Neubeck, A., and L. Van Gool. 2006. “Efficient Non-Maximum Suppression.” In 18th International Conference on Pattern Recognition (ICPR’06), Vol. 3, 850-855.). Based on the local consistency of the features, for each stable feature point P(x p p , y Figure 3 ​​​​The similarity between the reference image feature map and the candidate corner point on the target image (i.e., the search region of the image feature map to be registered) is calculated. That is, a matching window centered on each corner point on the reference image feature map is determined, and each matching window is used as a template to slide in the search region to find the point on the image feature map to be registered with the highest similarity to the corner point in each matching window. This point with the highest similarity is the point registered with the corresponding corner point in the matching window. In order to speed up the entire calculation process, the embodiment converts the pixel-by-pixel feature representation map (i.e., the reference image feature map and the image feature map to be registered) to the frequency domain using a three-dimensional fast Fourier transform (FFT) (De Castro, E., and C. Morandi. 1987. “Registration of Translated and Rotated Images Using Finite Fourier Transforms.” IEEE Transactions on Pattern Analysis and Machine Intelligence (5): 700-703.) and performs the above pixel-by-pixel matching to obtain a similarity map, and the position of the maximum value of the similarity map is taken as the matching position of the image to be registered.

[0051] As shown in Figures 4(a) to 4(d) The embodiment introduces a multi-modal remote sensing image matching dataset, i.e., the MMD dataset. The MMD dataset includes 40 pairs of satellite multi-modal remote sensing images collected from Landsat 8 satellite (infrared data) and GF1-WFV satellite (optical data). According to the seasonal attribute, these data are divided into four sub-datasets, which correspond to spring, summer, autumn and winter respectively. For each image pair, the size of the GF1-WFV image is 801x801, and the size of the Landsat 8 image is 512x512. Using the key points with the same geographical coordinates, the transformation matrix can be calculated and regarded as the label of the image pair in the image matching task.

[0052] The following ablation experiments are performed to demonstrate the effectiveness of the method of the embodiment. Then, the method of the embodiment is compared with the prior art method in terms of matching performance, computational complexity and inference time. Finally, the performance of the method of the embodiment in different modes is compared, where the different modes include Visible-SAR, Visible-Infrared and Optical-LiDAR. The number of correctly matched points (CMN), matching accuracy (MA) and running time (Time) are used as performance evaluation indicators. All experiments are performed on a PC equipped with an AMD Ryzen 7 5800H CPU and a NVIDIA GeForce RTX 3060 notebook GPU.

[0053] The present embodiment first studies the effectiveness of the coarse-to-fine matching strategy. Specifically, the effectiveness of the present embodiment solution is verified here by removing the coarse-to-fine matching stage described above (e.g., Model 2). Further, the coarse matching stage is removed here, and the edge features are directly fed into the feature matching stage (fine matching stage). As shown in Table 1, due to the absence of the coarse-to-fine matching strategy, no matching points with correct matches are obtained. Due to the large offset between the two images, it is difficult for the feature matching stage to directly obtain accurate matching results. However, using the coarse-to-fine matching strategy of the present embodiment 1, the impact of the large offset can be overcome in the coarse matching stage, thereby obtaining better matching results.

[0054] In Table 1, the ablation results are achieved by different model variants. C2F represents the coarse-to-fine matching strategy of the present embodiment, FFT represents three-dimensional fast Fourier transform, MSF represents multi-scale feature descriptor, and Features represent different edge descriptors in MSF. As can be seen from Table 1, the registration results obtained by the present embodiment 1 method are the best in terms of the number of correctly matched points (CMN), matching accuracy (MA), and running time (Time).

[0055] Matching results of different methods in Table 1

[0056]

[0057] In the present embodiment, 3D FFT (three-dimensional fast Fourier transform) is used in the feature matching stage for acceleration. In order to verify its effectiveness, a variant of the present embodiment method is introduced here by removing 3D-FFT (i.e., Model-3). As shown in Table 1, compared with Model 3, the comparative standard of the present embodiment (Model 1) benefits from 3D FFT, and the image registration speed is improved by more than 200 times. This fully proves the effectiveness of 3D FFT in the present embodiment.

[0058] In the present embodiment, multi-scale edge features are used in the feature matching stage. In order to prove the importance of multi-scale features for matching, a variant of the present embodiment method is introduced here by using edge features at only a single scale (i.e., Model 4). As can be seen from Table 1, using features at only one scale in the feature matching stage will cause the performance of Model 4 to drop significantly. In contrast, the multi-scale feature descriptors of the present embodiment help the comparative standard of the present embodiment (Model 1) to achieve better performance.

[0059] To further investigate the effectiveness of the edge feature in the method of the present embodiment, two variants of the method of the present embodiment are introduced by replacing the edge feature with the feature extracted by CHOG (Chandrasekhar, V., G. Takacs, D. M. Chen, S. S. Tsai, Y. Reznik, R. Grzeszczuk, and B. Girod. 2012. “Compressed Histogram of Gradients: A Low-Bitrate Descriptor.” International Journal of Computer Vision 96 (3): p. 384-399.) and VGG (Simonyan, Karen, and Zisserman, Andrew. 2014. “Very deep convolutional networks for large scale image recognition.” arXiv preprint arXiv: 1409.1556.) respectively (corresponding to Model 5 and Model 6 respectively). The CHOG feature is a relatively broad hand-crafted feature, while the VGG feature is a widely used CNN-based feature. As can be seen from Table 1 above, the performance of Model 5 and Model 6 is worse and the processing time is longer compared to the model of the present embodiment (i.e. Model 1). This clearly shows that the multi-scale edge feature of the present embodiment is simple and effective, and helps to achieve efficient and accurate registration.

[0060] The method of the present embodiment is compared with existing image matching methods through experiments. Six typical image matching methods are used for comparison with the method of the present embodiment, including SIFT (Lowe, David G. 2004. “Distinctive image features from scale-invariant keypoints.” International Journal of Computer Vision 60(2): 91-110.), SURF (Neubeck, A., and L. Van Gool. 2006. “Efficient Non-Maximum Suppression.” In 18th International Conference on Pattern Recognition (ICPR’06), Vol. 3, 850-855.), ORB (Rublee, Ethan, Vincent Rabaud, Kurt Konolige, and Gary Bradski. 2011. “ORB: An efficient alternative to SIFT or SURF.” In 2011 International Conference on Computer Vision, 2564-2571.), RIFT (Li, Jiayuan, Qingwu Hu, and Mingyao Ai. 2019. “RIFT: Multi-modal image matching based on radiation-variation insensitive feature transform.” IEEE Transactions on Image Processing 29: 3296-3310.), DFM (Efe, Ufuk, Kutalmis Gokalp Ince, and Aydin Alatan. 2021. “Dfm: A performance baseline for deep feature matching.” In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 4284-4293.), and Patch2Pixel (Zhou, Qunjie, Torsten Sattler, and Laura Leal-Taixe. 2021.Patch2pix: Epipolar-guided pixel-level correspondences. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 4669-4678.) and SuperGlue (Sarlin, Paul-Edouard, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. 2020. SuperGlue: Learning feature matching with graph neural networks. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 4938- 4947.). Among them, RIFT, SURF, SIFT and ORB are traditional image registration methods, while DFM, Patch2Pixel and SuperGlue are learning-based methods.

[0061] Table 2 shows the quantitative analysis results. As can be seen from Table 2, in most cases, the method of the present embodiment achieves the best matching accuracy in terms of MA index. Although traditional methods including SIFT, SURF and ORB have high processing efficiency, the artificially set descriptors limit the performance of these methods. The RIFT method has more correct points and good effect, but it takes too long. In contrast, in most cases, learning-based descriptors help the DFM, Patch2Pixel and SuperGlue methods achieve higher registration accuracy. However, the generalization performance of learning-based methods is limited. For example, the MA index score of SuperGlue on the spring corresponding sub-dataset is 0.504, while the MA index score on the autumn corresponding sub-dataset is only 0.096. The method of the present embodiment integrates the CNN descriptor into the traditional framework, achieving higher matching accuracy while maintaining excellent generalization performance in different scenarios.

[0062] Table 2 Registration results of the method of the present embodiment and traditional methods

[0063]

[0064] Figures 5(a) to 5(h)For the quantitative results obtained by the method of the embodiment, it further shows several quantitative results produced by the method of the embodiment. It is obvious that the method of the embodiment can well realize multi-modal image registration and overcome large offset to produce more accurate matching results.

[0065] For the practicability of the image registration method in actual application, the generalization performance is crucial. In order to further verify the excellent generalization ability of the method of the embodiment to the out-of-distrubution modal, here three groups of images with different modalities are collected, as shown in Table 3. As can be seen from Table 3, on these modalities, the method of the embodiment is obviously superior to other methods. Although the method of the embodiment is only trained in the visible-infrared (Visible-Infrared) scene, it can achieve an accuracy of more than 0.8MA in the visible SAR (Visible-SAR) and Lidar depth-visible (LidarDepth-Visible) modes. This fully proves the superior generalization performance of the method of the embodiment.

[0066] Table 3 Three groups of multi-modal image data sets for generalization experiments

[0067]

[0068] In Table 3, GSD represents the distance of the ground sample.

[0069] The embodiment proposes a hybrid multi-modal remote sensing image matching method by integrating CNN-based descriptors and traditional template matching framework. In addition, the embodiment introduces a new dataset for multi-modal remote sensing image matching, namely MMD. It has been proved that the method of the embodiment benefits from CNN-based descriptors, high image matching accuracy, while maintaining excellent generalization performance. A large number of experiments show that the method of the embodiment is superior to the traditional method and the learning-based method in the prior art in terms of accuracy and registration efficiency.

[0070] Embodiment 2

[0071] The embodiment 2 of the present application provides a multi-modal remote sensing image registration system corresponding to the above-mentioned embodiment 1, which comprises:

[0072] An edge feature extraction module is configured to extract edge features on different scales of the reference image and the remote sensing image.

[0073] An offset calculation module is configured to select edge features on the largest scale of the reference image and edge features on the largest scale of the remote sensing image, respectively, and calculate the offset between the selected edge features.

[0074] The first feature map generation module is configured to connect edge features of different scales of the reference image to obtain a reference image feature map.

[0075] The second feature map generation module is configured to connect edge features of different scales of the remote sensing image to obtain a remote sensing image feature map.

[0076] The search region determination module is configured to determine a search region of the remote sensing image feature map by using the offset.

[0077] The registration module is configured to select a plurality of corner points on the search region of the reference image feature map and the remote sensing image feature map respectively, and implement registration of the remote sensing image by using a pixel-by-pixel matching method.

[0078] Embodiment 3

[0079] Embodiment 3 of the present application provides a terminal device corresponding to the above-mentioned embodiment 1. The terminal device can be a processing device for a client, such as a mobile phone, a notebook computer, a tablet computer, a desktop computer, etc., to execute the method of the above-mentioned embodiment.

[0080] The terminal device of the present embodiment comprises a memory, a processor and a computer program stored in the memory; the processor executes the computer program in the memory to realize the steps of the method of embodiment 1.

[0081] In some implementations, the memory can be a high-speed random access memory (RAM), and can also include a non-volatile memory, such as at least one disk memory.

[0082] In other implementations, the processor can be a central processing unit (CPU), a digital signal processor (DSP) or various types of general-purpose processors, which are not limited here.

[0083] Embodiment 4

[0084] Embodiment 4 of the present application provides a computer readable storage medium corresponding to the above-mentioned embodiment, which stores a computer program / instruction. The computer program / instruction is executed by the processor to realize the steps of the method of embodiment 1.

[0085] The computer readable storage medium can be a tangible device that maintains and stores instructions for use by an instruction execution device. The computer readable storage medium can be, for example but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination of the above.

[0086] Those skilled in the art will appreciate that embodiments of the application can be devised for a variety of other systems which are currently developed or later developed. Practitioners skilled in the art will recognize the equivalents of the various features from the preceding description and drawings. There is no intent, therefore, to limit the scope of the application to these embodiments. The intent is to cover all alternatives, modifications, and equivalents.

[0087] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatuses can also be implemented as, special purpose logic Figure 1 Figure 1

[0088] Figure 1 Figure 1

[0089] Although preferred embodiments of the application have been described herein, it will be apparent to those skilled in the art that various modifications can be made within the scope of the application. Accordingly, the appended claims are intended to encompass all such modifications and variations as fall within the scope of the application.

[0090] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.​​​​​

Claims

1. A multi-modal remote sensing image registration method, characterized in that, The method comprises the following steps: S1, extracting edge features of different scales of a reference image and a remote sensing image; S2, selecting edge features of the largest scale of the reference image and edge features of the largest scale of the remote sensing image respectively, and calculating the offset between the selected edge features; S3, connecting the edge features of different scales of the reference image and the remote sensing image respectively to obtain a reference image feature map and a remote sensing image feature map; S4, determining a search area of the remote sensing image feature map by using the offset; S5, selecting a plurality of corner points on the search area of the reference image feature map and the remote sensing image feature map respectively, and realizing the registration of the remote sensing image by using the pixel-by-pixel matching method. In step S2, the calculation formula of the offset (δx, δy) is: wherein (x, y) is the position coordinate of a pixel point on the reference image feature map F ref_3 , (x + δ x , y + δ y ) is the position coordinate of the pixel point on the feature map F sen_3 of the remote sensing image; The specific implementation process of realizing the registration of the remote sensing image by using the pixel-by-pixel matching method comprises: determining a matching window with each corner point on the reference image as the center on the reference image feature map, taking each matching window as a template, sliding in the search area, and finding the point on the remote sensing image feature map with the highest similarity to the corner point in each matching window. The point with the highest similarity is the point registered with the corner point in the corresponding matching window.

2. The multi-modal remote sensing image registration method of claim 1, wherein, In step S1, the pre-trained edge detection network is used to extract edge features of different scales of the reference image and the remote sensing image.

3. The multi-modal remote sensing image registration method of claim 2, wherein, The pre-trained edge detection network adopts a convolutional neural network.

4. The multi-modal remote sensing image registration method according to one of claims 1 to 3, characterized in that, In step S1, edge features of three scales of the reference image and the remote sensing image are extracted respectively.

5. The multi-modal remote sensing image registration method of claim 1, wherein, Before determining the matching window, the reference image feature map and the remote sensing image feature map are converted into frequency domain feature maps by three-dimensional fast Fourier transform. At this time, based on the frequency domain feature maps, the registration of the remote sensing image is realized by using the pixel-by-pixel matching method.

6. A multi-modal remote sensing image registration system, characterized in that, It comprises: An edge feature extraction module for extracting edge features of different scales of a reference image and a remote sensing image; An offset calculation module for selecting edge features of the largest scale of the reference image and edge features of the largest scale of the remote sensing image respectively, and calculating the offset between the selected edge features; A first feature map generation module for connecting edge features of different scales of the reference image to obtain a reference image feature map; A second feature map generation module for connecting edge features of different scales of the remote sensing image to obtain a remote sensing image feature map; A search area determination module for determining a search area of the remote sensing image feature map by using the offset; A registration module for selecting a plurality of corner points on the search area of the reference image feature map and the remote sensing image feature map respectively, and realizing the registration of the remote sensing image by using the pixel-by-pixel matching method. In step S2, the calculation formula of the offset (δx, δy) is: wherein (x, y) is the position coordinate of the pixel point on the reference image feature map F ref_3 , (x + δ x , y + δ y ) is the position coordinate of the pixel point on the feature map F sen_3 of the remote sensing image. The specific implementation process of realizing the registration of the remote sensing image by using the pixel-by-pixel matching method comprises: determining a matching window with each corner point on the reference image as the center on the reference image feature map, taking each matching window as a template, sliding in the search area, and finding the point on the remote sensing image feature map with the highest similarity to the corner point in each matching window. The point with the highest similarity is the point registered with the corner point in the corresponding matching window.

7. A terminal device comprising a memory, a processor, and a computer program stored on the memory; characterized in that, The processor executes the computer program to realize the steps of the method in any one of claims 1-5.

8. A computer readable storage medium having stored thereon computer programs / instructions; characterized in that, The computer program / instructions, when executed by the processor, implement the steps of the method according to any one of claims 1 to 5.

Images