Intelligent multi-scale registration method of cross-modal remote sensing images for complex ground scenes

Abstract

The present invention discloses an intelligent multi-scale registration method for cross-modal remote sensing images for complex ground scenes, and belongs to the field of aerospace remote sensing image processing and computer vision technology. On the one hand, the present invention improves the robustness of the similarity metric for cross-modal remote sensing image representation by aligning modal alignable structures and filtering structures that are difficult to align between modalities; on the other hand, by adopting a coarse-to-fine multi-level matching strategy, it effectively avoids searching for each pixel in the area, improves the search efficiency, and also avoids the missed matching phenomenon caused by the method of too strong a priori region. Through the above two aspects, the present invention achieves efficient acquisition of high-precision pixel-level matching between cross-modal remote sensing images.

Description

Technical Field

[0001] The present invention belongs to the field of aerospace remote sensing image processing and computer vision technology, and relates to a cross-modal remote sensing image representation learning and an image registration method based on representation similarity, and specifically relates to an intelligent multi-scale registration method for cross-modal remote sensing images for complex ground scenes. Background Art

[0002] Space remote sensing technology is an important means of obtaining geospatial information. New sensors are constantly emerging, and geospatial information collection has evolved from relying on a single sensor in the past to using multiple types of sensors at the same time to obtain multimodal remote sensing images with different resolutions, different phases, and different spectral properties. These multimodal images reflect the different properties of the ground objects and can provide complementary information for surface observation and analysis. Therefore, how to fully integrate these multimodal remote sensing images to provide more comprehensive observations and more comprehensive analysis of the surface is crucial for various remote sensing applications. Among them, the most critical basic work is the cross-modal image registration technology for complex ground scenes. This technology is a key preprocessing link for many remote sensing applications and analyses, such as cross-modal image fusion, ground object recognition, change detection, 3D reconstruction, and remote sensing quantitative information analysis. The registration accuracy will have a very important impact on many subsequent remote sensing applications.

[0003] Existing cross-modal remote sensing image registration methods usually use traditional manual descriptors to extract multi-modal image features for subsequent registration work. Specifically, they are divided into feature-based methods and region-based methods. The feature-based method first performs feature detection, detects two sets of feature points and corresponding feature descriptors from the reference image and the floating image respectively, and then matches them according to the feature descriptors (Fully Automatic Multi-modal Remote Sensing Image Registration Method Based on Feature Points and Boundary Information [J]. Remote Sensing of Land and Resources, 2014). Due to the huge grayscale and texture differences between multi-modal images, these manually designed local descriptors cannot accurately extract the invariant features between modalities, and thus cannot obtain high-precision matching results. At the same time, there is a large amount of noise on the remote sensing image, and some points will be mistakenly detected as key points, which greatly affects the registration accuracy of cross-modal remote sensing images. The region-based method is also called template matching. It calculates the similarity of the image blocks in the corresponding template windows in the main and secondary images, and selects a pair of template windows with the greatest similarity as the matching result. The region-based method is sensitive to noise and nonlinear grayscale differences, and the matching speed is slow. With the widespread application of deep learning, researchers have begun to propose the use of style transfer networks to align images of different modalities to solve the problem of modality differences. It converts multimodal images into images with consistent styles, and then performs the traditional matching process. This type of method introduces an additional style transfer network, which further increases the computational complexity of the traditional method and reduces the practicality of the method; in addition, the style transfer network does not convert the features required by the registration method, so it still cannot effectively remove the modality differences in the feature descriptor, but may introduce a lot of noise, affecting subsequent matching. The newer deep learning-based method uses a twin convolutional network to calculate the similarity between two image blocks. The learned feature descriptor is more robust to nonlinear grayscale value differences, which makes it initially show its advantages in the matching of cross-modal remote sensing images. However, due to the simple design of the existing deep learning-based methods, it does not show the features that are difficult to align between modalities in complex ground scenes, which affects the calculation of the degree of similarity; in addition, these methods do not consider the high computational complexity caused by the need for fine and large-scale search for high-precision pixel-level registration accuracy, so it is difficult to efficiently obtain high-precision registration results.

[0004] In view of the above situation, in order to improve the registration accuracy and speed of cross-modal remote sensing image registration in complex ground scenes, the present invention proposes a cross-modal remote sensing image intelligent multi-scale registration method for complex ground scenes. First, starting from the representation difference of cross-modal remote sensing data, a representation similarity measurement method between cross-modal images based on modal difference representation alignment and modal redundant information filtering is proposed to obtain similarity measurements of ground object scene categories, modal representation differences, and data acquisition noise robustness; on this basis, in view of the complex scenes, different regional displacement field ranges, and slow registration speed in cross-modal images, a pyramid registration network architecture under multi-scale resolution is proposed to achieve an image registration method that quickly and accurately registers from global to local pixel levels. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a cross-modal remote sensing image intelligent multi-scale registration method for complex ground scenes, that is, by learning and processing the representation differences between cross-modal remote sensing images, the features of aligned data of different modalities are obtained by removing redundant information, and then a progressive multi-scale matching strategy is used to gradually refine the registration from global to local to obtain accurate and effective matching.

[0006] The present invention takes the registration of an optical remote sensing image (hereinafter referred to as optical image) and a synthetic aperture radar (hereinafter referred to as SAR image) as an example, and the specific technical solution is as follows:

[0007] A cross-modal remote sensing image intelligent multi-scale registration method for complex ground scenes includes the following steps:

[0008] In the first step, feature maps of multiple resolutions are extracted from the optical image and SAR image to be registered, and their respective feature pyramids are obtained.

[0009] In the second step, the feature map is selected, the matching range is determined, the matching body is constructed, and the region-level similarity score is calculated within the matching range.

[0010] The third step is to perform similarity score calculation that is robust to the differences in cross-modal remote sensing data representation, enhance cross-modal alignable areas, and suppress areas that are difficult to align between modalities; the filtered region-level similarity scores are fused in the spatial dimension, and the best matching position is calculated.

[0011] The fourth step is to traverse the feature pyramid starting from the lowest-scale feature map, repeating the second and third steps until the highest-scale feature map of the pyramid.

[0012] The beneficial effects of the present invention are as follows: the present invention proposes a cross-modal remote sensing image intelligent multi-scale registration method for complex ground scenes, which realizes robust and fast cross-modal remote sensing image pixel-level precise registration from global to local. The present invention has the following advantages:

[0013] (1) A similarity measurement method that is robust to differences in representation of cross-modal remote sensing data is proposed. The similarity measurement target of the task itself is used to align the modal alignable structures, which is conducive to the matching of consistent structures between modalities. The learned structural attention is used to filter the structures that are difficult to align between modalities to avoid misleading the matching due to these redundant information. This method effectively reduces the impact of inconsistent information or noise between modalities and uses the invariant structural information between modalities for registration, thereby improving the robustness of cross-modal remote sensing image registration.

[0014] (2) An end-to-end cross-modal remote sensing image intelligent registration method for complex ground scenes is proposed. The framework adopts a multi-level matching strategy from coarse to fine, uses multi-scale pyramid features, first performs a global coarse search, gradually locates the region, and then performs a fine search. This effectively avoids searching every pixel in the region, and also avoids the missed match phenomenon caused by the method with too strong a priori region. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is the overall flow chart of the present invention.

[0016] Figure 2 This is the registration result diagram of the present invention. DETAILED DESCRIPTION

[0017] The present invention is based on a cross-modal remote sensing image intelligent registration method for complex ground scenes. By searching the position of the modality 1 image on the modality 2 image, the images of modality 1 and modality 2 are registered, so that the different information contained in the two modalities can be fully utilized for geospatial information analysis and application. Taking the registration of a pair of optical images and SAR images as an example, the specific implementation plan is as follows:

[0018] In the first step, a two-dimensional convolutional network with the same structure and non-shared weights is used to extract features from optical images and SAR images respectively, and a multi-scale feature pyramid of the optical image is obtained. (Shape is H o , W o are the height and width of the optical image, respectively, and C is the number of features) and the multi-scale feature pyramid of the SAR image (Shape is H s , W sare the height and width of the SAR image respectively), where i is the scale index and l is the number of pyramid layers.

[0019] The second step is to select feature maps, determine matching ranges, construct matching bodies, and calculate region-level similarity scores within the matching range. The specific steps are as follows:

[0020] Step 2.1: Select the optical image feature map in the two feature pyramids SAR image feature map (For the first execution, select the lowest scale optical image feature map And SAR image feature map ).

[0021] Step 2.2: Match the results from the last match As the center point, select candidate matching positions in the two-dimensional area [-k, k] around it, and get N = (2k + 1) 2 candidate matching positions. The position is the upper left corner, and N=n is selected in the n×n area 2 candidate matching positions.

[0022] Step 2.3: Take all N candidate matching positions as the upper left corner starting point, Upper cut and Optical feature blocks of the same size, and the N optical feature blocks are respectively connected to the SAR image feature map on the feature dimension C Cascaded to construct a shape The matching body, where H s , W s are the height and width of the SAR image respectively.

[0023] Step 2.4: Feed the matching body constructed in step 2.3 into a multi-layer two-dimensional convolutional network, and the output shape is The tensor represents the similarity score of each point on the SAR image feature map at each candidate matching position, that is, the region-level similarity score.

[0024] The third step is to perform a similarity score calculation that is robust to the differences in representation of cross-modal remote sensing data, enhance cross-modal alignable regions, and suppress regions that are difficult to align between modalities; fuse the filtered regional similarity scores in the spatial dimension and calculate the best matching position. The specific steps are as follows:

[0025] Step 3.1: Convert the optical image feature map It is sent to the salient information detection module and outputs a structural attention map of size The structural attention map is used to filter data points that are difficult to align between modalities. The structural attention map is multiplied with the region-level similarity score to enhance the feature areas of aligned modal differences and suppress the feature areas that are difficult to align.

[0026] Step 3.2: The filtered region-level similarity scores are and The sum is taken in two dimensions to obtain N similarity scores of N candidate matching positions.

[0027] Step 3.3: Use the softmax function to normalize the N similarity scores to obtain {p c}, where c∈{0, 1, ..., N-1} is the candidate matching position index, and the matching result of the current stage is calculated based on this score The calculation is as follows:

[0028]

[0029] where s c Represents the c-th candidate matching position coordinates, which is two-dimensional.

[0030] The best matching position obtained at this stage is

[0031]

[0032] The fourth step is to traverse the feature pyramid starting from i=l-1, repeating the second and third steps until the last layer of features in the pyramid, that is, i=0, to obtain the final matching position.

Claims

1. A cross-modal remote sensing image intelligent multi-scale registration method for complex ground scenes, characterized in that: The steps include: The first step is to extract feature maps of multiple resolutions from the optical image and SAR image to be registered, and obtain their respective feature pyramids; The second step is to select feature maps, determine matching ranges, construct matching bodies, and calculate region-level similarity scores within the matching ranges. The third step is to perform a similarity score calculation that is robust to the differences in representation of cross-modal remote sensing data, enhance cross-modal alignable regions, and suppress regions that are difficult to align between modalities; the filtered region-level similarity scores are fused in the spatial dimension, and the best matching position is calculated; The fourth step is to traverse the feature pyramid starting from the lowest scale feature map, repeating the second and third steps until the highest scale feature map of the pyramid is reached; The specific steps of the third step are as follows: Step 3.1: Convert the optical image feature map It is sent to the salient information detection module and outputs a structural attention map of size The structural attention map is used to filter data points that are difficult to align between modalities. The structural attention map is multiplied with the region-level similarity score to enhance the feature areas of aligned modal differences and suppress the feature areas that are difficult to align. Step 3.2: The filtered region-level similarity scores are and The sum of the two dimensions is calculated to obtain N similarity scores of N candidate matching positions; Step 3.3: Use the softmax function to normalize the N similarity scores to obtain {p c }, where c∈{0,1,…,N-1} is the candidate matching position index, and the matching result of the current stage is calculated based on this score The calculation is as follows: Among them, s c Represents the c-th candidate matching position coordinates, which is two-dimensional; The best matching position obtained at this stage is The specific steps of the third step are as follows: Step 3.1: Convert the optical image feature map It is sent to the salient information detection module and outputs a structural attention map of size The structural attention map is used to filter data points that are difficult to align between modalities. The structural attention map is multiplied with the region-level similarity score to enhance the feature areas of aligned modal differences and suppress the feature areas that are difficult to align. Step 3.2: The filtered region-level similarity scores are and The sum of the two dimensions is calculated to obtain N similarity scores of N candidate matching positions; Step 3.3: Use the softmax function to normalize the N similarity scores to obtain {p c }, where c∈{0,1,…,N-1} is the candidate matching position index, and the matching result of the current stage is calculated based on this score The calculation is as follows: Among them, s c Represents the c-th candidate matching position coordinates, which is two-dimensional; The best matching position obtained at this stage is 2. According to claim 1, a cross-modal remote sensing image intelligent multi-scale registration method for complex ground scenes is characterized in that: In the first step, a two-dimensional convolutional network with the same structure and no shared weights is used to extract features from the optical image and the SAR image, and a multi-scale feature pyramid of the optical image is obtained. Multi-scale feature pyramid of SAR images Where i is the scale index, l is the number of pyramid layers; the shape of the multi-scale feature pyramid of the optical image is The shape of the multi-scale feature pyramid of SAR image is Among them, H o , W o are the height and width of the optical image, H s , W s are the height and width of the SAR image respectively, and C is the number of features.

3. The cross-modal remote sensing image intelligent multi-scale registration method for complex ground scenes according to claim 1 or 2, characterized in that: The specific steps of the second step are as follows: Step 2.1: Select the optical image feature map in the two feature pyramids SAR image feature map Among them, the first execution selects the lowest scale optical image feature map And SAR image feature map Step 2.2: Match the results from the last match As the center point, select candidate matching positions in the two-dimensional area [-k, k] around it, and get N = (2k + 1) 2 candidate matching positions; the first execution is The position is the upper left corner, and N=n is selected in the n×n area 2 candidate matching positions; Step 2.3: Take all N candidate matching positions as the upper left corner starting point, Upper cut and Optical feature blocks of the same size, and the N optical feature blocks are respectively connected to the SAR image feature map on the feature dimension C Cascaded to construct a shape The matching body; Step 2.4: Feed the matching body constructed in step 2.3 into a multi-layer two-dimensional convolutional network, and the output shape is The tensor represents the similarity score of each point on the SAR image feature map at each candidate matching position, that is, the region-level similarity score.

Images