A method for automatic matching of air-ground remote sensing images
By extracting multi-scale deep features and structural feature attention maps, combining global local context information enhancement and image matching network, the problem of insufficient viewing angle distortion and robustness in space image matching is solved, and high-precision and robust space image are achieved Automatic matching.
Patent Information
- Application Number
- CN202411233501.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-04
AI Technical Summary
The existing open-ground image matching method has low performance when dealing with significant viewing angle distortion in extremely wide baseline scenarios, and deep learning-based methods still have the problem of insufficient local descriptor robustness.
Multi-scale deep feature expression graphs are extracted using residual networks, and the structural feature attention graph and global local context information enhancement is combined with Sinkhorn algorithm and image matching network to build a loss function based on graph structure for training, and a matching matrix is generated to realize the space-ground image. automatic matching.
High-precision matching of space images is achieved, wrong local optimal matching is avoided, and robustness and matching accuracy are improved, especially in matching performance in weak textures and repeated texture areas.
Smart Images

Figure CN118736255B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of remote sensing image processing, and in particular to a high-precision automatic matching method for air-to-ground remote sensing images. Background Art
[0002] When facing the air-ground image matching scenario, the existing image feature matching algorithms can be roughly divided into two categories: matching algorithms based on artificially designed features and matching algorithms based on deep learning. Among them, matching algorithms based on artificially designed features can be further divided into three categories: the first category is direct matching methods, that is, directly calculating feature descriptors on air-ground images, such as scale-invariant feature transform (SIFT) (Lowe, 2004), gradient location and orientation histogram (GLOH), speeded up robust features (SURF), etc. Although these descriptors are effective and robust for general baseline scenes, they have low performance in dealing with ground-air image matching tasks involving significant perspective distortion in extremely wide baseline scenes. The second category is matching methods based on geometric correction, that is, first using prior information to geometrically correct air-ground images, such as using prior geometric information or surface structure to correct air and ground images to the same reference plane. However, this type of method has a quadratic time complexity because descriptors need to be extracted from the corrected image, and the quality of image pairwise correction will also affect the robustness of air-ground matching. The third category is methods based on three-dimensional rendering. The prerequisite for implementing this type of method is to use a 3D model with aerial images and ground views to generate realistic synthetic ground view images. However, this synthetic view image often has problems such as local geometric deformation, holes, and texture aliasing, making it difficult to maintain robust matching performance.
[0003] Matching algorithms based on deep learning can extract high-level semantic information from a large amount of training data using the strategy of "learning-feedback-adjustment". Therefore, deep learning has attracted extensive research attention in the field of remote sensing image matching. Compared with traditional hand-designed descriptors, convolutional neural networks can more easily mine deep information features of images. Early learning-based methods focused on designing powerful local features, some of which revolved around the improvement of local feature detectors with deep architectures (KeyNet, Affnet, and Quad-networks), and some studies focused on using learned representations to enhance local feature descriptors (L2net, HardNet, and Contextdesc).
[0004] With the advancement of deep learning, the detection and description of image features have been integrated into the same network architecture. In the algorithm R2D2, not only are reliability and repeatability jointly evaluated during the feature extraction process, but feature detection and description are also jointly learned at the same time. In addition, different learning mechanisms have also been applied to the extraction and description of image features. Superpoint uses a self-supervisory mechanism to establish a data set, avoiding the annotation of a large number of samples. At the same time, it enhances the robustness of feature point descriptors by performing random homography transformations on images in the training set, thereby achieving self-supervised extraction of image features. Disk introduces reinforcement learning into the local feature extraction framework, making the training and reasoning of the probability model closer, thereby improving the sophistication of image features.
[0005] Although the robustness of local descriptors has gradually improved with the development of research, for open space image scenes with significant changes in perspective, illumination and scale, it is still impossible to obtain reliable matching relationships by relying solely on local descriptors. The existing feature matching methods based on deep learning still have some problems that need to be further solved. Summary of the invention
[0006] In order to solve the problems in the background technology, the present invention provides an automatic matching method for air-to-ground remote sensing images with small error, strong robustness and accurate matching.
[0007] A method for automatic matching of air-ground remote sensing images comprises the following steps:
[0008] S1, build a dataset of air-ground image pairs:
[0009] Acquire an aerial image set and a ground image set containing the same area, and form an air-ground image pair with the corresponding aerial images and ground images; form an air-ground image pair data set with all acquired air-ground image pairs, and divide the air-ground image pair data set into a training data set and a test data set;
[0010] S2, extract deep feature expression map:
[0011] Use the residual network encoder to extract the features of the training data set in S1 and obtain deep feature expression graphs of different scales;
[0012] S3, obtain the structural feature attention map with weak boundary attributes:
[0013] First, the deep feature expression maps of different scales obtained by S2 are spliced to obtain the feature map ; Then, in the feature map The multi-scale deep features are averaged in the channel dimension to obtain a channel-averaged feature map; finally, the channel-averaged feature map is Convolutional layers and Activation function operation, outputting a structural feature attention map with weak boundary properties;
[0014] S4, performs global and local context information enhancement, specifically:
[0015] S41, perform global context information enhancement:
[0016] Extracting feature descriptors of global context in the deep feature expression graph ;
[0017] S42, perform local context information enhancement:
[0018] Extracting feature descriptors of local context in the deep feature expression graph ;
[0019] S43, the With the Fusion obtains feature descriptors with global and local information ;
[0020] S5, training image matching network:
[0021] The The structural feature attention map obtained by S3 is sent to the image matching network, and then the Sinkhorn algorithm and the garbage bin channel are used to obtain the matching matrix ;
[0022] Using the matching matrix Constructing a loss function based on graph structure , we have the following formula:
[0023]
[0024] in, The matching matrix The maximum number of rows, The matching matrix The maximum number of columns, For the matrix the number of rows, For the matrix The number of columns of the trash can channel is ;
[0025] Train the image matching network until the loss function Converge and obtain the trained space-land matching network model;
[0026] S6, output matching results:
[0027] Use the trained space-ground matching network model in S5 to predict the points with the same name in the test data set in S1 and output the matching results.
[0028] In S1, the rotation angle difference between the aerial image and the ground image in the air-ground image pair is within 20°.
[0029] The deep feature expression graphs of different scales in S2 include .
[0030] The deep feature expression graph in S2 is specifically: and ;in is the height, is the width, is the number of channels, .
[0031] S41 includes the following steps:
[0032] S411, in all the deep feature expression graphs, dividing the deep feature expression graphs with higher dimensions into a number of feature blocks of fixed sizes by adaptive average pooling;
[0033] S412, dividing each of the characteristic image blocks into a plurality of flat vectorized two-dimensional patches , each of the vectorized two-dimensional patches The size is ; For the vectorized 2D patch Serial number;
[0034] S413, position code:
[0035] Use the following formula to calculate the feature containing the position code ;
[0036]
[0037] in, is the position information of the vectorized two-dimensional patch in the feature block, is the total number of vectorized 2D patches, is a trainable linear projection;
[0038] S414, obtain hidden features of the global context:
[0039] First, construct the transformer operation, which has the following two formulas:
[0040]
[0041]
[0042] Among them, MSA stands for multi-head self-attention, MLP stands for multi-layer perceptron, and LN stands for layer normalization operator. Represents the number of iterations of the transformer operation, For the The result of the transformer operation. For the The result of the transformer operation. It is the intermediate quantity of transformer operation. When , it means that no transformer operation is performed;
[0043] Then, the feature containing the position code obtained in S413 conduct Transformer operation to obtain the feature descriptor of the global context .
[0044] S413 specifically uses trainable linear projection Vectorize 2D patches Mapped to a potential 128-dimensional embedding space, which is then added to Embedded in; used in Preserve vectorized 2D patches in The position information in the corresponding feature block.
[0045] Preferably, in S414 .
[0046] S42 specifically includes the following steps:
[0047] S421, interpolate all deep feature expression maps obtained in S2 to the same spatial size, and then aggregate them to obtain the interpolated feature map ;
[0048] S422, use Convolutional layers and 3 Dilated convolutional layers to interpolate feature maps Improve the feature receptive field and obtain the feature descriptor of the local context .
[0049] Preferably, the three The dilation rates of the dilated convolutional layers are 6, 12, and 18 respectively.
[0050] Compared with the prior art, the present invention has the following beneficial effects:
[0051] 1. The method of the present invention can effectively extract multi-scale information from air-ground images and generate corresponding multi-scale deep feature descriptions; at the same time, the present invention can also induce consistent attention responses from the same area of different air-ground images for constructing feature descriptors.
[0052] 2. The structural feature attention map with weak boundary attributes obtained by the present invention can avoid falling into erroneous local optimal matching in areas with large differences in deep common features.
[0053] 3. In the present invention, the feature descriptor of the local context can obtain context perception from large to global and small to the surroundings of feature points, which effectively enhances the robustness of the descriptor to the matching of weak texture and repeated texture areas. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 is a flow chart of the present invention;
[0055] Figure 2 It is a schematic diagram of the matching results using the method of the present invention in an embodiment of the present invention. DETAILED DESCRIPTION
[0056] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and embodiments.
[0057] like Figure 1 As shown, the air-ground remote sensing image automatic matching method of the present invention comprises the following steps:
[0058] S1, build a dataset of air-ground image pairs:
[0059] Aerial image sets and ground image sets covering the same area are obtained, and the corresponding aerial images and ground images are combined into air-ground image pairs, where the rotation angle difference between the two images in the air-ground image pair should be within 20°. All acquired air-ground image pairs are combined into an air-ground image pair dataset, and the air-ground image pair dataset is divided into a training dataset and a test dataset.
[0060] S2, extract deep feature expression map:
[0061] A multi-scale deep feature description strategy is adopted, and the residual network encoder is used to extract the features of the training data set obtained by S1 to obtain five deep feature expression graphs of different scales. .
[0062] In a preferred embodiment of the present invention, the deep feature expression map is specifically: ,in is the height, is the width, is the number of channels, .
[0063] S3, obtaining a structural feature attention map with weak boundary attributes, specifically including the following steps:
[0064] S31, get S2 After splicing, we get the feature map .
[0065] S32, in the feature map The multi-scale deep features are averaged in the channel dimension to obtain the channel-averaged feature map.
[0066] S33, performing channel averaging on the feature map Convolutional layers and The activation function is operated to output a structural feature attention map with weak boundary properties.
[0067] S4, performs global and local context information enhancement, specifically:
[0068] S41, performing global context information enhancement, including the following steps:
[0069] S411, in all deep feature expression graphs, the deep feature expression graphs with higher dimensions are divided into several feature blocks of fixed sizes by adaptive average pooling;
[0070] In one embodiment of the present invention, the size of the feature block is 64×64.
[0071] S412, split the features of each feature block into several flat vectorized two-dimensional patches , each vectorized 2D patch The size is 16×16. To vectorize a 2D patch Number.
[0072] S413, position code:
[0073] Using Trainable Linear Projection The vectorized 2D patch Mapped to a potential 128-dimensional embedding space, which is then added to Embed to preserve vectorized 2D patches The position information in the corresponding feature block is used to obtain the feature containing the position code. , we have the following formula:
[0074]
[0075] Trainable Linear Projection Used to transform the spatial dimension of a 2D patch.
[0076] in, To vectorize the position information of the two-dimensional patch in the feature block in S411, is the total number of vectorized 2D patches.
[0077] S414, obtain hidden features of the global context:
[0078] First, construct the transformer operation, which has the following two formulas:
[0079]
[0080]
[0081] Among them, MSA stands for multi-head self-attention, MLP stands for multi-layer perceptron, and LN stands for layer normalization operator. Represents the number of iterations of the transformer operation, For the The result of the transformer operation. For the The result of the transformer operation. is the intermediate quantity of transformer operation. , it means that no transformer operation is performed. Then the feature containing position encoding obtained in S413 is conduct Transformer operation to obtain the feature descriptor of the global context .
[0082] In one embodiment of the present invention, Preferably 8.
[0083] S42, performing local context information enhancement, including the following steps:
[0084] S421, interpolate all deep feature expression maps obtained in S2 to the same spatial size, and then aggregate them to obtain the interpolated feature map , It is used to improve the utilization of deep feature expression maps and integrate information of different scales.
[0085] S422, use Convolutional layers and 3 Dilated convolutional layers to interpolate feature maps Improve the feature receptive field and obtain the feature descriptor of the local context ;
[0086] In one embodiment of the present invention, the preferred expansion rates of the expanded convolutional layer are 6, 12 and 18 respectively; this step can further enhance the representation capability of the feature descriptor of the local context from different receptive fields.
[0087] S43, the feature descriptor of the global context obtained in S414 Feature descriptor integrated into the local context obtained in S422 In this paper, we obtain a feature descriptor with global and local information. .
[0088] S5, training image matching network:
[0089] The feature descriptors of the global and local information The structural feature attention map obtained by S33 is sent to the image matching network, and the Sinkhorn algorithm is used to Perform correlation operations with the structural feature attention map, and then combine it with the dustbin channel (Dustbin) in the Sinkhorn algorithm to obtain the matching matrix .
[0090] Using the Matching Matrix Constructing a loss function based on graph structure , we have the following formula:
[0091]
[0092] in, For the matrix The maximum number of rows, For the matrix The maximum number of columns, For the matrix the number of rows, For the matrix The number of columns of the trash can channel is .
[0093] Train the image matching network until the loss function Convergence, and obtain the trained space-land matching network model.
[0094] S6, output matching results:
[0095] Use the trained space-ground matching network model in S5 to predict the points with the same name in the test data set in S1 and output the matching results.
[0096] In order to verify the performance of the automatic matching method of air-ground remote sensing images of the present invention, a set of matching results of air-ground images are first selected from a test set of an embodiment of the present invention for visual inspection, such as Figure 2 As shown; Figure 2The matching results are displayed, and the specific areas with dense distribution of same-name points in the matching results are locally enlarged. It can be seen that although there are differences in geometric distortion, viewpoint change and lighting change between the two images of each set of air-ground image pairs, the position accuracy of these matched same-name points is accurate.
[0097] Then, the above matching results are compared and analyzed with the other four matching methods to verify the effectiveness of the method of the present invention.
[0098] The matching performance is evaluated by quantitatively comparing the number of correct matches and the matching accuracy of the method of the present invention with four matching methods: OANet, Superglue, COTR and LightGlue. The matching results are shown in Table 1:
[0099] Table 1
[0100] Comparison Method OANet Superglue COTR LightGlue The present invention Correct Matches 121 134 74 489 789 Matching accuracy 37.12% 70.90% 74% 80.29% 91.31%
[0101] From the data in Table 1, it can be seen that the matching method of the present invention has the largest number of correct matches, the highest matching accuracy, and the best matching performance among all methods.
Claims
1. A method for automatic matching of air-ground remote sensing images, characterized in that: The steps include: S1, build a dataset of air-ground image pairs: Acquire an aerial image set and a ground image set containing the same area, and form an air-ground image pair with the corresponding aerial images and ground images; form an air-ground image pair data set with all acquired air-ground image pairs, and divide the air-ground image pair data set into a training data set and a test data set; S2, extract deep feature expression map: Use the residual network encoder to extract the features of the training data set in S1 and obtain deep feature expression graphs of different scales; S3, obtain the structural feature attention map with weak boundary attributes: First, the deep feature expression maps of different scales obtained by S2 are spliced to obtain the feature map ; Then, in the feature map The multi-scale deep features are averaged in the channel dimension to obtain a channel-averaged feature map; finally, the channel-averaged feature map is Convolutional layers and Activation function operation, outputting a structural feature attention map with weak boundary properties; S4, performs global and local context information enhancement, specifically: S41, perform global context information enhancement: Extracting feature descriptors of global context in the deep feature expression graph ; S42, perform local context information enhancement: Extracting feature descriptors of local context in the deep feature expression graph ; S43, the With the Fusion obtains feature descriptors with global and local information ; S5, training image matching network: The The structural feature attention map obtained by S3 is sent to the image matching network, and then the Sinkhorn algorithm and the garbage bin channel are used to obtain the matching matrix ; Using the matching matrix Constructing a loss function based on graph structure , we have the following formula: ; in, The matching matrix The maximum number of rows, The matching matrix The maximum number of columns, For the matrix the number of rows, For the matrix The number of columns of the trash can channel is ; Train the image matching network until the loss function Converge and obtain the trained space-land matching network model; S6, output matching results: Use the trained space-ground matching network model in S5 to predict the points with the same name in the test data set in S1 and output the matching results.
2. The method for automatic matching of air-ground remote sensing images according to claim 1, characterized in that: In S1, the rotation angle difference between the aerial image and the ground image in the air-ground image pair is within 20°.
3. The method for automatic matching of air-ground remote sensing images according to claim 1 is characterized in that: The deep feature expression graphs of different scales in S2 include .
4. The method for automatic matching of air-ground remote sensing images according to claim 3 is characterized in that: The deep feature expression graph in S2 is specifically: and ;in is the height, is the width, is the number of channels, .
5. The method for automatic matching of air-ground remote sensing images according to claim 3 is characterized in that: S41 includes the following steps: S411, in all the deep feature expression graphs, dividing the deep feature expression graphs with higher dimensions into a number of feature blocks of fixed sizes by adaptive average pooling; S412, dividing each of the characteristic image blocks into a plurality of flat vectorized two-dimensional patches , each of the vectorized two-dimensional patches The size is ; For the vectorized 2D patch Serial number; S413, position code: Use the following formula to calculate the feature containing the position code ; in, is the position information of the vectorized two-dimensional patch in the feature block, is the total number of vectorized 2D patches, is a trainable linear projection; S414, obtain hidden features of the global context: First, construct the transformer operation, which has the following two formulas: ; ; Among them, MSA stands for multi-head self-attention, MLP stands for multi-layer perceptron, and LN stands for layer normalization operator. Represents the number of iterations of the transformer operation, For the The result of the transformer operation. For the The result of the transformer operation. is the intermediate quantity of transformer operation, When , it means that no transformer operation is performed; Then, the feature containing the position code obtained in S413 conduct Transformer operation to obtain the feature descriptor of the global context .
6. The method for automatic matching of air-ground remote sensing images according to claim 5 is characterized in that: S413 specifically uses trainable linear projection Vectorize 2D patches Mapped to a potential 128-dimensional embedding space, which is then added to embedded in; used in Preserve vectorized 2D patches in The position information in the corresponding feature block.
7. The method for automatic matching of air-ground remote sensing images according to claim 5 is characterized in that: S414 .
8. The method for automatic matching of air-ground remote sensing images according to claim 3 is characterized by: S42 specifically includes the following steps: S421, interpolate all deep feature expression maps obtained in S2 to the same spatial size, and then aggregate them to obtain the interpolated feature map ; S422, use Convolutional layers and 3 Dilated convolutional layers to interpolate feature maps Improve the feature receptive field and obtain the feature descriptor of the local context .
9. The method for automatic matching of air-ground remote sensing images according to claim 8 is characterized in that: 3 of the above mentioned The dilation rates of the dilated convolutional layers are 6, 12, and 18 respectively.
