Hierarchical heterogenous remote sensing image matching method and device based on aggregation attention

Abstract

The invention discloses a hierarchical heterogenous remote sensing image matching method and device based on aggregation attention, and relates to the technical field of deep learning. A multi-branch structure of the lightweight convolutional neural network in the training stage is converted into a high-efficiency reasoning single-branch network through a re-parameterization technology, and the single-branch network is used to extract a feature descriptor of a to-be-matched image; based on the extracted feature descriptor of the to-be-matched image, alternately acting on the feature descriptor of the to-be-matched image through a self-attention mechanism and a cross attention mechanism to obtain an enhanced feature descriptor; hierarchical matching is carried out by adopting a preliminary rough matching strategy and a two-stage fine matching strategy so as to realize sub-pixel-level accurate matching; a confidence index prediction mechanism is introduced, and the confidence coefficient of each feature point is quantitatively evaluated according to the matching relation of the feature points between the to-be-matched images. The problems of low matching precision, long reasoning time and low matching efficiency in weak texture, large rotation or cross-modal scenes in the prior art are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and particularly relates to a hierarchical heterogeneous remote sensing image matching method and device based on aggregated attention. BACKGROUND

[0002] Matching of unmanned aerial vehicle (UAV) images and satellite images is one of the core tasks of cross-modal remote sensing image processing, aiming to achieve high-precision positioning, geographic information fusion or three-dimensional reconstruction by establishing geometric correspondence between images.

[0003] In the prior art, when performing a matching task, the search range of the image pair to be matched is narrowed down through spatial consistency retrieval to ensure that the image pair has common features in spatial coverage. By detecting key points and constructing local descriptors, robustness to rotation and scale changes is achieved. And based on convolutional neural networks or Transformer models, high-level semantic features are extracted to enhance adaptability to imaging differences.

[0004] However, the prior art cannot effectively adapt to the differences in view, illumination, resolution and modal between UAV aerial images and satellite images, resulting in inconsistent feature representation of the same ground object, low matching accuracy in weak texture, large rotation or cross-modal scenes, and dependence on deep networks or intensive computation, long inference time and low matching efficiency. SUMMARY

[0005] In the embodiments of the present application, by providing a hierarchical heterogeneous remote sensing image matching method based on aggregated attention, the prior art has low matching accuracy in weak texture, large rotation or cross-modal scenes, and depends on deep networks or intensive computation, solving the problem of long inference time and low matching efficiency of the prior art.

[0006] In the first aspect, the embodiments of the present application provide a hierarchical heterogeneous remote sensing image matching method based on aggregated attention, which comprises: converting the multi-branch structure of a lightweight convolutional neural network in the training stage into a single-branch network for efficient inference through a reparameterization technique, and extracting feature descriptors of the images to be matched using the single-branch network; wherein the images to be matched are UAV aerial images and satellite images; based on the extracted feature descriptors of the images to be matched, the self-attention mechanism and the cross-attention mechanism are alternately applied to the feature descriptors of the images to be matched to obtain enhanced feature descriptors; based on the enhanced feature descriptors, a hierarchical matching is performed using a preliminary coarse matching strategy and a two-stage fine matching strategy, a score matrix is constructed and a mutual nearest neighbor matching strategy is used to realize accurate matching at the sub-pixel level; by introducing a confidence index prediction mechanism, the matching relationship between the feature points of the images to be matched is quantitatively evaluated by quantitatively evaluating the confidence of each feature point.

[0007] In a possible implementation, the feature descriptor of the extracted image to be matched is alternately subjected to a self-attention mechanism and a cross-attention mechanism to obtain an enhanced feature descriptor, including: the expression of the self-attention mechanism is Attention(Q, K, V) = softmax(QK T )T V; wherein Q is a query matrix generated by linear transformation of the feature descriptor of the image to be matched, K is a key matrix generated by linear transformation of the feature descriptor of the image to be matched, V is a value matrix generated by linear transformation of the feature descriptor of the image to be matched, softmax is a normalization function, Attention is a self-attention mechanism function, and T is a transposition operation; the expression of the cross-attention mechanism is: wherein Q A is a query matrix generated by linear transformation of the feature descriptor of the aerial photograph of the unmanned aerial vehicle, K B is a key matrix generated by linear transformation of the feature descriptor of the satellite photograph, V B is a value matrix generated by linear transformation of the feature descriptor of the satellite photograph, Cross-Attention A is a cross-attention mechanism function of the aerial photograph of the unmanned aerial vehicle; the self-attention mechanism and the cross-attention mechanism are stacked for a preset number of times, and the output of each time is used as the input of the next time to form a recursive feature enhancement process.

[0008] In a possible implementation, the process of alternately subjecting the feature descriptor of the image to be matched to the self-attention mechanism and the cross-attention mechanism to obtain the enhanced feature descriptor specifically includes: using a deep convolutional network to aggregate the feature map of the aerial photograph of the unmanned aerial vehicle to obtain an aggregated first feature map; using a maximum pooling layer to aggregate the feature map of the satellite photograph to obtain an aggregated second feature map; and processing the feature descriptors of the first feature map and the second feature map according to two-dimensional rotation position encoding and the self-attention mechanism and the cross-attention mechanism; the expression of the two-dimensional rotation position encoding is: wherein a ij is an attention score between the feature descriptors, q i is a query matrix generated by linear transformation of the i th feature descriptor, k j is a key matrix generated by linear transformation of the j th feature descriptor, R(x j -x i , y j -y i ) is a block diagonal matrix, x i is a horizontal direction coordinate of the i th feature descriptor on the first feature map, y i is a vertical direction coordinate of the i th feature descriptor on the first feature map, x jis the coordinate of the jth feature descriptor in the horizontal direction on the second feature map, y j is the coordinate of the jth feature descriptor in the vertical direction on the second feature map; the first feature map and the second feature map processed by the two-dimensional rotation position coding, the self-attention mechanism and the cross-attention mechanism are up-sampled to restore the size of the original feature map; wherein the original feature map is the feature map of the unmanned aerial vehicle aerial photograph and the feature map of the satellite map; the feature maps after up-sampling are fused with the original feature maps respectively to obtain the first final feature map and the second final feature map, so as to obtain the enhanced feature descriptor.

[0009] In a possible implementation, the preliminary coarse matching strategy includes: performing inner product on the first final feature map and the second final feature map to construct a score matrix, and the expression of the score matrix is: wherein S(i,j) is an element in the score matrix, representing the similarity score between the ith feature descriptor in the first final feature map and the jth feature descriptor in the second final feature map, is the position of the ith feature descriptor in the first final feature map, is the position of the jth feature descriptor in the second final feature map, <·,·> is the inner product operation, θ c is a score threshold; a two-dimensional softmax operator is applied on the score matrix to obtain a probability matrix of mutual nearest neighbor matching, and the expression of the probability matrix is: wherein is an element in the probability matrix, representing the matching probability between the ith feature descriptor in the first final feature map and the jth feature descriptor in the second final feature map, softmax(S(i,·)) j is the jth element after applying the softmax function on the ith row of the score matrix, softmax(S(·,j)) i is the ith element after applying the softmax function on the jth row of the score matrix; under the condition of meeting the mutual nearest neighbor matching constraint, the matching items higher than the score threshold are selected to establish the coarse matching, and the expression of the coarse matching is: wherein is the established coarse matching set, is a matching pair, representing the matching between the ith feature descriptor in the first final feature map and the jth feature descriptor in the second final feature map, is the ith feature descriptor in the first final feature map, is the jth feature descriptor in the second final feature map, is the mutual nearest neighbor matching set, is an element in the probability matrix, θ c is a score threshold.

[0010] In a possible implementation, the two-stage fine matching strategy comprises: performing inner product on corresponding fine features in the coarse matching set to obtain a local block score matrix: wherein S'(i,j) is an element in the local block score matrix, representing a similarity score between an i-th fine feature descriptor in the first final feature map and a j-th fine feature descriptor in the second final feature map, is a fine feature set corresponding to the first final feature map, is a fine feature set corresponding to the second final feature map; performing mutual nearest neighbor matching search on the local block score matrix, selecting a matching pair with the highest similarity score to perform intermediate pixel-level matching to limit the number of matches; associating the feature points of the first final feature map with feature blocks centered on corresponding intermediate pixel-level matching points in the second final feature map, and applying a softmax operator to process the associated feature data to obtain a matching distribution matrix, and obtaining a final fine matching result by calculating an expected value of the matching distribution matrix.

[0011] In a possible implementation, the confidence index prediction mechanism is introduced to quantitatively evaluate the confidence of each feature point for the matching relationship between the feature points of the to-be-matched images, comprising: at the end of each layer of self-attention mechanism and cross-attention mechanism, a confidence index of each feature point is calculated, and the expression of the confidence index is: c i =Sigmoid(MLP(x i ))∈[0,1];wherein c i is the confidence index, Sigmoid is a Sigmoid function, MLP is a multi-layer perception, x i is a feature vector corresponding to the feature point; for a given layer, a decay confidence threshold is determined according to the verification accuracy of each classifier, and the expression of the decay confidence threshold is: wherein is the decay confidence threshold, L is the total number of layers of the network, is the current layer number; if the confidence index of the feature point corresponding to the to-be-matched image is greater than the decay confidence threshold of the current layer, the confidence of the feature point is high; the ratio of the feature points with high confidence in all feature points is counted, and if the ratio of the feature points with high confidence is greater than a preset value in the current layer, the inference in the current layer is ended; the judgment formula for ending the inference is: wherein exit is a Boolean variable, used to represent whether the inference in the current layer is ended, N feature is the total number of feature points, I∈{A,B} is an index variable, A is a UAV aerial image, B is a satellite image, i is an index variable, is an index set of the feature points, For the indicator function, when the condition in the parentheses is true, the function value is 1, otherwise 0, is a confidence index of a feature point corresponding to the image pair to be matched, and a is a preset value.

[0012] In a second aspect, the embodiments of the present application provide a hierarchical heterogeneous remote sensing image matching device based on aggregated attention, which comprises: a conversion module configured to convert a multi-branch structure of a lightweight convolutional neural network in a training stage into a single-branch network for efficient inference through a reparameterization technique, and extract feature descriptors of images to be matched using the single-branch network; wherein the images to be matched are unmanned aerial vehicle aerial images and satellite images; an action module configured to alternately act on the feature descriptors of the images to be matched through a self-attention mechanism and a cross-attention mechanism based on the extracted feature descriptors of the images to be matched, and obtain enhanced feature descriptors; a hierarchical matching module configured to perform hierarchical matching based on the enhanced feature descriptors using a preliminary coarse matching strategy and a two-stage fine matching strategy, construct a score matrix, and use a mutual nearest neighbor matching strategy to realize accurate matching at a sub-pixel level; and an evaluation module configured to quantitatively evaluate the confidence of each feature point by introducing a confidence index prediction mechanism for the matching relationship between the feature points of the images to be matched.

[0013] In a third aspect, the embodiments of the present application provide a hierarchical heterogeneous remote sensing image matching server based on aggregated attention, comprising a memory and a processor; the memory is configured to store computer executable instructions; the processor is configured to execute the computer executable instructions to implement the method of the first aspect or any possible implementation manner of the first aspect.

[0014] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, which stores executable instructions, and a computer executing the executable instructions can implement the method of the first aspect or any possible implementation manner of the first aspect.

[0015] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects: the embodiments of the present application provide a hierarchical heterogeneous remote sensing image matching method based on aggregated attention, a multi-branch structure of a lightweight convolutional neural network in a training stage is converted into a single-branch network for efficient inference through a reparameterization technique, and a feature descriptor of a to-be-matched image is extracted using the single-branch network, while the feature extraction capability is maintained, the computational overhead in an inference stage is significantly reduced. Based on the extracted feature descriptor of the to-be-matched image, the self-attention mechanism and the cross-attention mechanism are alternately applied to the feature descriptor of the to-be-matched image, an enhanced feature descriptor is obtained, the global correlation between aerial images and satellite images is explicitly modeled, and the adaptability to differences in perspective, scale and modalities is enhanced. Based on the enhanced feature descriptor, a hierarchical matching is performed using a preliminary coarse matching strategy and a two-stage fine matching strategy, a score matrix is constructed, and a mutual nearest neighbor matching strategy is used to realize sub-pixel level accurate matching, the computational efficiency is significantly improved on the premise of ensuring accuracy, a confidence index prediction mechanism is introduced, the confidence of each feature point is quantitatively evaluated according to the matching relationship between the feature points of the to-be-matched images, and it is dynamically determined whether to terminate the inference in advance. The problems of low matching accuracy in the prior art in weak texture, large rotation or cross-modality scenes, dependence on deep networks or dense calculation, long inference time and low matching efficiency are solved. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the description of the embodiments of the present application or the prior art will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 A flowchart of a hierarchical heterogeneous remote sensing image matching method based on aggregated attention provided by the embodiments of the present application;

[0018] Figure 2 A hierarchical heterogeneous remote sensing image matching method based on aggregated attention provided by the embodiments of the present application;

[0019] Figure 3 A schematic diagram of a training stage provided by the embodiments of the present application;

[0020] Figure 4 A schematic diagram of a reparameterization technique provided by the embodiments of the present application;

[0021] Figure 5 A schematic diagram of a self-attention mechanism provided by the embodiments of the present application;

[0022] Figure 6A schematic diagram of an improved aggregate attention mechanism provided in an embodiment of the present application;

[0023] Figure 7 Schematic diagram of image pairs that are easy to match and image pairs that are difficult to match provided in an embodiment of the present application;

[0024] Figure 8 A schematic diagram of the visualization effects of various models on a pair of typical scene images provided in an embodiment of the present application;

[0025] Figure 9 A schematic diagram of the visualization effect of matching each model provided in the embodiment of the present application on a weakly textured heterogeneous remote sensing image;

[0026] Figure 10 A schematic diagram of the quantization results of each model provided in the embodiments of the present application on an image pair with a difference in rotation angle;

[0027] Figure 11 A schematic diagram illustrating the distribution of matching points and alignment effects of various methods provided in the embodiments of the present application on scale-difference images;

[0028] Figure 12 A schematic diagram of a hierarchical heterogeneous remote sensing image matching device based on aggregated attention provided in an embodiment of the present application;

[0029] Figure 13 A schematic diagram of a hierarchical heterogeneous remote sensing image matching server based on aggregated attention provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0031] The following description of some of the technologies involved in the embodiments of this application is provided to facilitate understanding and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for the sake of clarity and conciseness, some descriptions of well-known functions and structures are omitted from the following description.

[0032] The embodiment of the present application provides a hierarchical heterogeneous remote sensing image matching method based on aggregated attention, such as Figure 1 As shown, the method includes steps S101 to S104. Figure 1The execution sequence shown is only one of the embodiments of the present application, and does not represent the only execution sequence of the hierarchical heterogeneous remote sensing image matching method based on aggregated attention. In the case of achieving the final result, Figure 1 The steps shown can be executed in parallel or in reverse.

[0033] S101: Convert the multi-branch structure of the lightweight convolutional neural network in the training stage into a single-branch network for efficient inference through a reparameterization technique, and extract the feature descriptor of the image to be matched using the single-branch network. The image to be matched is a UAV aerial photograph and a satellite image.

[0034] Figure 2 The overall framework of the hierarchical heterogeneous remote sensing image matching method based on aggregated attention provided by the embodiments of the present application is shown. In the matching task of UAV aerial photographs and satellite images, feature extraction is a key link. The present application first extracts image feature descriptors through a lightweight convolutional neural network (CNN), and then performs feature enhancement and matching. Unlike many methods that use multi-branch networks (such as ResNet) for feature extraction, the present application uses a reparameterized lightweight single-branch network to significantly improve inference efficiency while ensuring model performance.

[0035] Figure 3The schematic diagram of the training stage provided by the embodiment of the present application is shown. In order to obtain stronger representation capability, the training stage adopts a convolutional neural network with a multi-branch structure with residual connection. The feature extraction part takes a single-channel grayscale image as input and passes through multiple RepVGG modules. In the first stage, the first RepVGG module includes a 3×3 convolution branch (convolution kernel size 3×3, stride 2, padding 1, containing a batch normalization layer, used to extract local features) and a 1×1 convolution branch (convolution kernel size 1×1, stride 2, no padding, containing a batch normalization layer, used to extract global features). The results are added and activated by ReLU, and the output channel is 64, and the feature map length and width become 1 / 2 of the original image. In the second stage, two RepVGG modules with a stride of 1 are stacked to ensure that the feature map size does not change, while extracting deeper features, and the channel number remains unchanged at 64. The RepVGG in this stage is obtained by adding a 3×3 convolution branch, a 1×1 convolution branch and a branch with only batch normalization. At this time, the feature map length and width are still 1 / 2 of the original image, and can be directly used in the subsequent fine matching stage. In the third stage, four RepVGG modules are stacked. The first module has a stride of 2, an output channel of 128, and no identity mapping; the last three modules have a stride of 1, a channel number of 128, and contain identity mapping. At this time, the output feature map length and width are both 1 / 4 of the original image, the channel number is 128, and after the subsequent fusion network, it can be used in the fine matching stage. In the fourth stage, 14 RepVGG modules are stacked. The first module has a stride of 2, an output channel of 256, and no identity mapping, which is used to reduce the resolution and increase the receptive field; the last 13 modules have a stride of 1, a channel number of 256, and contain identity mapping, which can extract deeper features. At this time, the output feature map length and width are both 1 / 8 of the original image, the channel number is 256, and can be used in the subsequent coarse matching stage. Through the above operations, feature maps with lengths and widths of 1 / 2, 1 / 4 and 1 / 8 of the original image can be obtained, which are used in the subsequent preliminary coarse matching strategy and two-stage fine matching strategy. RepVGG uses different size convolution kernels in the training stage to extract features of different scales.

[0036] Figure 4 The schematic diagram of the reparameterization technology provided by the embodiment of the present application is shown. Through the reparameterization technology, the multi-branch structure of the convolutional neural network is converted into a single-branch network with a 3×3 convolution branch, which improves the running efficiency and optimizes the space occupation, and has a significant impact on the speed improvement of the entire matching network.

[0037] Due to the change of UAV flight and perspective, the rotation change of UAV aerial images relative to satellite images is common, and the rotation angle is diverse. The rotation equivariant convolution kernel developed by the academia has limitations. On the one hand, it can only realize rotation invariance of a specific angle and cannot realize high-precision rotation invariance of all angles; on the other hand, it needs to modify the convolution operator, which will reduce the feature extraction efficiency. To solve these challenges, the rotation equivariant regularization strategy is adopted. The strategy adaptively imposes appropriate symmetry constraints on the data while maintaining the representation accuracy of the network. It is a regularizer designed to enhance rotation equivariance, which innovatively extends the ideas of data augmentation-based and equivariant-based methods, and is realized through self-supervised learning and spatial rotation and cyclic channel shift of feature maps inferred in the equivariant framework.

[0038] Let Ψ be a convolutional mapping from the input feature space to the output feature space, and S be a subgroup of the rotation transformation, that is, where A k is a rotation matrix in the subgroup S, k is an integer, which is used to specify different rotation angles, and t determines the number of rotation matrices contained in the subgroup S. Ψ is equivariant to S, and for any rotation matrix there is: where I is an input image, which can be a UAV aerial image or a satellite image, and the rotation matrix A k can be applied to the input image I to obtain the rotated image and represent the effects of the transformation on the input image and the output feature, respectively, and [·] represents the composition of functions.

[0039] In the equivariant network, the rotation of the input image I∈R h ×w×n will cause the rotation of the feature map F∈R h ×w×n in the convolution. In addition to the rotation in the spatial dimension, the feature map F will also be accompanied by the pattern transformation of the rotation group S. Specifically, the rotation of the feature map will be consistent with the spatial rotation in the first two dimensions and will be cyclically moved along the final dimension. Formally, for there is: where is a rotation matrix in the subgroup S, is a transformation operator, which represents the application of the rotation transformation to the feature map F, is the result of the feature map after the rotation transformation, is a tensor with size h×w×n, which is treated as an n-channel image when performing spatial rotation .

[0040] To overcome the limitations brought by equivariant network error and effectively incorporate the rotational symmetry prior in the image, the regularization term is introduced to constrain the feature map of each convolutional layer, reduce the equivariant error and enhance the equivariant of the network. wherein L layer is a regularization term, is the first l layers of the convolutional network. is a rotation matrix randomly selected from {2kπ / t|k=1,2,...,t}, is the square of the Frobenius norm. For there is: wherein F l is the feature map obtained by processing the input image I through the first l layers of the convolutional network. For there is: wherein r(I) is the rotated input image, is the feature representation obtained by inputting the rotated image into the first l layers of the convolutional network, is the image rotated at different angles. According to the above formula, the error between the feature map obtained by rotating the image before convolution and the feature map obtained by convolving the image first and then rotating and circularly shifting in the channel is calculated. Instead of using the equivariant convolution architecture which will cause representation accuracy error, the ordinary convolution layer which can maintain high accuracy is still used. The goal of the present application is to guide the feature map after each convolutional layer of the image so that they can approach the strictly equivariant feature map in the equivariant network. The regularization term of the entire network can be represented as: wherein L equi is the regularization term of the entire network, is the mathematical expectation, representing the expectation of the rotation matrix .

[0041] The present application uses L equi to guide the training, allowing the model to learn an adaptive symmetry prior from the data. The CNN trained with the regularization term of the equation is not constrained by the strict rotational symmetry prior hypothesis imposed by the rotational equivariant convolution. On the contrary, they learn a rotational symmetry prior that adapts to the actual data distribution. In addition, unlike other methods with relaxed equivariant constraints, the method of the present application is not limited by the equivariant error unique to the equivariant convolution framework.

[0042] S102: Based on the extracted feature descriptor of the to-be-matched image, the feature descriptor of the to-be-matched image is alternately acted on by the self-attention mechanism and the cross-attention mechanism to obtain an enhanced feature descriptor.

[0043] Based on the extracted feature descriptor of the to-be-matched image, the feature descriptor of the to-be-matched image is alternately acted on by a self-attention mechanism and a cross-attention mechanism to obtain an enhanced feature descriptor, including the following steps.

[0044] Specifically, the self-attention mechanism realizes the learning of each feature point in the image to other global features, and the cross-attention mechanism realizes the learning of features between to-be-matched images.

[0045] The expression of the self-attention mechanism is: Attention(Q, K, V) = softmax(QK T )V. Wherein, Q is a query matrix generated by linear transformation of the feature descriptor of the to-be-matched image, K is a key matrix generated by linear transformation of the feature descriptor of the to-be-matched image, V is a value matrix generated by linear transformation of the feature descriptor of the to-be-matched image, softmax is a normalization function, Attention is a self-attention mechanism function, and T is a transposition operation.

[0046] Figure 5 The self-attention mechanism provided by the embodiment of the application is shown in the schematic diagram. The feature descriptor X o is Figure 5 one of the elements in fi.

[0047] Specifically, in the foregoing part, a feature map is obtained, wherein the feature point of each position is composed of the values of all channels to form a feature descriptor X i . Each feature descriptor can be regarded as a token (a basic unit). The entire feature map is flattened, that is, the feature descriptors X i of all positions are arranged in a certain order to obtain a sequence of tokens. Each token of the sequence corresponds to a position in the feature map. The sequence is input into a self-attention module, and the purpose is to enable each position to learn the feature information of other positions, thereby realizing long-distance feature information interaction. Specifically, each feature descriptor X i will be projected to obtain a query qi, a key ki, and a value vi through three matrices Wq, Wk, and Wv, respectively. That is, qi=X i Wq, ki=X i Wk, and vi=X i Wv. The core of the self-attention mechanism is to calculate the similarity of each query qi and all keys kj, and then normalize it through a softmax function to obtain an attention weight. The calculation of the attention weight is: Σsoftmax(qik j T )vj, wherein, k j Tis the transpose of the j-th key vector in the key matrix. Finally, the output feature of each position is obtained by weighted summation, i.e., the expression of self-attention mechanism.

[0048] It should be noted that Q, K, V are global matrices corresponding to the linear transformation results of all feature descriptors. qi, kj, vj are local vectors corresponding to the projection results of a single token.

[0049] The expression of cross-attention mechanism is: where Q A is the query matrix generated by linear transformation of the feature descriptor of the aerial photo of the unmanned aerial vehicle, K B is the key matrix generated by linear transformation of the feature descriptor of the satellite image, V B is the value matrix generated by linear transformation of the feature descriptor of the satellite image, Cross-Attention A is the cross-attention mechanism function of the aerial photo of the unmanned aerial vehicle.

[0050] Stack the self-attention mechanism and cross-attention mechanism for a preset number of times. In each stacking, the output of the previous time is taken as the input of the next time, forming a recursive feature enhancement process.

[0051] Specifically, each stacking redistributes the information weight through similarity measurement, so that the feature descriptor can dynamically adapt to different levels of semantic requirements. If the similarity between two features is high, more relevant information is extracted from each other's feature descriptor vectors. Such operation is conducive to the subsequent matching of self-attention mechanism and cross-attention mechanism. The alternative stacking of self-attention mechanism and cross-attention mechanism realizes the integration of information from local to global and from single modality to multi-modality.

[0052] Through the alternative action of self-attention mechanism and cross-attention mechanism on the feature descriptors of the image to be matched, the enhanced feature descriptors are obtained, which specifically include the following steps.

[0053] The feature map of the aerial photo of the unmanned aerial vehicle is aggregated using a deep convolutional network to obtain an aggregated first feature map.

[0054] The feature map of the satellite image is aggregated using a max-pooling layer to obtain an aggregated second feature map.

[0055] The feature descriptors of the first feature map and the second feature map are processed according to the two-dimensional rotation position encoding and the self-attention mechanism and the cross-attention mechanism.

[0056] Specifically, in the self-attention mechanism, directly flattening the feature map into a token sequence will lose the spatial position information of the image. Although the traditional method (such as absolute position encoding or relative position encoding) can partially alleviate this problem, it has the following shortcomings: sensitive to the training image size, if the image size changes in the inference stage, the matching effect will decrease significantly. Although easy to implement, it is prone to overfitting on the training set, and has insufficient robustness to geometric transformations (such as rotation, translation). The present application explicitly models the relative position relationship between features through two-dimensional rotation position encoding (RoPE), so that the model pays more attention to the interaction between features rather than the absolute position.

[0057] The two-dimensional rotation position encoding expression is: wherein a ij is the attention score between feature descriptors, q i is the query matrix generated by the linear transformation of the i th feature descriptor, k j is the key matrix generated by the linear transformation of the j th feature descriptor, R (x j -x i , y j -y i ) is a block diagonal matrix, x i is the coordinate of the i th feature descriptor in the horizontal direction of the first feature map, y i is the coordinate of the i th feature descriptor in the vertical direction of the first feature map, x j is the coordinate of the j th feature descriptor in the horizontal direction of the second feature map, y j is the coordinate of the j th feature descriptor in the vertical direction of the second feature map.

[0058] Specifically, wherein R (Δx, Δy) is a block diagonal matrix composed of a plurality of sub-matrices arranged diagonally, there are 4k / d sub-matrices, R k (Δx, Δy) is the specific form of each sub-matrix, Δx = x j -x i , Δy = y j -y i , θ k is a frequency parameter, k is the index of the sub-matrix, k ∈ [1, 2, …, d / 4], d is the total dimension of the feature descriptor. The present application encodes the index of the feature channel, and uses two-dimensional rotation position encoding to make the model pay more attention to the interaction between features rather than their specific positions, which is conducive to capturing the context of local features. It has stronger robustness to rotation, translation and scaling, and is suitable for local feature matching of different views. Moreover, the position encoding is only enabled in the self-attention mechanism, and is not used in the cross-attention mechanism.

[0059] The first feature map and the second feature map processed through two-dimensional rotation position coding and self-attention mechanism and cross-attention mechanism are up-sampled to restore the size of the original feature map. The original feature map is the feature map of the unmanned aerial vehicle aerial photograph and the feature map of the satellite photograph.

[0060] The up-sampled feature map is fused with the original feature map respectively to obtain the first final feature map and the second final feature map, so as to obtain an enhanced feature descriptor.

[0061] Specifically, in the unmanned aerial vehicle aerial photograph and satellite photograph matching task, although the traditional self-attention mechanism and cross-attention mechanism can effectively enhance the expression ability of the feature descriptor, the calculation complexity is O(N 2 d) (N is the number of tokens, and d is the length of the feature descriptor), which leads to low calculation efficiency and is difficult to meet the actual application requirements. To solve this problem, an improved aggregated attention mechanism is proposed in the present application, which optimizes the calculation efficiency through the following two key observations: 1. The attention area of adjacent query tokens is usually similar, and the redundant calculation can be reduced by aggregating adjacent tokens. 2. The attention weight of each query token is mainly concentrated in a few key tokens, and the calculation amount can be reduced by filtering significant tokens. The feature map is down-sampled through deep convolution and max-pooling operation to reduce the number of tokens while retaining key information.

[0062] Figure 6 The schematic diagram of the improved aggregated attention mechanism provided by the embodiment of the present application is shown in the figure. For the feature map processing of the unmanned aerial vehicle aerial photograph: f i ′ =Conv2D(f i ). Wherein, Conv2D is a stride depth convolution with a size of s x s, and the max-pooling layer is the same, f i is the feature map of the input unmanned aerial vehicle aerial photograph, f i ′ is the aggregated first feature map. For the feature map processing of the satellite photograph: f j ′ =MaxPool(f j ). Wherein, MaxPool is a max-pooling layer, f j is the feature map of the input satellite photograph, f j ′ is the aggregated second feature map. The number of tokens of the aggregated first feature map and the aggregated second feature map is reduced to the original s 2 , which significantly improves the efficiency of the attention stage.

[0063] Specifically, the application reduces the token number through deep convolution and max pooling, and reduces the attention calculation complexity from O(N 2 d) to O(N 2 d / s 2 ). Compared with linear attention, the application retains the nonlinear interaction ability through two-dimensional rotation position encoding and feature fusion. The rotation invariance of two-dimensional rotation position encoding improves the adaptability of the model to view angle changes and geometric distortion.

[0064] S103: Based on the enhanced feature descriptor, a preliminary coarse matching strategy and a two-stage fine matching strategy are used for hierarchical matching, a score matrix is constructed, and a mutual nearest neighbor matching strategy is used to realize sub-pixel level accurate matching.

[0065] In the task of matching unmanned aerial vehicle aerial images and satellite images, the following problems exist in matching directly through similarity measurement (such as inner product or Euclidean distance) of feature descriptors. Simply relying on threshold screening leads to multiple feature points being incorrectly matched to the same target point, destroying the uniqueness of matching. In a local area, the descriptors of multiple feature points can be highly similar, leading to nearest neighbor search mis-matching. The choice of similarity measurement (such as inner product, cosine similarity, etc.) directly affects the matching performance and needs to be carefully designed. To solve the above problems, the application uses a preliminary coarse matching strategy and a two-stage fine matching strategy for hierarchical matching, and finally realizes sub-pixel level accurate matching.

[0066] The preliminary coarse matching strategy includes the following steps.

[0067] The inner product of the first final feature map and the second final feature map is calculated to construct a score matrix, and the expression of the score matrix is: where S(i,j) is an element in the score matrix, representing the similarity score between the i-th feature descriptor in the first final feature map and the j-th feature descriptor in the second final feature map, is the position of the i-th feature descriptor in the first final feature map, is the position of the j-th feature descriptor in the second final feature map, and <·,·> is the inner product operation. c is a score threshold.

[0068] Specifically, the size of the first final feature map and the second final feature map is 1 / 8 of the original image. Each feature point corresponds to an enhanced feature descriptor. The attention mechanism enhances the discriminative ability of the descriptor, but does not change the dimension of the feature map, i.e. each feature point still corresponds to a vector.

[0069] It should be noted that the score threshold θ cThe numerical range for controlling the similarity score. The greater the similarity score, the more likely it is that the i-th feature descriptor in the first final feature map and the j-th feature descriptor in the second final feature map correspond to matching feature points. The specific value of this score threshold is dynamic, and in the early stage of training, a smaller value of the order of 0.002 is generally taken according to the actual situation, and the value is gradually increased as the network training progresses. Generally, 0.1 is used in testing.

[0070] A softmax operator in two dimensions is applied on the score matrix to obtain a probability matrix of mutual nearest neighbor matching, and the expression of the probability matrix is: wherein, is an element of the probability matrix, representing the matching probability between the i-th feature descriptor in the first final feature map and the j-th feature descriptor in the second final feature map, softmax(S(i,·)) k is the j-th element after applying the softmax function to the i-th row of the score matrix, softmax(S(·,j)) i is the i-th element after applying the softmax function to the j-th row of the score matrix.

[0071] Under the condition of meeting the mutual nearest neighbor matching constraint, the matching items higher than the score threshold are selected to establish a coarse matching, and the expression of the coarse matching is: wherein, is the established coarse matching set, is a matching pair, representing the matching between the i-th feature descriptor in the first final feature map and the j-th feature descriptor in the second final feature map, is the i-th feature descriptor in the first final feature map, is the j-th feature descriptor in the second final feature map, is a mutual nearest neighbor matching set, is an element in the probability matrix, θ c is the score threshold.

[0072] Specifically, in the preliminary coarse matching strategy stage, a high-confidence coarse matching set is screened out through inner product and mutual nearest neighbor (MNN) matching However, the accuracy of the coarse matching is only at the pixel level, which cannot meet the demand for sub-pixel level accuracy in the matching of unmanned aerial vehicle aerial images and satellite images. Therefore, the present application proposes a two-stage fine matching strategy, which gradually refines the coarse matching to ultimately realize sub-pixel level accurate matching.

[0073] The two-stage fine matching strategy includes the following steps.

[0074] Specifically, unlike methods such as LoFTR that require an additional feature conversion network, the present application adaptively fuses multi-scale features (1 / 8, 1 / 4, 1 / 2 resolution) of the initial feature extraction backbone network through a shallow feedforward network to generate a fine feature map and The advantage is to avoid introducing additional computational overhead, and only to improve the feature resolution through convolution and up-sampling operations. The fine feature block integrates multi-scale context information and enhances the discriminative ability. The fine feature map and The local feature block is cropped from the position of each matched pair The local feature block is cropped from the position of each matched pair and The position of the i-th feature descriptor in the first final feature map and the position of the j-th feature descriptor in the second final feature map The feature block size needs to balance the computational efficiency and matching accuracy.

[0075] The inner product of the corresponding fine features in the coarse matching set is performed to obtain a local block score matrix: Where S'(i,j) is an element in the local block score matrix, representing the similarity score between the i-th fine feature descriptor in the first final feature map and the j-th fine feature descriptor in the second final feature map, is the fine feature set corresponding to the first final feature map, is the fine feature set corresponding to the second final feature map.

[0076] The local block score matrix is subjected to mutual nearest neighbor matching search, and the matching pair with the highest similarity score is selected for intermediate pixel-level matching to limit the number of matches.

[0077] The feature points of the first final feature map are associated with the feature blocks centered on the corresponding intermediate pixel-level matching points in the second final feature map, and the softmax operator is applied to process the associated feature data to obtain a matching distribution matrix. The final fine matching result is obtained by calculating the expectation value of the matching distribution matrix.

[0078] Specifically, the size of the feature block centered on the corresponding intermediate pixel-level matching point in the second final feature map can be 3x3. And this size is used as a local window, avoiding the interference of irrelevant areas on the matching result, significantly reducing the position variance. The use of local window makes the similarity score calculation only need to be performed within a limited range, which not only maintains the computational efficiency, but also enhances the robustness of the matching.

[0079] Further, to realize the end-to-end training of the hierarchical matching, the present application supervises the preliminary coarse matching strategy and the two-stage fine matching strategy respectively, and optimizes the network parameters through a weighted loss function. The specific strategy is as follows.

[0080] The goal of the preliminary coarse matching strategy is to supervise the score matrix generated by the coarse matching, so that it can accurately predict the true value matching label Through the depth map and image pose, the grid points of the first final feature map are mapped to the second final feature map to generate the true value matching label Using log-likelihood loss (cross entropy loss), Matching probability at position To conduct supervision: in, is the supervised loss function of the coarse matching stage. The goal of the two-stage fine matching strategy is to supervise the sub-pixel matching results generated by the two-stage fine matching. Make it as close to the true value as possible The first-stage loss of the fine matching strategy It is based on pixel-level ground truth fine matching by minimizing each fine local score matrix S l The log-likelihood loss is similar to the coarse loss, that is: in, is the ground truth matching label of the first stage of the fine matching strategy. The second stage loss of the fine matching strategy The position of the final sub-pixel matching result and true value position The loss between. That is: in, is the second stage matching pair of the fine matching strategy gather, is the feature point of the first final feature map, i.e. the query point, is the corresponding matching point in the second final feature map inferred by the model, and Indicates the actual corresponding matching point in the second final feature map. For each query point We also calculate the total variance of the corresponding heatmap by The goal is to optimize the refined position with lower uncertainty and obtain the final weighted loss function. from Map to And use the true value to calculate In calculation When, if The final implied position falls on Outside the local window, ignore During training, the gradients do not pass Backpropagation. Ultimately, the total loss is the weighted sum of all supervisions: in, is the total loss, α is 1.0, β is 0.25, and γ is 1×10 -6 .

[0081] S104: By introducing a confidence index prediction mechanism, the confidence of each feature point is quantitatively evaluated for the matching relationship between the feature points of the image pairs to be matched.

[0082] Figure 7 The schematic diagram of the easy-to-match image pair and the difficult-to-match image pair provided by the embodiment of the present application. Figure 7 (a) in the above is an easy-to-match image pair. Figure 7 (b) in the above is a difficult-to-match image pair. Intuitively, the reasoning speed on the easy-to-match image pair should be much faster than on the challenging pair, which is inspired by the way humans process visual information. To achieve this, two aspects need to be considered: 1. Predict the matching situation after each attention calculation. 2. Enable the model to learn and predict whether further calculation is needed. The present application adjusts the number of reasoning layers dynamically to adapt the model to the matching difficulty of different image pairs, realizes fast reasoning for easy-to-match image pairs, and terminates calculation at an early stage. For difficult-to-match image pairs, sufficient calculation is performed.

[0083] By introducing a confidence index prediction mechanism, the confidence of each feature point is quantitatively evaluated for the matching relationship between the feature points of the image pairs to be matched, including the following steps.

[0084] At the end of each layer of self-attention mechanism and cross-attention mechanism, the confidence index of each feature point is calculated, and the expression of the confidence index is: c i =Sigmoid(MLP(x i ))∈[0,1]. Wherein, c i is the confidence index, Sigmoid is the Sigmoid function, MLP is the multi-layer perception, x i is the feature vector corresponding to the feature point.

[0085] Specifically, the confidence index represents the reliability of the current feature point, and the higher the value, the closer the matching result is to the final prediction. The MLP only increases about 2% reasoning time in the worst case, which is usually much lower than this value.

[0086] For a given layer, according to the verification accuracy of each classifier, the decay confidence threshold is determined, and the expression of the decay confidence threshold is: wherein, is the decay confidence threshold, L is the total number of network layers, is the current layer number.

[0087] Specifically, the confidence of the early classifier is usually low, which needs to be adapted to the actual accuracy through the decay threshold.

[0088] If the confidence index of the feature point corresponding to the image pair to be matched is greater than the decay confidence threshold of the current layer, the confidence of the feature point is high.

[0089] The ratio of the feature points with high confidence is counted among all the feature points. If the ratio of the feature points with high confidence is greater than a preset value in the current layer, the inference is ended in the current layer.

[0090] The judgment formula for ending the inference is: Wherein, exit is a Boolean variable, used to indicate whether the inference is ended in the current layer, N feature is the total number of feature points, I is an index variable in I∈{A, B}, A is the aerial image of the unmanned aerial vehicle, B is the satellite image, is an index variable, is the index set of the feature points, is an indicator function, the function value is 1 when the condition in the parentheses is true, otherwise it is 0, is the confidence index of the feature point corresponding to the image pair to be matched, and a is a preset value.

[0091] Specifically, if a = 95%, the current feature point is considered to have high confidence, and the algorithm stops the network inference. In this case, the average running time is reduced by 25%, and there is almost no loss in accuracy on downstream tasks.

[0092] Further, the inference ending judgment can be trained separately from the whole matching process, and will not affect the expression of the whole confidence index. The MLP predicts whether the output of each layer is the same as the last layer. When training, the binary cross-entropy loss of each layer is minimized.

[0093] The target of the present application is to optimize the cross-modal matching efficiency and accuracy of the aerial image of the unmanned aerial vehicle and the satellite image, and to adapt to the challenge scenarios such as small visual overlap and large appearance change. The confidence index prediction mechanism is introduced to dynamically adjust the number of inference layers, realize fast inference of easy-to-match image pairs and deep calculation of difficult-to-match image pairs. The hierarchical matching mechanism is adopted to improve the robustness of the model to rotation difference and weak texture scene.

[0094] The satellite image of the present application is downloaded from QGIS and other platforms as a large image of level 18 (0.57m resolution). The aerial image is collected by the ANAS system carried by the unmanned aerial vehicle, with a resolution of 1080p, covering typical landforms such as cities, gobi and plateaus. The training data is aligned with the aerial large image and the satellite large image through GPS information, cropped to 512x512 pairs, and subjected to rotation and scaling disturbance. The test data is obtained by real-time acquisition of aerial images, and the nearest image in the satellite image library is searched for fine matching. The evaluation indexes are AUC@5 / 10 / 15px, Repetition (Rep) and Reprojection Error.

[0095] Table 1 is a matching quality result table of each method on a complex heterogeneous remote sensing image dataset. OURs are the models of the present application. px is pixel. Model is model. Matches is the number of matches. AUC is the area under the curve, the higher the value, the better the model performance. Rep is the repeatability, the higher the value, the better the model performance. H error is the reprojection error, the lower the value, the better. The arrow in the table indicates that the higher the corresponding index is better. The arrow in the table indicates that the lower the corresponding index is better. The dataset is 500 pairs of typical scene images, covering weak texture, rotation, scale difference, etc. The comparison method is SuperGlue, LoFTR, DKM, EfficientLoFTR, LightGlue. The number of matching points (3267) of the present application (OURs) is significantly higher than that of other methods, indicating that it can extract more effective matching points. The AUC@5px (0.23) of the present application is better than that of other methods, which reflects its superior performance in high-precision demand scenarios. The reprojection error (24.92) of the present application is much lower than that of other methods, indicating that it has higher matching alignment accuracy. The method of the present application has wide coverage and high consistency of matching points, and the alignment effect is optimal.

[0096] Figure 8 is a schematic diagram of the visual effect of each model provided by the embodiment of the present application on a typical scene image pair. It contains feature point matching correspondence and projection mapping relationship. The colored lines in each group of images represent the correspondence between the points on the aerial photograph and the points on the satellite photograph. After RANSAC is performed on all matching pairs to remove outliers and estimate the projection matrix, the projection matrix H is obtained. The aerial photograph is projected and superimposed on the satellite photograph using the projection matrix H to obtain the right side effect of each group of images. From Figure 8 It can be seen that SuperGlue has more matching pairs, and most of the effective matching points are on the edge of the building, especially the edge with significant color segmentation, but the alignment effect is not very good because of some false matching. The LoFTR method should be affected by the scale change, and the extracted feature points are relatively scattered, but the matching is all wrong. The number of matching pairs of DKM is also relatively large, and the consistency of the matching pairs is good, but the alignment effect still has room for improvement. EfficientLoFTR has a large number of matching pairs, but also has a large number of false matching pairs, but it can be corrected by RANSAC, and the alignment effect is good. LightGlueh and OmniGlue have fewer and more scattered matching points, but LightGlue extracts some road features, so the final matching result is correct. The method proposed in the present application has more matching points, a larger coverage range, and better consistency of matching pairs, and the final alignment effect is the best.

[0097] Table 1

[0098]

[0099] Table 2 is a table of matching results of each method on a weak-texture cross-sensor remote sensing image dataset. The dataset is 500 groups of weak-texture cross-sensor remote sensing image pairs (such as night infrared images). The number of matching points (2150) of the present application still remains at a high level. The repeatability (0.66) of the present application is higher than that of other methods. The road features extracted by the present application are richer and the alignment effect is better.

[0100] Figure 9 The schematic diagram of the visual effect of matching of each model provided in the embodiments of the present application on weak-texture cross-sensor remote sensing images. Figure 9 A night infrared image is selected as an example for visual display. Except for SuperGlue, a plurality of methods can extract a large number of matching points in the area without obvious high-level semantic features to support subsequent matching, but the method proposed in the present application extracts more matching points and has better consistency, and only the method proposed in the present application extracts the road features on the lower side of the image with better distinguishability, so the alignment effect is better, and other methods do not extract the road features on the lower end with high distinguishability, so there is a certain deviation in alignment.

[0101] Table 2

[0102]

[0103] Table 3 is a table of matching results of each method on a cross-sensor remote sensing image dataset with rotation difference. The dataset is 500 groups of rotation difference image pairs (0-360 degrees). The AUC@15px (0.68) of the present application is significantly better than that of other methods. The reprojection error (23.16) of the present application is much lower than that of other methods.

[0104] Figure 10 The schematic diagram of the quantitative results of each model provided in the embodiments of the present application on the image pair with rotation angle difference. Figure 10 The visual results of each model on the image with rotation angle difference are shown. The method proposed in the present application still has the most matching pairs, and the consistency of the matching pairs is also good, and the final matching result is correct. The number of matching pairs of SuperGlue is the least, and all the matching pairs are false matching, so the final alignment result is false. The matching points of LoFTR are still distributed relatively dispersedly, but should be affected by the position coding, and the final matching result is also false. The number of matching pairs of DKM is still relatively large, and there is part of false matching, and the final alignment effect is slightly poor. The number of matching points of EfficientLoFTR and OmniGlue is also relatively large, but most of them are false, and the final matching result is false. Lightglue is affected by the limited training dataset, and no effective matching pair is generated.

[0105] Table 3

[0106]

[0107] The method of the present application is significantly better than existing methods in matching point quantity, accuracy (AUC), and alignment effect (reprojection error). It performs outstandingly in weak texture and rotation difference scenarios. Through the confidence index prediction mechanism, the average inference time is reduced by 25%, and the precision loss is negligible. It provides an efficient and robust cross-modal matching solution for real-time positioning and navigation of unmanned aerial vehicles.

[0108] Table 4 is a matching result table of each method on the heterogenous remote sensing image dataset with scale difference. The test data in Table 4 is 500 pairs of heterogenous remote sensing images with significant scale difference (unmanned aerial vehicle flight height 50-9000 meters, aerial photograph and satellite image area ratio difference). The method of the present application (OURs) is slightly lower than DKM in matching accuracy (AUC@30px = 0.67), but the inference time is extremely short (0.139 seconds), and the reprojection error is close to DKM (H error = 47.55), achieving a good balance between efficiency and accuracy.

[0109] Figure 11 The schematic diagram of the matching point distribution and alignment effect of each method provided by the embodiments of the present application on the scale difference image. SuperGlue: matching points are concentrated but all are wrong, and the alignment effect is poor. LoFTR: matching points are scattered and wrong, and are obviously affected by position coding. EfficientLoFTR / OmniGlue: matching points are scattered, and part of the road features are matched correctly, but the overall alignment has deviation. DKM: the number of matching points is the most, but contains part of the wrong matching, and the alignment effect is better. The method of the present application (OURs): the number of matching points is large and the consistency is high, all matching pairs are correct, and the alignment effect is the best.

[0110] Table 4

[0111]

[0112] Table 5 is a table of strategy comparison experimental results. The arrows in the table indicate that the higher the corresponding indicator, the better. The arrows in the table indicate that the lower the corresponding indicator, the better. Using linear attention instead of the aggregated attention mechanism used in this application results in a slight decrease in efficiency and a significant decrease in accuracy. Removing layered matching will reduce model matching accuracy and slightly improve efficiency, especially the significant decrease in AUC@5px, which shows the importance of layered matching for high-precision matching. After replacing the reparameterized RepVGG with a multi-branch ResNet, the accuracy is basically the same, but the time consumption increases by 61%. After removing the rotation equivariant regularization, the model efficiency does not change significantly, but the accuracy decreases. After replacing the rotation position encoding with the absolute position encoding, the accuracy decreases and the efficiency increases slightly by 12%. After removing the early stopping strategy, the matching accuracy does not change significantly, but the time consumption increases by 25%. Therefore, aggregated attention and layered matching are the key to improving model accuracy, especially in high-precision matching scenarios. The design of reparameterized RepVGG and rotation position encoding significantly improves model efficiency while maintaining high accuracy. The early stopping strategy, which ends the reasoning early while ensuring accuracy, significantly optimizes the reasoning efficiency and is suitable for real-time matching tasks of drones.

[0113] Table 5

[0114]

[0115] The embodiment of the present application also provides a hierarchical heterogeneous remote sensing image matching device 1200 based on aggregated attention, such as Figure 12 As shown, the device includes: a conversion module 1201, an action module 1202, a hierarchical matching module 1203 and an evaluation module 1204.

[0116] The conversion module 1201 is used to convert the multi-branch structure of the lightweight convolutional neural network in the training phase into a single-branch network for efficient inference through reparameterization technology, and use the single-branch network to extract feature descriptors of the images to be matched. The images to be matched are drone aerial images and satellite images.

[0117] The action module 1202 is used to obtain an enhanced feature descriptor by alternately acting on the feature descriptor of the image to be matched through a self-attention mechanism and a cross-attention mechanism based on the extracted feature descriptor of the image to be matched.

[0118] The hierarchical matching module 1203 is used to perform hierarchical matching based on the enhanced feature descriptors, adopt a preliminary coarse matching strategy and a two-stage fine matching strategy, construct a score matrix and use a mutual nearest neighbor matching strategy to achieve sub-pixel accurate matching.

[0119] The evaluation module 1204 is used to quantitatively evaluate the confidence of each feature point based on the matching relationship between the feature points between the images to be matched by introducing a confidence index prediction mechanism.

[0120] Some of the modules in the apparatus described in the present application can be described in the general context of computer-executable instructions, such as program modules, that are executed by the computers. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0121] The apparatus or modules described in the above application embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. The methods, apparatuses or modules described in the present application can be implemented in computer-readable program codes in any appropriate manner.

[0122] As shown in Figure 13 The application further provides a hierarchical heterogeneous remote sensing image matching server based on aggregated attention, including a memory 1301 and a processor 1302; the memory 1301 is configured to store computer executable instructions; the processor 1302 is configured to execute the computer executable instructions to implement the hierarchical heterogeneous remote sensing image matching method based on aggregated attention described above in the embodiments of the present application.

[0123] The application further provides a computer readable storage medium, which stores executable instructions, and a computer executes the executable instructions to implement the hierarchical heterogeneous remote sensing image matching method based on aggregated attention described above in the embodiments of the present application.

[0124] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on such an understanding, the technical solutions of the present application can be embodied in the form of a software product, or can be embodied in the form of data migration in the implementation process. The computer software product can be stored in a storage medium, including a number of instructions to make a computer device execute the methods described in the embodiments of the present application.

[0125] The various embodiments in the specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. The whole or part of the present application can be used in many general or special computer system environments or configurations.

[0126] The above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can still be modified, or some or all of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. A hierarchical heterogeneous remote sensing image matching method based on aggregated attention, characterized in that: include: The multi-branch structure of the lightweight convolutional neural network in the training phase is converted into a single-branch network with efficient inference through reparameterization technology. The single-branch network is then used to extract feature descriptors of the images to be matched. The images to be matched are drone aerial images and satellite images. Based on the extracted feature descriptor of the image to be matched, the self-attention mechanism and the cross-attention mechanism are alternately applied to the feature descriptor of the image to be matched to obtain an enhanced feature descriptor; Based on the enhanced feature descriptor, a preliminary coarse matching strategy and a two-stage fine matching strategy are used for hierarchical matching. A score matrix is ​​constructed and a mutual nearest neighbor matching strategy is used to achieve sub-pixel accurate matching. By introducing a confidence index prediction mechanism, the confidence of each feature point is quantitatively evaluated based on the matching relationship of feature points between images to be matched.

2. The hierarchical heterogeneous remote sensing image matching method based on aggregated attention according to claim 1 is characterized in that: The feature descriptor of the image to be matched is extracted, and the self-attention mechanism and the cross-attention mechanism are alternately applied to the feature descriptor of the image to be matched to obtain an enhanced feature descriptor, including: The expression of the self-attention mechanism is: Attention(Q,K,V)=softmax(QK T )V; where Q is the query matrix generated by linear transformation of the feature descriptor of the image to be matched, K is the key matrix generated by linear transformation of the feature descriptor of the image to be matched, V is the value matrix generated by linear transformation of the feature descriptor of the image to be matched, softmax is the normalization function, Attention is the self-attention mechanism function, and T is the transpose operation; The expression of the cross attention mechanism is: Among them, Q A K is the query matrix generated by linear transformation of the feature descriptor of the UAV aerial image. B is the key matrix generated by linear transformation of the satellite image feature descriptor, V B The value matrix generated by linear transformation of the satellite image feature descriptor, Cross-Attention A Cross-attention mechanism function for drone aerial images; The self-attention mechanism and the cross-attention mechanism are stacked a preset number of times. In each stack, the output of the previous one is used as the input of the next one, forming a recursive feature enhancement process.

3. The hierarchical heterogeneous remote sensing image matching method based on aggregated attention according to claim 2, characterized in that: By alternating the self-attention mechanism and the cross-attention mechanism on the feature descriptor of the image to be matched, the enhanced feature descriptor is obtained, specifically including: Use a deep convolutional network to aggregate the feature maps of the drone aerial image to obtain the aggregated first feature map; Use the maximum pooling layer to aggregate the feature map of the satellite image to obtain the aggregated second feature map; Processing the feature descriptors of the first feature map and the second feature map according to the two-dimensional rotation position encoding and the self-attention mechanism and the cross-attention mechanism; The two-dimensional rotation position encoding expression is: Among them, a ij is the attention score between feature descriptors, q i is the query matrix generated by linear transformation of the i-th feature descriptor, k j is the key matrix generated by linear transformation of the j-th feature descriptor, R(x j -x i ,y j -y i ) is a block diagonal matrix, x i is the horizontal coordinate of the i-th feature descriptor on the first feature map, y i is the vertical coordinate of the i-th feature descriptor on the first feature map, x j is the horizontal coordinate of the jth feature descriptor on the second feature map, y j is the vertical coordinate of the j-th feature descriptor on the second feature map; Upsampling the first feature map and the second feature map after two-dimensional rotation position encoding and self-attention and cross-attention mechanisms to restore the size of the original feature maps; wherein the original feature maps are feature maps of the drone aerial image and feature maps of the satellite image; The upsampled feature maps are fused with the original feature maps to obtain the first final feature map and the second final feature map to obtain the enhanced feature descriptor.

4. The hierarchical heterogeneous remote sensing image matching method based on aggregated attention according to claim 3 is characterized in that: The preliminary rough matching strategy includes: The inner product of the first final feature map and the second final feature map is performed to construct a score matrix. The expression of the score matrix is: Among them, S(i,j) is an element in the score matrix, which represents the similarity score between the i-th feature descriptor in the first final feature map and the j-th feature descriptor in the second final feature map. is the position of the i-th feature descriptor in the first final feature map, is the position of the jth feature descriptor in the second final feature map, <·,·> is the inner product operation, θ c is the score threshold; Applying the softmax operator in two dimensions on the score matrix, we can get the probability matrix of the nearest neighbor matching. The expression of the probability matrix is: in, is an element of the probability matrix, representing the matching probability between the i-th feature descriptor in the first final feature map and the j-th feature descriptor in the second final feature map, softmax(S(i,·)) j is the jth element after applying the softmax function to the i-th row of the score matrix, softmax(S(·,j)) i is the i-th element after applying the softmax function to the j-th row of the score matrix; Under the condition of satisfying the mutual nearest neighbor matching constraint, the matching items with scores higher than the threshold are selected to establish a coarse match. The expression for establishing a coarse match is: in, To establish a rough matching set, is a matching pair, indicating the first feature descriptor and the first Matching between feature descriptors, is the set of mutual nearest neighbor matches, is an element in the probability matrix, θ c is the score threshold.

5. The hierarchical heterogeneous remote sensing image matching method based on aggregated attention according to claim 4 is characterized in that: The two-stage fine matching strategy includes: Perform the inner product on the corresponding fine features in the coarse matching set to obtain the local block score matrix: Among them, S′(i,j) is an element in the local block score matrix, which represents the similarity score between the i-th fine feature descriptor in the first final feature map and the j-th fine feature descriptor in the second final feature map. is the fine feature set corresponding to the first final feature map, is the fine feature set corresponding to the second final feature map; Perform mutual nearest neighbor matching search on the local block score matrix and select the matching pairs with the highest similarity score for intermediate pixel-level matching to limit the number of matches; The feature points of the first final feature map are associated with the feature blocks centered on the corresponding intermediate pixel-level matching points in the second final feature map, and the softmax operator is applied to process the associated feature data to obtain a matching distribution matrix. The final fine matching result is obtained by calculating the expected value of the matching distribution matrix.

6. The hierarchical heterogeneous remote sensing image matching method based on aggregated attention according to claim 5, characterized in that: The confidence index prediction mechanism is introduced to quantitatively evaluate the confidence of each feature point based on the matching relationship between the feature points of the images to be matched, including: At the end of each layer of self-attention mechanism and cross-attention mechanism, the confidence index is calculated for each feature point. The expression of the confidence index is: i =Sigmoid(MLP(x i ))∈[0,1]; where c i is the confidence index, Sigmoid is the Sigmoid function, MLP is the multi-layer perceptron, x i is the eigenvector corresponding to the feature point; For a given layer, the decay confidence threshold is determined based on the verification accuracy of each classifier. The expression of the decay confidence threshold is: l =0.8+0.1e -4l / L ; Among them, λ l is the decay confidence threshold, L is the total number of layers in the network, and l is the current number of layers; If the confidence index of the feature point corresponding to the image to be matched is greater than the decay confidence threshold of the current layer, the confidence of the feature point is high; Count the ratio of feature points with high confidence among all feature points. If the ratio of feature points with high confidence in the current layer is greater than the preset value, the inference is terminated at the current layer. The judgment formula for ending reasoning is: Among them, exit is a Boolean variable used to indicate whether to end the reasoning at the current layer, N feature is the total number of feature points, I∈{A,B} is the index variable, A is the drone aerial image, B is the satellite image, i∈Z is the index variable, and Z is the index set of feature points. It is an indicator function. When the condition in the brackets is met, the function value is 1, otherwise it is 0. is the confidence index of the feature point corresponding to the image to be matched, and α is a preset value.

7. A hierarchical heterogeneous remote sensing image matching device based on aggregated attention, characterized in that: include: A conversion module is used to convert the multi-branch structure of the lightweight convolutional neural network in the training phase into a single-branch network for efficient inference through reparameterization technology, and use the single-branch network to extract feature descriptors of the images to be matched; the images to be matched are drone aerial images and satellite images; An action module is used to alternately act on the feature descriptor of the image to be matched based on the extracted feature descriptor of the image to be matched through a self-attention mechanism and a cross-attention mechanism to obtain an enhanced feature descriptor; The hierarchical matching module is used to perform hierarchical matching based on the enhanced feature descriptors, using a preliminary coarse matching strategy and a two-stage fine matching strategy. It constructs a score matrix and uses a mutual nearest neighbor matching strategy to achieve sub-pixel accurate matching. The evaluation module is used to quantitatively evaluate the confidence of each feature point based on the matching relationship between feature points between the images to be matched by introducing a confidence index prediction mechanism.

8. A hierarchical heterogeneous remote sensing image matching server based on aggregated attention, characterized in that: including memory and processor; The memory is used to store computer-executable instructions; The processor is configured to execute the computer-executable instructions to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores executable instructions, and when a computer executes the executable instructions, the method according to any one of claims 1 to 6 can be implemented.

Images