A deep feature fusion matching method for multi-source remote sensing images

Abstract

The present invention belongs to the field of image feature matching, and specifically relates to a multi-source remote sensing image deep feature fusion matching method. The method obtains fine fusion features with both high-level and low-level information through feature extraction, and performs feature transformation on the high-level features to improve the similarity between matching points in the high-level features of the image to be matched, so that the matching results are more reliable. In the later feature matching, the high-level features are first roughly matched to obtain the global matching results, and then the fine fusion features are used for correction, so that the matching results take into account the reliability of both global aspects and precision, with higher resolution and more accurate positioning. Before feature transformation, the high-level features are sinusoidally encoded so that there is a unique correspondence between the features, avoiding mismatches caused by excessive similarity between features in sparse texture areas; when performing coarse matching, the score threshold is adaptively reduced through the sliding window, and the number of matching points screened out in the sparse area is increased, further improving the matching effect of the sparse area.

Description

Technical Field

[0001] The present invention belongs to the field of image feature matching, and in particular relates to a multi-source remote sensing image depth feature fusion matching method. Background Art

[0002] Multi-source remote sensing images of the same area usually contain different characteristics of the same ground feature information. Therefore, combining multi-source remote sensing images can fully display the ground feature information. The premise of jointly processing multi-source remote sensing images to discover ground feature remote sensing information is to match multi-source images and then identify the same-name points in two or more multi-source images.

[0003] Current multi-source remote sensing image matching methods typically apply convolutional neural networks to match multiple influences, such as the CMM-Net and LoFTR algorithms. The CMM-Net algorithm uses convolutional neural networks to extract high-dimensional feature maps from multi-source images. It selects feature points based on the principle of simultaneously satisfying both channel maximization and local maxima to ultimately complete multi-source image matching. While this algorithm is relatively adaptable to multi-source image matching, it can encounter inconsistent similarities between the feature vectors of each image, making it difficult to guarantee matching accuracy. The LoFTR algorithm uses the Transformer to enhance feature relevance, combining coarse matching with fine matching to achieve dense matching results. While it excels in matching sparsely textured areas, its feature extraction network is limited by the scale of a single image and cannot meet multi-faceted matching requirements.

[0004] As can be seen from the above, existing multi-source image feature matching methods still have various drawbacks, such as difficulty in feature representation, large differences in feature vector similarity, or a small number of features in sparse textured areas of multi-source remote sensing images, which leads to matching difficulties. These shortcomings affect the accuracy of the final matching results. Therefore, it is necessary to propose a remote sensing feature matching solution that can solve these problems. Summary of the Invention

[0005] The purpose of the present invention is to provide a multi-source remote sensing image deep feature fusion matching method to solve the problems existing in the existing multi-source image feature matching methods, such as difficulty in feature representation, large differences in feature vector similarity, or small number of features in sparse texture areas of multi-source remote sensing images leading to matching difficulties, which affect the accuracy of the final matching results.

[0006] In order to achieve the above object, the present invention provides a multi-source remote sensing image deep feature fusion matching method, the steps are as follows:

[0007] 1) Constructing a matching model, wherein the matching model includes a feature extraction network, a feature transformation module, a dense matching module, and a calibration optimization module;

[0008] 2) The acquired remote sensing image pairs to be matched are input into the feature extraction network, which extracts features from each image to be matched, and obtains high-level features of each image to be matched, as well as fine fusion features that have both high-level fine positioning information and low-level global information;

[0009] 3) The obtained high-level features of the image pairs to be matched are simultaneously input into the feature transformation module. By fusing the neighborhood information of the own image and the high-level features of the image to be matched, feature transformation and fusion are performed on each image to be matched, and high-level fusion features with correlation are obtained for each image to be matched in the image pair to be matched;

[0010] 4) Densely match all feature vectors on the high-level fusion features of the image pair to be matched, and obtain a rough matching result based on the similarity between the feature vectors;

[0011] 5) Map the coarse matching results to the fine fusion features to perform calibration and optimization on the dense matching to obtain the fine matching results.

[0012] The beneficial effects of this method are as follows: in the early stage of feature extraction, through feature extraction and fusion, the obtained image fine fusion features contain both high-level information and low-level information, which can ensure both positioning accuracy and globality and anti-interference capabilities; and the extracted high-level features are transformed and fused to obtain high-level fusion features to improve the similarity between matching points in the high-level features of the image to be matched, making the matching results more reliable. In the later stage of feature matching, the high-level fusion features are first compared for coarse matching to obtain a matching result that is more consistent with the global features, and then the matching result is corrected by comparing the fine fusion features, so that the matching result takes into account the reliability of both global aspects and precision, with higher resolution and more accurate positioning.

[0013] Furthermore, the feature extraction network includes three downsampling layers and two upsampling layers;

[0014] The first downsampling layer is used to downsample the input image to be matched to obtain a high-level feature map of 1 / 2 of the original dimension of the image to be matched; the second downsampling layer is used to downsample the high-level feature map of 1 / 2 of the original dimension of the image to be matched to obtain a high-level feature map of 1 / 4 of the original dimension of the image to be matched; the third downsampling layer is used to downsample the high-level feature map of 1 / 4 of the original dimension of the image to be matched to obtain a high-level feature map of 1 / 8 of the original dimension of the image to be matched;

[0015] The first upsampling layer is used to upsample the high-level feature map of the original dimension 1 / 8 of the input image to be matched into a low-level feature map of the original dimension 1 / 4 of the image to be matched, and at the same time, perform convolution processing on the high-level feature map of the original dimension 1 / 4, and then fuse the low-level feature map of the original dimension 1 / 4 and the high-level feature map of the original dimension 1 / 4 to obtain a fused feature map of the original dimension 1 / 4; the second upsampling layer is used to upsample the fused feature map of the original dimension 1 / 4 of the input image to a low-level feature map of the original dimension 1 / 2, and at the same time, perform convolution processing on the high-level feature map of the original dimension 1 / 2, and then fuse the low-level feature map of the original dimension 1 / 2 and the high-level feature map of the original dimension 1 / 2 to obtain a fine fused feature map of the original dimension 1 / 2;

[0016] Through the three downsampling layers and two upsampling layers, a high-level feature map with a dimension of 1 / 8 of the original image size and a fine fused feature map with a dimension of 1 / 2 of the original image size are output for fusion feature matching. To improve the information globality, positioning accuracy, and anti-interference ability of multi-source remote sensing image features, during feature extraction, high-level features are fused with low-level features twice, so that the final fine fused feature map includes information from features at all layers. The richer the feature information included, the higher the feature globality and positioning accuracy.

[0017] Furthermore, the fusion feature matching specific steps of the feature transformation module are as follows:

[0018] 1) Add position information to the high-level feature map of the original dimension of the image to be matched, so that the features uniquely correspond to their positions on the original image;

[0019] 2) Flatten the high-level feature maps of the images to be matched with position information into one-dimensional vectors, input them into the feature transformation module, and after multiple interleaving processes through the attention layer, output the high-level fusion features with correlation for each image to be matched;

[0020] The attention layer includes a self-attention layer, a cross-attention layer and an attention layer; the self-attention layer is used to fuse the features with position information of the input reference image in the image pair to be matched with its own local neighborhood information to generate a new feature map; the cross-attention layer fuses the input reference image features with position information with the features of the other image to be matched in the image pair to be matched; the attention layer selects relevant information by measuring the similarity between the query vector and each key feature, normalizes the selection result, and superimposes it with the one-dimensional vector flattened by the high-level feature map with position information of each image to be matched to obtain a fusion feature of the fused position information, its own neighborhood information and the image information to be matched; the multiple interleaving processes refer to inputting the obtained fusion features into the attention layer for re-interleaving, repeating the above process, and finally outputting the high-level fusion features of each image to be matched.

[0021] Since the correlation between features with the same name in multi-source remote sensing images is often low, the similarity between matching points is low during matching, which makes it difficult to accurately identify matching points. Therefore, by improving the correlation of high-level features of the images to be matched through feature transformation, the accuracy of matching point identification can be improved, thereby obtaining more reliable matching results.

[0022] Furthermore, in order to reduce parameters and computational complexity while ensuring downsampling accuracy, the three downsampling layers all adopt a convolutional block structure, superimposing the input feature map and the output feature map element by element, and the feature map obtained after superposition is used as the downsampling result.

[0023] The convolutional block structure described above allows for a simple identity mapping during the forward propagation of a neural network, stacking the results of a single convolution with those of three convolutions to produce a downsampled result. This identity connection approach adds neither additional parameters nor computational complexity, while simultaneously improving training efficiency and ensuring sampling accuracy. Furthermore, this structure does not affect the backward propagation of training.

[0024] Furthermore, in order to improve the matching effect of sparse texture areas of multi-source images, the adding of position information is to add sinusoidal coding to each pixel feature.

[0025] Since sinusoidal coding can provide unique position information for each pixel, the feature corresponds uniquely to its position on the original image, and there is a unique and definite correspondence between features at different levels. This helps to avoid the mismatch problem caused by excessive similarity between feature vectors in sparse texture areas of the image, thereby improving the matching effect of sparse texture areas.

[0026] Furthermore, the coarse matching represents the similarity between all feature vectors by means of a score matrix between high-level fusion feature vectors. If the similarity is greater than a score threshold, it is considered a correct match.

[0027] The score matrix S between the vectors is determined by the following formula, where <,> represents the inner product;

[0028]

[0029] Among them, F A_tr 、F B_tr is the high-level fusion feature of the images A and B to be matched, and A×B is all possible correspondences between the pixels in the images A and B to be matched;

[0030] Calculate the score matrix for all possible matching methods By maximizing the total score ∑ i,j S i,j P i,jTo obtain the optimal allocation matrix P; the optimal allocation matrix P can represent the optimal correspondence between the high-level fusion feature vectors in the image A to be matched and the high-level fusion feature vectors in the image B to be matched.

[0031] Furthermore, in order to accurately increase the coarse matching results in sparse texture areas, the coarse matching through the score matrix adopts a sliding window adaptive score threshold detection algorithm, which is as follows:

[0032] 1) Assuming the initial score threshold as θ, setting the area of the sliding window, the horizontal sliding step size, and the vertical sliding step size, a sliding detection is performed on the score of the high-level fusion feature vector;

[0033] 2) If the scores s of all high-level fused feature vectors in the current window are less than θ, the adaptive threshold avgθ in the matching sparse node of the window is calculated; the high-level fused feature vectors in the window are traversed, and if there is a vector in the current window with a score s> avgθ, the vector is added to the coarse matching point set and the window continues to slide;

[0034] 3) If the score of the feature vector in the current window is s≥θ, the window continues to slide;

[0035] 4) Repeat the above steps until the window slides through the high-level fusion features of the image to be matched and outputs a coarse matching point set;

[0036] The area, horizontal sliding step length, and vertical sliding step length of the sliding window are calculated as follows:

[0037]

[0038] ws is the area of the sliding window, hl is the horizontal sliding step size, and vl is the vertical sliding step size;

[0039] The calculation formula of the adaptive threshold avgθ in the matching sparse node is as follows

[0040]

[0041] Where n is the number of feature vectors in the sliding window, s i is the matching score of each feature vector in the sliding window.

[0042] Since there are fewer matching point pairs in sparse texture areas or single texture areas during multi-source remote sensing image matching, the number of high-scoring matching point pairs may be lower than that in dense matching areas. Therefore, by adaptively lowering the score threshold for matching sparse areas, we can screen and increase the number of low-scoring matching point pairs in matching sparse areas to supplement the matching data, while not selecting redundant low-scoring point pairs in dense matching areas to avoid introducing errors.

[0043] Furthermore, in order to improve the matching accuracy, the process of mapping the coarse matching results to the low-level features to perform calibration and optimization on the dense matching is as follows:

[0044] 1) Centered on N pairs of feature points on the correlated high-level fusion features of each image to be matched, the N pairs of feature points refer to the coarse matching point set screened after coarse matching; crop N pairs of local windows of size m×m on the corresponding high-level fusion features;

[0045] 2) Mapping the N pairs of windows to the fine fusion features of the image to be matched, obtaining N pairs of local fine windows centered on the coarse matching feature point pairs, inputting the N pairs of local fine windows into the feature transformation module, performing several transformations on them, and generating N pairs of local fine fusion feature maps of images A and B centered on the coarse matching feature point pairs.

[0046] 3) Each The eigenvector corresponding to the center point P and the corresponding All vectors in are associated to generate The expected value of the matching probability distribution of each pixel with P; the expected value of the matching probability distribution is calculated as follows

[0047]

[0048] Where V A (P) is The eigenvector of the center point P, V B (x) is The feature vector of a certain pixel point x, y is the pixel gradient of pixel point x on image B; the pixel point with the highest calculated probability value is the fine matching result with sub-pixel accuracy of point P on image A on image B, and this result is used as the final matching result.

[0049] Since the coarse matching result is obtained under high-level fusion feature matching, and the high-level fusion feature description may have matching errors for multi-source remote sensing images with large differences, the coarse matching result is re-positioned to the fine fusion feature and the most matching fine feature point is selected to obtain a precise matching result of multi-source remote sensing images with higher resolution and more accurate positioning.

[0050] Furthermore, to ensure the accuracy of the matching results, after obtaining the fine matching results, the PROSAC algorithm is used to check and eliminate false matches again. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a schematic diagram of a feature extraction network according to Example 1 of the method of the present invention;

[0052] Figure 2 Schematic diagram of the convolution block structure of Example 1 of the method of the present invention;

[0053] Figure 3 This is a schematic diagram of the characteristic connection structure of Example 1 of the method of the present invention;

[0054] Figure 4a This is the H / 8×W / 8×256 high-level feature map visualization of Example 1 of the method of the present invention. Figure 4b This is the visualization of the H / 2×W / 2×128 fine fusion feature map of Example 1 of the method of the present invention;

[0055] Figure 5 This is a flow chart of feature transformation processing in Example 1 of the method of the present invention;

[0056] Figure 6a This is the optical image of the drone of the first set of image pairs of the comparative example of the present invention. Figure 6b This is the UAV thermal infrared image of the first set of image pairs of the comparative example of the present invention. Figure 6c This is the ZY-3 full-color image of the second image pair of the comparative example of the present invention. Figure 6d This is the GF-3SAR image of the second set of image pairs of the comparative example of the present invention, Figure 6e This is the summer Google image of the third set of image pairs in the comparative example of the present invention. Figure 6f This is the winter Google image of the third set of image pairs in the comparative example of the present invention. Figure 6g This is the Google optical image of the fourth image pair of the comparative example of the present invention. Figure 6h This is the ZY-3 full-color image of the fourth image pair of the comparative example of the present invention. Figure 6i This is the Google optical image of the fifth image pair of the comparative example of the present invention. Figure 6j This is the GF-2 full-color image of the fifth image pair of the comparative example of the present invention. Figure 6k This is the Google optical image of the sixth image pair of the comparative example of the present invention. Figure 6l The OSM raster map image of the sixth image pair of the comparative example of the present invention;

[0057] Figure 7 A five-direction division diagram of a comparative example of the present invention;

[0058] Figure 8a This is a schematic diagram of the fine matching results of the first set of image pairs in the comparative example of the present invention. Figure 8b This is a schematic diagram of the fine matching results of the second set of image pairs in the comparative example of the present invention. Figure 8c This is a schematic diagram of the fine matching results of the third group of image pairs in the comparative example of the present invention. Figure 8d Schematic diagram of the fine matching results of the fourth group of image pairs in the comparative example of the present invention, Figure 8e This is a schematic diagram of the fine matching results of the fifth set of image pairs in the comparative example of the present invention. Figure 8fSchematic diagram of the fine matching results of the sixth group of image pairs in the comparative example of the present invention;

[0059] Figure 9a This is a schematic diagram of the purification results of the first group of image pairs in the comparative example of the present invention. Figure 9b This is a schematic diagram of the purified matching results of the second group of image pairs in the comparative example of the present invention. Figure 9c This is a schematic diagram of the purification results of the third group of image pairs in the comparative example of the present invention. Figure 9d This is a schematic diagram of the purification results of the fourth group of image pairs in the comparative example of the present invention. Figure 9e This is a schematic diagram of the purification results of the fifth group of image pairs in the comparative example of the present invention. Figure 9f Schematic diagram of the purification results of the sixth group of image pairs in the comparative example of the present invention;

[0060] Figure 10a The LoFTR algorithm of the comparative example of the present invention is purified by the PROSAC algorithm on the first, third and fourth groups of multi-source remote sensing image pairs. Figure 10b The matching results of the SuperPoint algorithm of the comparative example of the present invention after being purified by the PROSAC algorithm on the first, third and fourth groups of multi-source remote sensing image pairs are shown. Figure 10c The matching results of the SIFT algorithm of the comparative example of the present invention after being purified by the PROSAC algorithm on the first, third and fourth groups of multi-source remote sensing image pairs are shown. Figure 10d The matching results of the ContextDesc algorithm of the comparative example of the present invention after being purified by the PROSAC algorithm on the first, third and fourth groups of multi-source remote sensing image pairs are shown. Figure 10e The matching results of the FFM algorithm of the comparative example of the present invention after being purified by the PROSAC algorithm on the first, third and fourth groups of multi-source remote sensing image pairs;

[0061] Figure 11a This is the final registration result and the local windowed magnified image of the first set of image pairs in the comparative example of the present invention. Figure 11b This is the final registration result and the partially windowed magnified image of the second set of image pairs in the comparative example of the present invention. Figure 11c This is the final registration result and the local window enlargement image of the third group of image pairs in the comparative example of the present invention. Figure 11d This is the final registration result and the local windowed magnified image of the fourth image pair of the comparative example of the present invention. Figure 11e This is the final registration result and the partially windowed magnified image of the fifth image pair of the comparative example of the present invention. Figure 11f This is the final registration result and the local windowed magnified image of the sixth image pair of the comparative example of the present invention. DETAILED DESCRIPTION

[0062] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0063] Method Example

[0064] This embodiment provides a multi-source remote sensing image deep feature fusion matching method, the specific steps are as follows:

[0065] 1) Construct a matching model, which includes a feature extraction network, a feature transformation module, a dense matching module, and a calibration optimization module;

[0066] 2) The acquired remote sensing image pairs to be matched are input into the feature extraction network, which extracts features from each image to be matched separately to obtain high-level features of each image to be matched and fine fusion features that have both high-level fine positioning information and low-level global information.

[0067] Reference Figure 1 , the structure of the feature extraction network is as follows:

[0068] ① Image input, convert the input original image H×W into a feature image of H×W×1;

[0069] ②, ③, and ④ are the first, second, and third downsampling layers from low to high layers. ② extracts 128-dimensional high-level features (H / 2×W / 2×128) with half the original image dimension, ③ further extracts 196-dimensional high-level features (H / 4×W / 4×196) with a quarter of the original image dimension, and ④ further extracts 256-dimensional high-level features (H / 8×W / 8×256) with 1 / 8 of the original image dimension.

[0070] The downsampling of ②, ③, and ④ is a convolution operation on the original image. In order to reduce parameters and computational complexity, ②, ③, and ④ all use a convolution block structure. This structure can simply perform an identity mapping during the forward propagation of the neural network, and stack the result of one convolution with the result of three convolutions to obtain the downsampling result. This identity connection method neither adds additional parameters nor increases computational complexity, while improving training efficiency and ensuring sampling accuracy. In addition, this structure will not affect the reverse propagation of the training process. Figure 2 As shown in the figure, the input feature map f1 and the output feature map f2 are element-wise superimposed. This simple connection can improve the training effect of the model without adding additional parameters and computation to the network.

[0071] ⑤ and ⑧ are the bilinear interpolation parts from high layer to low layer, and ⑥ and ⑨ are the feature connection parts. Among them, ⑤ extracts 196-dimensional low-level fine features (H / 4×W / 4×196) of 1 / 4 dimension of the original image. ⑥ connects and fuses the 196-dimensional high-level features of 1 / 4 dimension of the original image extracted in ③ with the 196-dimensional low-level fine features of 1 / 4 dimension of the original image obtained in ⑤ to form ⑦, which is the first upsampling layer that fuses the features containing the bottom-level positioning information with the upsampling results of the high-level features rich in semantic information, and obtains the 196-dimensional fused features of 1 / 4 dimension of the original image; then ⑧ extracts 128-dimensional fused features of 1 / 2 dimension of the original image, and ⑨ connects and fuses the 128-dimensional high-level features of 1 / 2 dimension of the original image extracted in ② with the 128-dimensional fused features of 1 / 2 dimension of the original image obtained in ⑧ to form ⑩, which is the second upsampling layer that fuses the features containing the bottom-level positioning information with the upsampling results of the fused features after the first fusion, and obtains the 128-dimensional fine fused features of 1 / 2 dimension of the original image.

[0072] The above feature extraction network structure ultimately outputs a high-level feature map with a dimension of 1 / 8 the original size and a refined fused feature map with a dimension of 1 / 2 the original size for fusion feature matching. The final feature extraction result is achieved through downsampling of feature extraction, refined upsampling, and fusion of features from two different layers, which significantly improves the globality of feature information and positioning accuracy.

[0073] The characteristic connection structure of ⑥ and ⑨ is as follows Figure 3 As shown in Figure 6. Taking step 6 as an example, this part upsamples the semantically richer high-level features F3 (H / 8×W / 8×256) to obtain F4 (H / 4×W / 4×196). The previous layer features F1 (H / 4×W / 4×196) of F3 are convolved with a 1×1 convolution kernel to obtain F2 (H / 4×W / 4×196). The F4 features are then concatenated with F2 to obtain the fused features F5 (H / 4×W / 4×196) with high positioning accuracy and global information. This concatenation method adaptively adjusts the feature scale, allowing F2 and F4 to adapt to each other's scale differences, thereby breaking away from dependence on a single image scale. Furthermore, this operation can fuse the positioning details of low-level features with the rich semantic information of high-level features. This fused feature can significantly enhance the representation capabilities of multi-source remote sensing imagery features, counteracting geometric and scale differences between them.

[0074] In this embodiment, the images A and B to be matched are input into the above-mentioned feature extraction network. After feature extraction and fusion, a 256-dimensional high-level feature F with a dimension of 1 / 8 of the original image is finally extracted. A_C 、F B_C And the 128-dimensional fine fusion feature F with 1 / 2 of the original image dimension A_F 、FB_F ,The feature map visualization is shown in Figure 4, where, Figure 4a For H / 8×W / 8×256 high-level feature map visualization, Figure 4b Visualization of the H / 2×W / 2×128 fine fusion feature map.

[0075] 2) The high-level features of the images to be matched are input into the feature transformation module to perform feature transformation fusion to obtain the relevant high-level fusion features of each image to be matched.

[0076] Since the feature differences between multi-source remote sensing images are extremely large, especially the differences between features with the same name are too large, which will lead to low similarity between features with the same name, seriously affecting the subsequent matching effect and making it difficult to meet the requirements of precise matching. Therefore, the feature transformation module of this embodiment processes the 256-dimensional high-level features of 1 / 8 of the original image dimension obtained in step 1) through the attention layer in the local feature transformer to obtain relevant high-level fusion features.

[0077] Before feature transformation, the 256-dimensional high-level feature F of the original image dimension 1 / 8 obtained in step 1) is transformed. A_C 、F B_C Sine coding is added. Sine coding provides unique position information for each pixel, so that the features can uniquely correspond to their positions on the original image, and there is a unique and definite correspondence between features at different levels, so that high-level features can be subsequently mapped to low-level fine-fusion features. This helps to avoid the mismatch problem caused by excessive similarity between feature vectors in sparse texture areas of the image, and improve the matching effect of sparse texture areas.

[0078] Then, the high-level feature F with added position information A_C 、F B_C Flattened into one-dimensional vectors L1 and L2, both of length H / 8×W / 8. L1 and L2 are input to the feature transformation module; the feature transformation module improves the similarity between the high-level features of the two images by interleaving the attention layer. The processing flow is as follows: Figure 5 shown.

[0079] Reference Figure 5, the attention layer input vector can be compared to the query vector Query, key vector Key and value vector Value in the dictionary search. Its purpose is to transform the features and then fuse the neighborhood information of the own image with the feature information of the image to be matched. For the self-attention layer (Self Layer), Query=Key=Value=L1, which is equivalent to inputting features with position information to fuse its own local neighborhood information and generate a new feature map; for the cross-attention layer (Cross Layer), Query=L1、Key=Value=L2, which is to fuse the input reference image features with position information with the features of the image to be matched. The attention weight W between features is calculated by the dot product of Query and Value, and information is retrieved from Value; in this embodiment, elu(Q)+1, elu(K T )+1 instead of Query, Key, to achieve the purpose of reducing the computational cost. In addition, the associative law of the tensor matrix is used to calculate (elu(K T )+1)·V to simplify the calculation. The final formula is as follows:

[0080] W(Q,K,V)=(elu(Q)+1)·[(elu(K T )+1)·V]

[0081] The attention layer selects relevant information by measuring the similarity between the query vector and each key feature. The output vector is the sum of the value vectors weighted by the similarity score. Relevant information can be extracted from the value vectors with high similarity. In the attention layer, the result of the attention layer (Line Attention) is normalized and superimposed with L1 and L2 with position information to obtain a one-dimensional feature F that integrates the position information, the neighborhood information itself, and the image information to be matched. A_tr 、F B_tr Since there will be information loss during the normalization process of the attention layer, the fusion feature F A_tr 、F B_tr Return to L1 and L2 for further processing. Due to the large differences in texture and grayscale between multi-source remote sensing images, it is necessary to make the two high-level features have a stronger correlation. Therefore, in this embodiment, by increasing the number of interleaving processes, a more reliable and more correlated fusion feature is obtained; wherein, the value of N can be set according to different matching accuracy requirements. In a preferred embodiment, the number of interleaving times N = 8 can achieve the highest subsequent matching accuracy. After the above processing, the high-level fusion feature F that fuses the neighborhood information itself and the image information to be matched is output. A_tr 、F B_trAfter the feature transformation, the high-level fusion features of images A and B are more correlated with each other and the correlation between non-homologous features is weaker. Therefore, the similarity of homologous features is higher than that of other features, thereby reducing mismatching.

[0082] 3) Densely match all feature vectors on the high-level fusion features and obtain coarse matching results based on the similarity between feature vectors;

[0083] In step 2), the two high-level fused features have a strong correlation after feature transformation and fusion. A pixel-level dense matching is established on the high-level fused features to obtain a rough dense matching result. In this embodiment, the best matching layer is used to perform the initial matching between multi-source remote sensing images, i.e., coarse matching. The focus of coarse matching is to compare the similarity between features.

[0084] Since the output vector of the feature transformation fusion is the sum of value vectors weighted by similarity scores, which carries similarity information, the similarity between all feature vectors can be represented by the score matrix between vectors. If the similarity is greater than the score threshold, it means that the similarity between the two vectors is high and can be regarded as a correct match.

[0085] In this embodiment, the high-level fusion feature F A_tr 、F B_tr The score matrix S between is determined by the following formula:

[0086]

[0087] Where <,> represents the inner product; A×B represents all possible correspondences between pixels in the images A and B to be matched. For example, if the pixels in A are 640×480 and the pixels in B are 500×300, then there are a total of 640×480×500×300 possible correspondences between the pixels in A and the pixels in B. i represents one of the 640×480 pixels in the image A to be matched, and j represents one of the 500×300 pixels in the image B to be matched.

[0088] Calculate the score matrix for all possible matches (i.e. all possible correspondences between pixels in the images A and B to be matched) By maximizing the total score ∑ i,j S i,j P i,j To obtain the optimal allocation matrix P. The optimal allocation matrix P can represent the optimal correspondence between the high-level fusion feature vectors in the image A to be matched and the high-level fusion feature vectors in the image B to be matched.

[0089] In this embodiment, the optimal allocation matrix P can be calculated using the entropy regularization formula of the optimal transmission algorithm. The optimal transmission can be effectively solved for P using the Sinkhorn algorithm. Finally, the mutual nearest neighbor (MNN) criterion is implemented to filter out potentially abnormal matches. Combining these two selection criteria, a relatively reliable and evenly distributed matching result is obtained.

[0090] Since there are fewer matching point pairs in sparse texture areas or single texture areas during multi-source remote sensing image matching, it is impossible to select more high-scoring matching point pairs. This results in insufficient features for sparse texture areas or single texture areas to participate in matching, making it difficult to achieve ideal matching results. Therefore, a sliding window adaptive score threshold detection algorithm is established to improve the coarse matching results in sparse texture areas, as follows:

[0091] First, the initial score threshold is set to θ, the area of the sliding window, the horizontal sliding step size, and the vertical sliding step size are set, and the scores of the correlated high-level fusion feature vectors are subjected to sliding detection; if the score s ≥ θ exists in the feature vector in the current window, the window continues to slide.

[0092] When the scores s<θ of all high-level fused feature vectors in the current window, the adaptive threshold avgθ within the matching sparse node in the window is calculated; the high-level fused feature vectors in the window are traversed, and if there is a vector with a score s>avgθ in the current window, the vector is added to the coarse matching point set and the window continues to slide.

[0093] Repeat the above operation until the window slides through the high-level fusion features F A_tr 、F B_tr , output high-level fusion feature F A_tr 、F B_tr rough matching point set.

[0094] The calculation formulas for the sliding window area, horizontal sliding step size, and vertical sliding step size are as follows:

[0095]

[0096]

[0097] Where ws is the area of the sliding window, hl is the horizontal sliding step size, and vl is the vertical sliding step size;

[0098] The calculation formula for the adaptive threshold avgθ within the matching sparse node is as follows

[0099]

[0100] Where n is the number of feature vectors in the sliding window, s i is the matching score of each feature vector in the sliding window.

[0101] Since there are fewer matching point pairs in sparse texture areas or single texture areas during multi-source remote sensing image matching, the number of high-scoring matching point pairs may be lower than that in dense matching areas. Therefore, by adaptively lowering the score threshold for matching sparse areas, we can screen and increase the number of low-scoring matching point pairs in the matching sparse areas and supplement the matching data. At the same time, since the score threshold is adaptively lowered according to the feature sparsity of the area, redundant low-scoring point pairs in the dense matching areas will not be selected, thus avoiding the introduction of errors.

[0102] This embodiment also uses outdoor scenes in the MegaDepth dataset as a training set to train the rough matching results. MegaDepth is a large-scale depth dataset that can be used for monocular depth estimation, generated from massive Internet images. It contains about 100,000 outdoor 3D scenes. These 3D scenes can generate stereo pairs with strict transformation relationships and their camera parameters. The image points in the stereo pairs have a one-to-one matching pixel relationship. Through this correspondence, the real matching of the actual scene in the training set is calculated. As the true value, combined with the distribution matrix value that represents the rough matching result, the optimal matching result of the rough matching is obtained by minimizing the difference between the distribution matrix value and the true value, that is, the best match, and ultimately improving the stability and reliability of the rough matching. Based on the above principle, the loss function is shown as follows:

[0103]

[0104] Where, Calculate values for the allocation matrix, The purpose of this training is to make the coarse matching results approach the known true matching results. It is mainly used for training data with ground truth matching information, such as large differences in illumination, scale, and day and night image pairs, to learn their true matching relationships.

[0105] 4) Map the coarse matching results to the fine fusion features to perform calibration and optimization on the dense matching to obtain the fine matching results.

[0106] Since the coarse matching results are obtained at a resolution of 1 / 8 of the original image, their positions may be inaccurate when mapped to the original dimensions. This means that two high-level descriptors are extremely similar, but may not be the most similar. There may be errors of several pixels within the local window. For example, a rough match obtained at a high-level feature resolution may be a match between feature vectors extracted from an 8×8 pixel area, which cannot be accurately located at the pixel level. This high-level feature description may be inaccurate for multi-source remote sensing images with large differences. Therefore, the feature points obtained from the coarse matching in step 3) are positioned within the fine fusion features obtained in step 1) for calibration and optimization, achieving further fine-tuning of the match and thus obtaining a more precise matching result for multi-source remote sensing images with higher resolution.

[0107] Specifically, when performing fine matching, the N feature points in the coarse matching point set selected in step 3) are used as the center, and the high-level fusion feature F A_tr 、F B_tr Crop N pairs of local windows of size m×m; map the N pairs of local windows to the fine fusion features F of the images A and B to be matched A_F 、F B_F In the process, N pairs of local fine windows centered on the coarse matching feature point pairs are obtained; the N pairs of local fine windows are input into the feature transformation module, and are transformed several times to generate N pairs of local fine fusion feature maps of images A and B centered on the coarse matching feature point pairs. Among them, the value of m can be set specifically according to actual needs: if the time requirement is high, the value of m is set to about 5; if the final accuracy requirement of the matching result is high, it is set to about 8; if the hardware device is poor, it is set to about 3, with smaller running memory; in this embodiment, m is set to 5.

[0108] Then each The eigenvector corresponding to the center point P and the Pairs All vectors in the The expected value of the matching probability distribution of each pixel in P; the calculation formula for the expected value of the matching probability distribution is as follows:

[0109]

[0110] Where V A (P) is The eigenvector of the center point P, V B (x) is The feature vector of a pixel point x, y is the pixel gradient of pixel point x on image B; the highest probability value calculated is The pixel point is the fine matching result of point P on image A on image B with sub-pixel accuracy, and this result is taken as the final matching result.

[0111] In order to ensure the accuracy of the matching results, the precise matching results need to be checked and eliminated for mismatches again. This paper adopts the progressively consistent sampling algorithm, namely the PROSAC algorithm, to eliminate mismatches. The PROSAC algorithm samples from a continuously growing set of optimal matching point pairs. Although it is susceptible to the influence of too many mismatch points, which makes the algorithm extremely unstable, in the present invention, after the combined processing of coarse matching and fine matching calibration and optimization, the number of existing mismatch points is already extremely small. Therefore, the use of the PROSAC method to eliminate mismatch points has strong adaptability to the present invention.

[0112] Comparative Example

[0113] This comparative example tested the multi-source remote sensing image deep feature fusion matching method (hereinafter collectively referred to as the FFM algorithm) described in the method embodiment on the Ubuntu 18.04 operating system. The programming language was Python 3.6 and the programming platform was PyCharm. The hardware platform used was a laptop with an i7 CPU, 31GB of memory, and a GeForce RTX 2060 graphics card (6GB of video memory).

[0114] This comparative example selects six pairs of multi-source remote sensing images for testing. The first pair of images is a UAV optical image and a UAV thermal infrared image, the second pair of images is a ZY-3 panchromatic image and a GF-3SAR image, the third pair of images is a summer Google image and a winter Google image, the fourth pair of images is a Google optical image and a ZY-3 panchromatic image, the fifth pair of images is a Google optical image and a GF-2 panchromatic image, and the sixth pair of images is a Google optical image and an OSM raster map image. Figures 6a-6l As shown, Figure 6a The first set of image pairs is the drone optical image. Figure 6b The first set of image pairs is the UAV thermal infrared image. Figure 6c is the ZY-3 panchromatic image of the second image pair, Figure 6d is the GF-3SAR image of the second image pair, Figure 6e The third set of images is the summer Google image. Figure 6f This is the third set of image pairs for winter Google images. Figure 6g Google Optical Image for the fourth image pair, Figure 6h This is the ZY-3 panchromatic image of the fourth image pair. Figure 6i Google Optical Image for the fifth image pair, Figure 6j is the GF-2 panchromatic image of the fifth image pair, Figure 6k Google Optical Image for the sixth image pair, Figure 6l This is the OSM raster map image of the sixth image pair.

[0115] The comparative analysis of six sets of multi-source remote sensing image data is shown in Table 1:

[0116] Table 1 Comparative analysis of test data

[0117]

[0118]

[0119] This comparative example uses the number of correctly matched points (P), matching accuracy (MA), matching point root mean square error (RMSE), and matching time (t) to evaluate the performance of the matching algorithm. Because the matching algorithm in this comparative example focuses on achieving more uniform matching results, the matching point distribution uniformity (RSD) is used to measure the uniformity of the matching results.

[0120] Among them, the correct matching points refer to the number of points where the difference between the actual position of the feature point on the image to be matched and the actual position of the feature point on the reference image is within the threshold. The correct matching points are verified by the following formula:

[0121]

[0122] In the formula, H is the affine transformation model fitted by artificially selected points to replace the real affine transformation model of the two multi-source remote sensing images, and the feature point (x ′ i ,y ′ i ) is affine transformed to its point with the same name (x i ,y i ) is less than the threshold ε, it is considered a correct matching point; in this comparative example, the threshold is set to 3. The number of correct matching points (P) refers to the number of matching points that meet the above conditions. This indicator can reflect the basic performance of the feature matching algorithm.

[0123] The matching accuracy is the ratio of the number of correct matching points to the total number of matching points. This indicator can reflect the performance of the algorithm in successful matching.

[0124] The root mean square error of the matching point is the affine transformation result x of the correct matching point ′ The square root of the ratio of the sum of the squares of the differences from the true value x to the number of correct matching points n is as follows:

[0125]

[0126] The distribution uniformity of matching points is calculated based on the distribution uniformity of matching results in five directions. The image is divided into five directions and a total of ten areas, such as Figure 7As shown in Figure 2, matching point error estimation is an important metric for measuring matching effectiveness. The root mean square error (RMS) is very sensitive to very large or very small errors in a set of transformations, so it can effectively reflect the accuracy of multi-source remote sensing image matching results. The true value of the RMS error of matching points is the actual pixel coordinates, which are free of bias, making it more suitable for matching point error estimation.

[0127] According to statistical principles, sample variance is used to represent the difference in the number of matching points in image blocks in five different directions. If the distribution of matching points in the five directions is relatively uniform, then the sample variance of the number of matching points in the five directions is small, otherwise it is large. The uniformity of matching point distribution is shown in the following formula:

[0128]

[0129] Where V is the regional statistical distribution vector, which is composed of the number of matching points in the ten regions. The greater the uniformity of the matching point distribution, the more uniform the matching point distribution is, and vice versa.

[0130] The experiment selected several representative algorithms for multi-source remote sensing image matching for comparative analysis, including the SuperPoint algorithm, ContextDesc algorithm, LoFTR algorithm, and the classic SIFT algorithm, all based on deep learning features. The SuperPoint algorithm is a deep learning self-supervised algorithm for extracting feature points and descriptors; the ContextDesc algorithm is a deep learning matching algorithm designed specifically for multimodal imagery. ContextDesc enhances original feature descriptors such as DELF by combining high-order image visual information with geometric information about key point distribution. SIFT, or scale-invariant feature transform, is a local feature descriptor with certain affine invariance and anti-interference properties. The above four algorithms and the FFM algorithm used in this comparative example were applied to the six image pairs mentioned above for matching experiments. The matching experimental results are shown in Table 2.

[0131] Table 2 Comparison of matching test results

[0132]

[0133]

[0134] As shown in Table 2, the FFM algorithm achieves good matching results on six pairs of multi-source remote sensing images and obtains a sufficient number of correct matching points within a favorable time.

[0135] Comparing Table 2 shows that for multi-source remote sensing image pairs of different modes, the FFM algorithm can obtain a large number of correct matching points. It has its own advantages and disadvantages compared to the LoFTR algorithm for different image pairs, but the number of correct matching points is far higher than that of the other three algorithms. Due to the large grayscale differences in multi-source remote sensing images and inconsistent local gradient information of key points, the SIFT algorithm fails to match visible light images with thermal infrared images, panchromatic images with SAR images, and optical images with raster maps. Compared with the SIFT algorithm, the FFM algorithm is more stable for matching multi-source remote sensing images with large grayscale differences and inconsistent local gradient information. The SuperPoint algorithm has significantly improved the number of correct matching points, the root mean square error of matching points, and the time compared to the SIFT and ContextDesc algorithms, indicating that it is more adaptable to multi-source remote sensing imagery, but its overall performance is lower than that of the FFM algorithm. The ContextDesc algorithm combines multiple features for matching, but its matching effect is relatively poor, indicating that it is not fully adaptable to multi-source remote sensing images with large differences. The ContextDesc algorithm fails to match panchromatic images with SAR images, and optical images with raster maps, indicating that the algorithm has poor resistance to nonlinear radiation distortion and local gradient information differences in multi-source remote sensing images.

[0136] In terms of the root mean square error (RMSE) of matching points, the LoFTR and SuperPoint algorithms perform well compared to the SIFT and ContextDesc algorithms, but there is a certain gap compared to the FFM algorithm, especially for panchromatic and SAR images. This comparison shows that the FFM algorithm has higher feature positioning accuracy. Because the FFM algorithm goes through two stages: the initial matching sliding window search detection algorithm and the matching calibration, it is inferior to the SuperPoint and LoFTR algorithms in terms of time. The RMSE results of the FFM algorithm tested on the six groups of images show some differences. Compared with the third group of data, the first and second groups of data contain more buildings. Due to the different projection parallaxes of buildings in different images, there are large local deformations in the building areas between images. This local geometric deformation is difficult to eliminate using affine transformation models, so the RMSE of its matching results is relatively large.

[0137] Regarding the uniformity of matching point distribution, since other algorithms have fewer correct matching points, we focus on comparing the FFM algorithm with the LoFTR algorithm. The comparison results are shown in Table 3.

[0138] Table 3 Comparison of matching point distribution uniformity

[0139]

[0140] As shown in Table 3, the FFM algorithm has a greater uniformity of matching points on the six sets of multi-source remote sensing image pairs than LoFTR. The uniformity of matching points is calculated using logarithmic operations, which is reflected in the variance of the five-directional distribution. The FFM's matching point uniformity is significantly better than LoFTR. This experiment proves the effectiveness of the sliding window adaptive score detection algorithm in detecting and matching sparse area features. After the fine matching is completed in this comparative example, the matching results are as follows: Figures 8a-8f shown; among them, Figure 8a This is a schematic diagram of the fine matching results of the first set of image pairs. Figure 8b This is a schematic diagram of the fine matching results of the second set of image pairs. Figure 8c This is a schematic diagram of the fine matching results of the third group of image pairs. Figure 8d This is a schematic diagram of the fine matching results of the fourth group of image pairs. Figure 8e This is a schematic diagram of the fine matching results of the fifth group of image pairs. Figure 8f This is a schematic diagram of the fine matching results of the sixth group of image pairs.

[0141] It can be seen that the FFM algorithm has strong adaptability to multi-source remote sensing images, and a considerable number of matching point pairs are obtained, and the feature points are distributed relatively evenly. For the small number of mismatched points that still exist, the PROSAC algorithm is used to eliminate the mismatches to achieve the purpose of purifying the matching point pairs. The purification results are shown in the figure below. Figures 9a-9f As shown, Figure 9a This is a schematic diagram of the purification results of the first group of image pairs. Figure 9b This is a schematic diagram of the purified matching results of the second group of image pairs. Figure 9c This is a schematic diagram of the purification results of the third group of image pairs. Figure 9d This is a schematic diagram of the purification results of the fourth group of image pairs. Figure 9e This is a schematic diagram of the purification results of the fifth group of image pairs. Figure 9f This is a schematic diagram of the purification results of the sixth group of image pairs.

[0142] It can be seen that the PROSAC algorithm can effectively eliminate mismatched point pairs, and the matching point pairs finally retained also maintain a uniform distribution to a large extent, which lays a good foundation for subsequent image registration and fusion work.

[0143] In order to more intuitively show the performance of several algorithms, Figures 10a-10e The matching results of LoFTR algorithm, SuperPoint algorithm, ContextDesc algorithm, SIFT algorithm and FFM algorithm after being purified by PROSAC algorithm on the first, third and fourth groups of multi-source remote sensing image pairs are shown. Figure 10a This is the matching effect of the LoFTR algorithm on the first, third, and fourth groups of multi-source remote sensing image pairs after being purified by the PROSAC algorithm. Figure 10bThis is the matching effect of the SuperPoint algorithm on the first, third, and fourth groups of multi-source remote sensing image pairs after being purified by the PROSAC algorithm. Figure 10c This is the matching effect of the SIFT algorithm on the first, third, and fourth groups of multi-source remote sensing image pairs after being purified by the PROSAC algorithm. Figure 10d This is the matching effect of the ContextDesc algorithm on the first, third, and fourth groups of multi-source remote sensing image pairs after being purified by the PROSAC algorithm. Figure 10e This is the matching effect of the FFM algorithm on the first, third, and fourth groups of multi-source remote sensing image pairs after being purified by the PROSAC algorithm.

[0144] Figures 10a-10e Results show that, compared to SuperPoint, ContextDesc, and SIFT, FFM can better overcome the matching difficulties caused by grayscale gradient and scale differences between conventional optical and thermal infrared images, and between optical and panchromatic images, obtaining a significant number of correctly matched point pairs. For optical images taken at different times, FFM demonstrates significant advantages in areas with varying vegetation, thanks to its ability to learn the ground truth relationships between feature vectors during training.

[0145] Multi-source remote sensing image registration is one of the important purposes of image matching. Therefore, the quality of the registration effect can intuitively indicate the quality of the matching result. The purified matching point pairs are used in the multi-source image registration experiment. The affine transformation parameters are calculated by matching point pairs to correct the multi-source images. The final registration result and the local window enlargement are shown in the figure. Figures 11a-11f As shown, Figure 11a This is the final registration result of the first set of image pairs and the local window enlargement image. Figure 11b This is the final registration result of the second set of image pairs and the local window enlargement image. Figure 11c This is the final registration result of the third set of image pairs and the local window enlargement image. Figure 11d This is the final registration result of the fourth set of image pairs and the local window enlargement image. Figure 11e This is the final registration result of the fifth set of image pairs and the local window enlargement image. Figure 11f This is the final registration result of the sixth image pair and the local window enlargement image.

[0146] Depend on Figures 11a-11f The FFM algorithm is highly adaptable to the registration of visible light and thermal infrared images, panchromatic and SAR images, and other types of images. Relatively accurate registration is achieved in localized areas with large grayscale differences and distinct features, with the registration error in each region generally controlled within 3 pixels. This registration result demonstrates the high positional accuracy and uniform distribution of the FFM algorithm, demonstrating its strong performance.

[0147] The present invention can, during the early feature extraction, through feature extraction fusion, make the acquired image fine fusion features have both high-level information and low-level information, which can ensure both positioning accuracy and globality and anti-interference ability; and also perform feature transformation fusion on the extracted high-level features, improve the similarity between matching points in the high-level features of the image to be matched, and make the matching results more reliable. During the later feature matching, the high-level fusion features are first compared for coarse matching to obtain a matching result that is more in line with the global features, and then the fine fusion features are compared to correct the matching results, so that the matching results take into account the reliability of both global aspects and accuracy, with higher resolution and more accurate positioning. Before performing feature transformation fusion, all high-level feature vectors are sinusoidally encoded so that there is a unique and definite correspondence between features at different levels, avoiding the mismatch problem caused by the excessive similarity between feature vectors in the sparse texture area of the image; and when performing coarse matching, the score threshold is adaptively reduced through the sliding window, thereby increasing the number of matching points screened out in the sparse area, thereby comprehensively improving the matching effect of the sparse area from two aspects.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A multi-source remote sensing image deep feature fusion matching method, characterized in that: Here are the steps: 1) Constructing a matching model, wherein the matching model includes a feature extraction network, a feature transformation module, a dense matching module, and a calibration optimization module; 2) The acquired remote sensing image pairs to be matched are input into the feature extraction network, which extracts features from each image to be matched, and obtains high-level features of each image to be matched, as well as fine fusion features that have both high-level fine positioning information and low-level global information; 3) The obtained high-level features of the image pairs to be matched are simultaneously input into the feature transformation module. By fusing the neighborhood information of the own image and the high-level features of the image to be matched, feature transformation and fusion are performed on each image to be matched, and high-level fusion features with correlation are obtained for each image to be matched in the image pair to be matched; 4) Densely match all feature vectors on the high-level fusion features of the image pair to be matched, and obtain a rough matching result based on the similarity between the feature vectors; 5) Mapping the coarse matching results to the fine fusion features to perform calibration and optimization on the dense matching to obtain the fine matching results; The process of mapping the coarse matching results to the low-level features to calibrate and optimize the dense matching is as follows: I) Centering N pairs of feature points on the correlated high-level fusion features of each image to be matched, wherein the N pairs of feature points refer to a set of coarse matching points screened after coarse matching; cropping N pairs of local windows of size m×m on the corresponding high-level fusion features; II) Mapping the N pairs of windows to the fine fusion features of the image to be matched, obtaining N pairs of local fine windows centered on the coarse matching feature point pairs, inputting the N pairs of local fine windows into the feature transformation module, performing several transformations on them, and generating N pairs of local fine fusion feature maps of images A and B centered on the coarse matching feature point pairs. III) Each The eigenvector corresponding to the center point P and the corresponding All vectors in are associated to generate The expected value of the matching probability distribution of each pixel with P; the expected value of the matching probability distribution is calculated as follows Where V A (P) is The eigenvector of the center point P, V B (x) is The feature vector of a certain pixel point x, y is the pixel gradient of pixel point x on image B; the pixel point with the highest calculated probability value is the fine matching result with sub-pixel accuracy of point P on image A on image B, and this result is used as the final matching result.

2. The multi-source remote sensing image deep feature fusion matching method according to claim 1, characterized in that: The feature extraction network includes three downsampling layers and two upsampling layers; The first downsampling layer is used to downsample the input image to be matched to obtain a high-level feature map of 1 / 2 of the original dimension of the image to be matched; the second downsampling layer is used to downsample the high-level feature map of 1 / 2 of the original dimension of the image to be matched to obtain a high-level feature map of 1 / 4 of the original dimension of the image to be matched; the third downsampling layer is used to downsample the high-level feature map of 1 / 4 of the original dimension of the image to be matched to obtain a high-level feature map of 1 / 8 of the original dimension of the image to be matched; The first upsampling layer is used to upsample the high-level feature map of the original dimension 1 / 8 of the input image to be matched into a low-level feature map of the original dimension 1 / 4 of the image to be matched, and at the same time, perform convolution processing on the high-level feature map of the original dimension 1 / 4, and then fuse the low-level feature map of the original dimension 1 / 4 and the high-level feature map of the original dimension 1 / 4 to obtain a fused feature map of the original dimension 1 / 4; the second upsampling layer is used to upsample the fused feature map of the original dimension 1 / 4 of the input image to a low-level feature map of the original dimension 1 / 2, and at the same time, perform convolution processing on the high-level feature map of the original dimension 1 / 2, and then fuse the low-level feature map of the original dimension 1 / 2 and the high-level feature map of the original dimension 1 / 2 to obtain a fine fused feature map of the original dimension 1 / 2; Through the three downsampling layers and the two upsampling layers, a high-level feature map of 1 / 8 of the original dimension of each image to be matched and a fine fusion feature map of 1 / 2 of the original dimension are output for fusion feature matching.

3. The multi-source remote sensing image deep feature fusion matching method according to claim 1, characterized in that: The specific steps of fusion feature matching of the feature transformation module are as follows: ①Add position information to the high-level feature map of 1 / 8 of the original dimension of the image to be matched, so that the features uniquely correspond to their positions on the original image; ② Flatten the high-level feature maps of the image to be matched with position information into one-dimensional vectors, input them into the feature transformation module, and after multiple interleaving processes through the attention layer, output the high-level fusion features with correlation for each image to be matched; The attention layer includes a self-attention layer, a cross-attention layer and an attention layer; the self-attention layer is used to fuse the features of the reference image with position information in the input image pair to be matched with its own local neighborhood information to generate a new feature map; the cross-attention layer fuses the features of the reference image with position information in the input image pair to be matched with the features of the other image to be matched; The attention layer compares the similarity between the input query vector and each key feature, selects the feature information of the vector with high similarity, normalizes the selected result, and superimposes it with the flattened one-dimensional vector of the high-level feature map with position information of each image to be matched, thereby obtaining a fusion feature of the fusion position information, its own neighborhood information and the information of the image to be matched; The multiple interleaving processes refer to inputting the obtained fusion features into the focus layer for re-interleaving, repeating the above process, and finally outputting the high-level fusion features of each image to be matched.

4. The multi-source remote sensing image deep feature fusion matching method according to claim 2, characterized in that: The three downsampling layers all adopt a convolution block structure, and the input feature map and the output feature map are element-wise superimposed, and the feature map obtained after superposition is used as the downsampling result.

5. The multi-source remote sensing image deep feature fusion matching method according to claim 3, characterized in that: The adding of position information is to add sinusoidal coding to each pixel feature.

6. The multi-source remote sensing image deep feature fusion matching method according to claim 3, characterized in that: The coarse matching uses a score matrix between high-level fusion feature vectors to represent the similarity between all feature vectors. If the similarity is greater than a score threshold, it is considered a correct match. The score matrix S between the vectors is determined by the following formula, where <,> represents the inner product; Among them, F A_tr 、F B_tr is the high-level fusion feature of the images A and B to be matched, and A×B is all possible correspondences between the pixels in the images A and B to be matched; Calculate the score matrix for all possible matching methods M×N, by maximizing the total score ∑ i,j S i,j P i,j To obtain the optimal allocation matrix P; the optimal allocation matrix P can represent the optimal correspondence between the high-level fusion feature vectors in the image A to be matched and the high-level fusion feature vectors in the image B to be matched.

7. The multi-source remote sensing image deep feature fusion matching method according to claim 6, characterized in that: The coarse matching through the score matrix adopts the sliding window adaptive score threshold detection algorithm, which is as follows: a) setting the initial score threshold as θ, setting the area of the sliding window, the horizontal sliding step size, and the vertical sliding step size, and performing sliding detection on the score of the high-level fusion feature vector; b) If the scores s of all high-level fused feature vectors in the current window are less than θ, the adaptive threshold avgθ in the matching sparse node in the window is calculated; the high-level fused feature vectors in the window are traversed, and if there is a vector in the current window with a score s> avgθ, the vector is added to the coarse matching point set and the window continues to slide; c) If the score of the feature vector in the current window is s≥θ, the window continues to slide; d) Repeat the above steps until the window slides through the high-level fusion features that are relevant to the image to be matched, and output a coarse matching point set; The area, horizontal sliding step length, and vertical sliding step length of the sliding window are calculated as follows: ws is the area of the sliding window, hl is the horizontal sliding step size, and vl is the vertical sliding step size; The calculation formula of the adaptive threshold avgθ in the matching sparse node is as follows Where n is the number of feature vectors in the sliding window, s i is the matching score of each feature vector in the sliding window.

8. The multi-source remote sensing image deep feature fusion matching method according to claim 1, characterized in that: After obtaining the fine matching results, the PROSAC algorithm is used to check and eliminate false matches again.

Images