Remote sensing image matching method and device based on meta-learning

Abstract

The invention discloses a remote sensing image matching method and device based on meta-learning, and the method comprises the steps: selecting a first preset number of image pairs from an existing data set, carrying out the pairing marking of salient region points on each pair of images, enabling each pair of images to have a second preset number of matching pairs, and constructing a metadata set; executing a transformation operation on one image in each image pair from the existing data set to construct a coarse annotation data set; constructing a feature pairing network and a significance judgment network; and iteratively executing a double-layer optimization step through different data sets until the approximate optimal parameter values of the feature pairing network and the significance judgment network do not change significantly or reach a preset number of iterations, so as to obtain corresponding optimal parameter values. The method solves the problems that a traditional feature extraction operator is difficult to effectively extract significant features between different-source remote sensing images and is difficult to deal with complex and changeable practical application scenes.

Description

Technical Field

[0001] The present application relates to the field of remote sensing technology, and in particular to a remote sensing image matching method and device based on meta-learning. Background Art

[0002] Autonomous navigation and positioning technology mainly relies on image matching algorithms to determine the current position of the aircraft by comparing the real-time acquired image with a known map or reference image. In the field of heterogeneous remote sensing image matching, traditional methods are mostly based on feature extraction and matching frameworks, such as SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features) and other feature extraction operators. These methods detect key points in the image and extract feature descriptors around them, and then use the similarity between feature descriptors for matching, thereby achieving alignment between images.

[0003] However, due to the significant differences in imaging conditions, resolution, grayscale values ​​and scene features between heterogeneous remote sensing images (such as aerial photos and satellite images), traditional feature extraction operators often find it difficult to effectively extract significant features, resulting in limited matching accuracy. In addition, these methods usually rely on manually designed feature extraction rules, lack adaptability and generalization capabilities, and are difficult to cope with complex and changing practical application scenarios. Summary of the invention

[0004] In the embodiments of the present application, a remote sensing image matching method based on meta-learning is provided to solve the problem that due to the significant differences in imaging conditions, resolution, grayscale values ​​and scene features between heterogeneous remote sensing images, traditional feature extraction operators often find it difficult to effectively extract significant features and have difficulty in coping with complex and changeable practical application scenarios.

[0005] In a first aspect, an embodiment of the present application provides a remote sensing image matching method based on meta-learning, the method comprising: selecting a first preset number of image pairs from an existing data set, and performing paired annotation of salient area points on each pair of images so that each pair of images has a second preset number of matching pairs to construct a meta-data set in meta-learning; wherein the image pairs include aerial photos and satellite photos of the same area; selecting all image pairs from an existing data set, performing a transformation operation on one image in each image pair to construct a coarsely annotated data set; constructing a feature pairing network and a saliency judgment network; the feature pairing network comprises a convolutional feature extraction module, a position encoding module and an attention module, the network is used to extract similar features from the image pairs to be matched and perform pairing; the saliency judgment network comprises a feature input, a multi-layer perceptron network and a saliency input. The network is used to receive two sets of features processed by a feature pairing network and output a saliency score for each feature point; a loss function of the feature pairing network is set to guide it to predict the matching relationship between corresponding features in an image pair; a loss function of the saliency judgment network is set to obtain its ability to identify areas that have a significant impact on feature matching performance; an overall loss function is set to optimize the performance of the feature pairing network and the saliency judgment network by combining their loss values; a two-layer optimization step is iteratively performed through different data sets until the approximate optimal parameter values ​​of the feature pairing network and the saliency judgment network no longer change significantly or reach a preset number of iterations, so as to obtain the corresponding optimal parameter values; wherein the parameters of the feature pairing network are optimized through a coarsely annotated data set, and the parameters of the saliency judgment network are optimized through a meta-data set.

[0006] In a possible implementation, the transformation operation is performed on one image in each image pair, including: performing an overlay alignment operation on the image pair to ensure that the two images correspond in spatial position to obtain an overlay image; selecting a small area in the overlay image, the small area covering the corresponding parts of the aerial image and the satellite image at the same time; determining the center point of the small area of ​​the aerial image in the overlay image as the center, rotating and scaling it to obtain a first transformation area, and calculating a first transformation matrix from the small area of ​​the aerial image to the first transformation area; performing a random perturbation operation on the first transformation area to obtain a second transformation area, and calculating a second transformation matrix from the first transformation area to the second transformation area; obtaining a minimum rectangle surrounding the second transformation area as a third transformation area, and calculating a third transformation matrix from the second transformation area to the third transformation area; performing matrix multiplication on the first transformation matrix, the second transformation matrix and the third transformation matrix to obtain a homography transformation matrix from the small area of ​​the aerial image in the overlay image to the third transformation area; wherein the homography transformation matrix is ​​used to map feature points on the satellite image to corresponding positions in the third transformation area; and organizing all the image pairs that have undergone transformation operations and the corresponding homography transformation matrices into a coarsely labeled data set.

[0007] In a possible implementation, a convolutional neural network is used in a convolutional feature extraction module of a feature pairing network to extract multi-scale features from an input image pair, and its inductive bias is used to capture local features of the image, and a feature pyramid network is introduced to enhance the multi-scale feature extraction capability, thereby extracting a feature map; the extracted feature map is position-encoded in a position encoding module of the feature pairing network, and two-dimensional expansion is used to provide unique position information for each feature point in the feature map in a sinusoidal format, and the position encoding is fused with the feature map so that the feature points contain position information, thereby enhancing the ability of the feature pairing network to produce matches in indistinguishable areas; wherein the indistinguishable areas are areas with similar features but different positions; the attention module of the feature pairing network includes a self-attention layer and a cross-attention layer; different feature points on the same image are input in the self-attention layer to distinguish different areas within the image; corresponding feature points in the image pair are input in the cross-attention layer to explore the relationship between the feature points of the two images.

[0008] In a possible implementation, the input layer of the saliency judgment network is used to receive two sets of features processed by the feature pairing network and the loss value between the two sets of features; wherein the two sets of features are feature sets including multiple feature points extracted from the image pair; a fusion mechanism is designed to fuse the two sets of features and the loss values ​​between them to generate fused features; a multi-layer perceptron network is constructed to process the fused features to obtain perceptual features; and the output layer converts the perceptual features into saliency scores for each pair of feature points for output.

[0009] In a possible implementation, the loss function of the feature pairing network is set to guide the prediction of the matching relationship between corresponding features in the image pair. The loss function of the feature pairing network is expressed as follows: in, The feature pairing network matches pairs during training. The loss value, w is the parameter of the feature pairing network, Output confidence matrix P for feature pairing network c The value of the i-th row and j-th column on represents the confidence that the i-th point on the first image matches the j-th point on the second image. λ is a hyperparameter. is the confidence that the i-th point on the first image does not match the j-th point on the second image.

[0010] In a possible implementation, the loss function of the saliency judgment network is set to obtain its ability to identify regions that have a significant impact on feature matching performance. The loss function of the saliency judgment network is expressed as follows: in, Matching pairs for feature pairing networks The loss value is used as the input of the saliency judgment network, θ is the parameter of the saliency judgment network, s ij The pair matching output by the saliency judgment network Saliency score, V(.,θ) is the saliency judgment network.

[0011] In a possible implementation, the overall loss function is set by combining the loss value of the feature pairing network and the loss value of the significance judgment network, and optimizing their performance at the same time. The expression of the overall loss function is as follows: in, is a set of real matching pairs, which comes from the annotations of real point pairs in the metadata set, where each element is a matching pair if and is the pairing point, the position value is 1, otherwise it is 0, V(.,θ) is the saliency judgment network, is the loss value of the feature pairing network, L is the value of the total loss function, and V(.,θ) is the significance judgment network.

[0012] In a possible implementation, the iteratively performing the two-layer optimization step through different data sets includes: using the parameters of the current saliency judgment network of the coarsely labeled data set, iteratively optimizing the parameters of the feature pairing network through the gradient descent method, and obtaining the approximate optimal parameter values ​​of the feature pairing network; using the meta data set and the approximate optimal parameter values ​​of the feature pairing network, iteratively optimizing the parameters of the saliency judgment network through the gradient descent method, and obtaining the approximate optimal parameter values ​​of the saliency judgment network.

[0013] In a possible implementation, when the fixed optimal parameter value of the saliency judgment network is obtained, the expression of the optimization process of the parameters of the feature pairing network is: * (θ * ) = argmin w L C (w,θ * ), where w is the parameter of the feature pairing network, θ * is the optimal parameter value of the significance judgment network, L C (w,θ * ) is the optimal parameter value of the feature pairing network in the saliency judgment network, which is fixed to θ * The loss function for the coarsely labeled dataset when , argmin is the operator used to find the loss function that makes L C (w,θ * )The value of the parameter w of the minimum feature pairing network, w * (θ * ) is the result of the optimization process of the parameters of the feature pairing network. When the optimal parameter value of the saliency judgment network is fixed to θ* When , the optimal parameter value of the feature pairing network is w * ; When the fixed optimal parameter value of the feature pairing network is obtained, the expression of the optimization process of the parameters of the significance judgment network is: θ * =argmin θ L meta (w * (θ)); where θ * When the optimal parameter value of the feature pairing network is fixed to w * When , the optimal parameter value of the saliency judgment network, θ is the parameter of the saliency judgment network, w * is the optimal parameter value of the feature pairing network, L meta (w * (θ)) is the optimal parameter value of the feature pairing network and is fixed to w * , the meta-loss function of the saliency judgment network on the meta-dataset, and argmin is the operator used to find the value that makes L meta (w * (θ)) is the value of the smallest parameter θ; where the expression of the meta-loss function is: Among them, L meta (w * (θ)) is the optimal parameter value of the feature pairing network and is fixed to w * , the meta-loss function of the saliency judgment network on the metadata dataset, M is the total number of matching pairs in the metadata set, is a pair of matches in the metadata set, To match this pair And the optimal parameter value w of the feature pairing network * The meta-loss function value of , V(.,θ) is the saliency judgment network, and meta data is the meta data set.

[0014] In a second aspect, an embodiment of the present application provides a remote sensing image matching device based on meta-learning, the device comprising: constructing a metadata data set module, for selecting a first preset number of image pairs from an existing data set, and performing pairing annotations on salient area points on each pair of images, so that each pair of images has a second preset number of matching pairs, so as to construct a metadata data set in meta-learning; wherein the image pairs include aerial photos and satellite photos of the same area; constructing a coarsely annotated data set module, for selecting all image pairs from an existing data set, performing a transformation operation on one image in each image pair, so as to construct a coarsely annotated data set; constructing a network module, for constructing a feature pairing network and a saliency judgment network; the feature pairing network comprises a convolutional feature extraction module, a position encoding module and an attention module, the network is used to extract similar features from the image pairs to be matched and perform pairing; the saliency judgment network comprises a feature input, a multi-layer perceptron network and a saliency judgment network. The network is used to output the significance level, and the network is used to receive two sets of features processed by the feature pairing network and output the significance score of each feature point; a loss function module is set to set the loss function of the feature pairing network to guide it to predict the matching relationship between corresponding features in the image pair; the loss function of the saliency judgment network is set to obtain its ability to identify areas that have a significant impact on the feature matching performance; an overall loss function is set to optimize the performance of the feature pairing network and the saliency judgment network by combining their loss values; an iterative execution of a double-layer optimization step module is used to iteratively execute the double-layer optimization step through different data sets until the approximate optimal parameter values ​​of the feature pairing network and the saliency judgment network no longer change significantly or reach a preset number of iterations, so as to obtain the corresponding optimal parameter values; wherein the parameters of the feature pairing network are optimized through the coarsely labeled data set, and the parameters of the saliency judgment network are optimized through the meta-data set.

[0015] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects:

[0016] The embodiment of the present application provides a remote sensing image matching method based on meta-learning. By constructing a feature pairing network, the network uses a convolutional feature extraction module to extract deep features of image pairs, and combines a position encoding module and an attention module to effectively capture subtle differences and similarities between images, thereby achieving more accurate feature pairing. This not only improves the accuracy of image matching, but also enhances the adaptability of the system to image pairs. The introduction of a saliency judgment network can automatically identify and annotate salient areas in an image. These areas are often the key to image matching and are of great significance for improving matching performance. By training the saliency judgment network, the system can learn and identify which feature points have a significant impact on matching performance, thereby further optimizing the matching results. By constructing a meta data set and a coarsely annotated data set, and using a two-layer optimization method to train the feature pairing network and the saliency judgment network, a variety of data samples can be exposed during the training process, thereby improving the generalization ability of the model. This training strategy enables the model to adapt faster and produce accurate matching results when facing a new data set. Although manual annotation is required when constructing a meta data set, this method significantly reduces costs compared to a method that relies entirely on manual annotation. By using the feature pairing network and the saliency judgment network for iterative optimization, the matching performance can be gradually improved with less manual intervention, reducing the reliance on large amounts of labeled data. The combination of the feature pairing network and the saliency judgment network makes the image matching process more efficient. By quickly extracting key features and identifying salient areas, the system can quickly find the best matching points between images, thereby speeding up the entire matching process. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A flowchart of a remote sensing image matching method based on meta-learning provided in an embodiment of the present application;

[0019] Figure 2 A specific flow chart of performing a transformation operation on one image in each image pair provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of each transformation area provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of feature points extracted using the Loftr matching method and matching results provided in an embodiment of the present application;

[0022] Figure 5 A schematic diagram of feature points and matching results extracted using the constructed feature pairing network and saliency judgment network provided in an embodiment of the present application;

[0023] Figure 6 A schematic diagram of feature points and matching results extracted in mountainous and Gobi regions using the Loftr matching method provided in an embodiment of the present application;

[0024] Figure 7 A schematic diagram of feature points and matching results extracted in mountainous and Gobi regions using the constructed feature pairing network and saliency judgment network provided in an embodiment of the present application;

[0025] Figure 8 A schematic diagram of the distribution convergence process of feature points in the training process of the present application in which a meta-learning module is added provided in an embodiment of the present application;

[0026] Fig. 9 A schematic diagram of a remote sensing image matching device based on meta-learning provided in an embodiment of the present application;

[0027] Fig.10 A schematic diagram of a remote sensing image matching server based on meta-learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] The following describes some of the techniques involved in the embodiments of the present application to facilitate understanding, and they should be considered as merely exemplary. Therefore, it should be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, some descriptions of well-known functions and structures are omitted in the following description.

[0030] The present application embodiment provides a remote sensing image matching method based on meta-learning, such as Figure 1 As shown, the method includes steps S101 to S107. Among them, Figure 1 This is only an execution order shown in the embodiment of the present application, and does not represent the only execution order of a remote sensing image matching method based on meta-learning. If the final result can be achieved, Figure 1The steps shown may be performed in parallel or reversed.

[0031] S101: Select a first preset number of image pairs from an existing data set, and perform paired annotation of salient region points on each pair of images, so that each pair of images has a second preset number of matching pairs, so as to construct a meta-data set in meta-learning. The image pairs include aerial images and satellite images of the same area.

[0032] Specifically, 100 representative pairs of images can be manually selected from the existing dataset, that is, the first preset number can be 100 pairs. The salient area points can be areas such as houses, roads, corners, etc. that can significantly distinguish the surrounding environment. The second preset number can be 2000-3000 groups, ensuring that there is enough matching information between each pair of images. The metadata dataset not only contains the images themselves in the image pair, but also contains detailed matching point pair information between the images.

[0033] S102: Select all image pairs from the existing data set, and perform a transformation operation on one image in each image pair to construct a coarsely annotated data set.

[0034] Figure 2 A specific flow chart of performing a transformation operation on one image in each image pair provided in the embodiment of the present application is as follows: Figure 2 As shown, it includes steps S201 to S207.

[0035] S201: Performing a superposition alignment operation on the image pair to ensure that the two images correspond in spatial position to obtain a superimposed image.

[0036] Specifically, in the present application, the image pair is an aerial image and a satellite image of the same area, and the overlay alignment is to ensure that the aerial image and the satellite image correspond in spatial position.

[0037] S202: Select a small area in the overlay image, where the small area covers corresponding parts of the aerial image and the satellite image.

[0038] Specifically, the selection of this small area should be based on specific research or application needs, such as focusing on the topography, building layout, vegetation coverage, etc. of a particular location.

[0039] Furthermore, the coordinates of the vertices of the small area selected in the overlay image in the satellite image are recorded and the coordinates are marked as [x1, x2, x3, x4]. The upper left corner of the overlay image can be used as the origin of the coordinates. Since the overlay image is formed by aligning the aerial image and the satellite image in space, the coordinates of the vertices of the small area selected in the overlay image in the aerial image are also [x1, x2, x3, x4].

[0040] S203: Determine the center point of the small area of ​​the aerial image in the overlay image as the center, perform rotation and scaling operations on it to obtain a first transformation area, and calculate a first transformation matrix from the small area of ​​the aerial image to the first transformation area.

[0041] Specifically, rotating and scaling the small area of ​​the aerial image in the overlay image can obtain four transformed vertices [x′1, x′2, x′3, x′4], and the first transformation matrix H1 from the small area of ​​the aerial image to the first transformation area can be calculated by the vertex coordinates before transformation [x1, x2, x3, x4] and the corresponding vertex coordinates of the first transformation area [x′1, x′2, x′3, x′4] using the coordinate transformation formula of homogeneous coordinates.

[0042] It should be noted that homogeneous coordinates are a commonly used representation method in projective geometry. Take the vertex x1 before transformation and the corresponding vertex x′1 after transformation as an example, and let their coordinates be (m1,n1) and (m′1,n′1) respectively, then we have For each pair of corresponding coordinate points, two equations can be established using the transformation formula of the homography matrix. This will result in 8 equations (because each pair of corresponding coordinate points contributes two equations, and there are four pairs of points in total). Therefore, the calculation formula for the first transformation matrix is: There are actually only 8 unknowns (h1 to h8) to solve, so the last element is 1. Since these equations are linear, but since the denominator h7m i +h8n j +1 (i=1,2,3,4), they are not standard linear equations. Through some algebraic transformations (such as cross multiplication to eliminate the denominator), these equations can be transformed into standard linear equations. Once we have this linear equation system, we can use linear algebra methods to solve it to find the values ​​of h1 to h8. In practical applications, more numerical methods such as least squares or SVD (singular value decomposition) can be used to solve H1.

[0043] S204: Perform a random perturbation operation on the first transformation region to obtain a second transformation region, and calculate a second transformation matrix from the first transformation region to the second transformation region.

[0044] Specifically, on the basis of the vertex coordinates [x′1, x′2, x′3, x′4] of the first transformed region, random small perturbations are added to each vertex. These perturbations can be random offsets along the x-axis (left-right direction) and y-axis (up-down direction) of the image coordinate system, and the offset is usually several pixels. The vertex coordinates of the second transformed region after the perturbation operation are [x″1, x″2, x″3, x″4]. Using the coordinate transformation formula of homogeneous coordinates, we can calculate the second transformation matrix H2 according to the vertex coordinates [x′1, x′2, x′3, x′4] of the first transformed region and the corresponding vertex coordinates [x″1, x″2, x″3, x″4] of the second transformed region. Similar to the process of calculating the first transformation matrix H1, it is necessary to establish a set of equations containing H2 elements for each pair of corresponding coordinate points. Then, these equations are converted into a standard linear system of equations by eliminating the denominator through cross multiplication. More numerical methods such as least squares method or SVD (singular value decomposition) can be used to solve H2.

[0045] S205: Acquire a minimum rectangle surrounding the second transformation region as a third transformation region, and calculate a third transformation matrix from the second transformation region to the third transformation region.

[0046] Specifically, a minimum rectangle that can enclose these vertices is determined based on the vertex coordinates [x″1, x″2, x″3, x″4] of the second transformation area. This rectangle will serve as the third transformation area, and its vertex coordinates are [x″′1, x″′2, x″′3, x″′4]. The process of calculating the third transformation matrix H3 is similar to the process of calculating the first transformation matrix H1.

[0047] S206: Perform matrix multiplication on the first transformation matrix, the second transformation matrix and the third transformation matrix to obtain a homography transformation matrix from the small area of ​​the aerial image in the overlay image to the third transformation area, wherein the homography transformation matrix is ​​used to map the feature points on the satellite image to corresponding positions in the third transformation area.

[0048] Figure 3 Schematic diagram of each transformation area provided in the embodiment of the present application. From left to right, Figure 3 The dark blue area of ​​the first image in the figure is a small area selected from the overlay image, which is the small area of ​​the satellite image in the overlay image. The dark blue area of ​​the second image is a small area of ​​the aerial image with the same vertex coordinates as the small area of ​​the satellite image in the overlay image. The light blue area is the first transformation area. The area formed by the orange line in the third image is the second transformation area. The yellow area in the third image is the third transformation area.

[0049] S207: Arrange all transformed image pairs and corresponding homography transformation matrices into a coarsely labeled dataset.

[0050] Specifically, all the image pairs that have undergone the transformation operation are a satellite image and a small area of ​​the aerial image that has the satellite image as an image pair.

[0051] Specifically, the overall transformation matrix H from the satellite image A to the aerial image B is calculated by matrix multiplication based on the first transformation matrix H1, the second transformation matrix H2 and the third transformation matrix H3 obtained in the previous step. A2B The formula is: H A2B =H1*H2*H3. The overall transformation matrix H is obtained. A2B After that, any feature point a in the satellite image can be mapped to the corresponding point b in the three transformation regions using the overall transformation matrix. The third transformation region comes from a small area of ​​the aerial image that is used as an image pair with the satellite image. The homogeneous coordinate representation of point a is expressed as (a x ,a y ,1), through the transformation matrix H A2B Transform and obtain the homogeneous coordinates of point b (b′ x ,b′ y ,b′ w ), where b′ w is the normalization factor. By converting the homogeneous coordinates to non-homogeneous coordinates, we get the final coordinates of point b This pair of points a and b constitutes a set of feature point pairs of the coarsely labeled dataset. This feature point pair is also included in the coarsely labeled dataset.

[0052] S103: Construct a feature pairing network and a saliency judgment network.

[0053] The feature pairing network includes a convolutional feature extraction module, a position encoding module and an attention module. The network is used to extract similar features from the image pairs to be matched and pair them.

[0054] In the convolutional feature extraction module of the feature pairing network, a convolutional neural network is used to extract multi-scale features from the input image pair, its inductive bias is used to capture the local features of the image, and a feature pyramid network is introduced to enhance the multi-scale feature extraction capability, thereby extracting the feature map.

[0055] Specifically, standard convolutional neural network structures such as ResNet and VGG can be used to gradually extract local features in the image by stacking multiple convolutional layers, activation layers (such as ReLU) and pooling layers (such as max pooling). In order to enhance the multi-scale feature extraction capability, the feature pyramid network (FPN) is introduced. FPN combines high-level semantic information with low-level detail information through top-down paths and lateral connections to generate multi-scale feature maps with rich semantic and detail information. The convolutional feature extraction module ultimately outputs a series of multi-scale feature maps, which will serve as inputs for subsequent position encoding and attention modules.

[0056] The extracted feature map is position-encoded in the position encoding module of the feature pairing network, and two-dimensional expansion is used to provide unique position information for each feature point in the feature map in a sinusoidal format. The position encoding is fused with the feature map so that the feature points contain position information, which enhances the ability of the feature pairing network to produce matches in indistinguishable areas. Indistinguishable areas are areas with similar features but different positions.

[0057] Specifically, in order to apply position encoding to image data, it is necessary to extend it from one dimension to two dimensions. In sequence models such as Transformer, position encoding usually appears in the form of a one-dimensional vector to provide position information for each element in the sequence. However, in image processing, it is necessary to provide each pixel or feature point with its position information in two-dimensional space. The position encoding of two-dimensional extension can be implemented in the following way: for each feature point in the feature map, assuming its position is (x, y), a one-dimensional position encoding vector is generated for its x coordinate and y coordinate respectively. These two vectors can be generated independently or combined into a two-dimensional vector. Then, these position encoding vectors are added or spliced ​​to the corresponding feature point representation. In two-dimensional extension, the sinusoidal format can be used to generate the position encoding. Fusion of the generated position encoding with the feature map is the last step of the position encoding module. There are two main ways of fusion: element-by-element addition and splicing. Element-by-element addition is to add the position encoding vector to the corresponding feature point representation vector element by element. This method is simple and direct, but it may change the representation of the original feature point. Splicing is to add the position encoding vector as a new feature dimension to the feature point representation. This method does not change the representation of the original feature point, but increases the dimension of the feature map. By introducing position encoding, the feature pairing network can better distinguish between areas with similar features but different positions. In indistinguishable areas, even if the visual features of two feature points are very similar, the network can make a distinction by comparing their position information because their position encodings are different. In this way, the feature pairing network can maintain high accuracy and robustness even in complex image matching tasks.

[0058] The attention module of the feature pairing network includes a self-attention layer and a cross-attention layer. Different feature points on the same image are input into the self-attention layer to distinguish different areas within the image. Corresponding feature points in the image pair are input into the cross-attention layer to explore the relationship between the feature points of the two images.

[0059] Specifically, the main purpose of the self-attention layer is to capture the contextual information and inter-regional dependencies within a single image by comparing and associating feature points at different positions. This helps to understand the importance of different regions in the image and their interactions. The input of the self-attention layer is the feature map of the same image after feature extraction and position encoding. This feature map contains feature points at multiple positions in the image, and each feature point is a vector that combines image information and position information. The output of the self-attention layer is an updated feature map, in which each feature point contains information about other feature points within the image, reflecting the contextual relationship and inter-regional dependencies within the image. The main purpose of the cross-attention layer is to explore the feature relationship between two images by comparing and associating feature points at corresponding positions (or similar features) between image pairs. This helps to understand the similarities, differences, and correspondences between image pairs. The input of the cross-attention layer is the feature map of the image pair after their respective feature extraction and position encoding. The two feature maps represent the feature information of the two images in the image pair. The output of the cross-attention layer is an updated feature map, in which each feature point integrates relevant information in the other image. This output reflects the relationship between feature points between image pairs.

[0060] The saliency judgment network includes feature input, multi-layer perceptron network and saliency output. The network is used to receive two sets of features processed by the feature pairing network and output the saliency score of each feature point.

[0061] The input layer of the saliency judgment network is used to receive two sets of features processed by the feature pairing network and the loss value between the two sets of features. The two sets of features are feature sets including multiple feature points extracted from the image pair. A fusion mechanism is designed to fuse the two sets of features and the loss value between them to generate fused features. A multi-layer perceptron network is constructed to process the fused features to obtain perceptual features. The output layer converts the perceptual features into the saliency scores of each pair of feature points for output.

[0062] Specifically, a fusion mechanism is designed to merge the two sets of features. This can be achieved by simple feature concatenation, feature addition, or more complex feature fusion methods such as attention weighted fusion. The purpose of fusion is to integrate the information in the two sets of features so that subsequent processing can take into account information from both images at the same time. If the feature pairing network also outputs a loss value between the two sets of features, this loss value can also be fused into the feature representation. This can be achieved by adding the loss value as an additional feature dimension to the feature map. The output layer converts the perceptual features processed by the multi-layer perceptron network into a saliency score for each pair of feature points. These scores reflect the importance or saliency of the feature points in the image. At the end of the multi-layer perceptron network, a fully connected layer is used to map the perceptual features to the space of saliency scores. Each neuron in the fully connected layer is connected to all neurons in the previous layer, so that global information can be integrated to generate the final saliency score. Before the output layer, an activation function is usually applied to normalize the score to a certain range. The output layer outputs a saliency score vector corresponding to the number of input feature points, where each element corresponds to the saliency score of a feature point.

[0063] S104: Setting a loss function of the feature pairing network to guide it to predict the matching relationship between corresponding features in the image pair.

[0064] The loss function of the feature pairing network is set to guide its prediction of the matching relationship between corresponding features in the image pair. The expression of the loss function of the feature pairing network is as follows: in, The feature pairing network matches pairs during training. The loss value, w is the parameter of the feature pairing network, Output confidence matrix P for feature pairing network c The value of the i-th row and j-th column on represents the confidence that the i-th point on the first image matches the j-th point on the second image. λ is a hyperparameter. is the confidence that the i-th point on the first image does not match the j-th point on the second image.

[0065] Specifically, the loss function defined is For each matching pair during training Calculate, where represents the i-th feature point in the first image, represents the jth feature point in the second image that is expected to match it. w represents all trainable parameters in the feature pairing network. The expression of the loss function consists of two parts, each of which is enhanced by a logarithmic function to enhance the significance of the difference. Positive sample loss Part of the measure is the correct matching of feature pairing networks Confidence Hope The closer it is to 1 (i.e. the higher the confidence), the smaller the loss will be. If the feature pairing network correctly predicts the matching relationship between the two feature points, then this part of the loss will be very low. Negative sample loss Some of them consider a feature point on the first image (indicates non-matching points) and the second image Point mismatch. Indicates the confidence level that the two points do not match, and we hope that this value is as small as possible. form, ensuring that When it is close to 0, this part of the loss is also small. λ is a hyperparameter used to adjust the relative importance between the positive and negative sample losses. The entire loss function guides the learning process of the feature pairing network through the weighted sum of these two parts. During the training process, the feature pairing network will constantly try to adjust its parameter w to minimize the total loss calculated for all training matching pairs. The feature pairing network can gradually learn how to accurately predict the matching relationship between feature points in the image pair.

[0066] S105: Setting a loss function of the saliency judgment network to obtain its ability to identify regions that have a significant impact on feature matching performance.

[0067] The loss function of the saliency judgment network is set to obtain its ability to identify areas that have a significant impact on feature matching performance. The expression of the loss function of the saliency judgment network is as follows: in, Matching pairs for feature pairing networks The loss value is used as the input of the saliency judgment network, θ is the parameter of the saliency judgment network, s ij The pair matching output by the saliency judgment network Saliency score, V(.,θ) is the saliency judgment network.

[0068] Specifically, the input of the saliency judgment network is the loss value generated by the feature pairing network during training. This loss value reflects the difficulty encountered by the feature pairing network when trying to match the feature points in the image pair. That is, the higher the loss value, the more difficult it is to match the two feature points, which means that the area around the two feature points has a significant impact on the matching performance. Then, the saliency judgment network receives these loss values ​​as input and processes them through its internal parameters θ, and finally outputs the saliency score of each feature point. This score can be regarded as the network's evaluation of the importance of the area around the feature point. The higher the score, the more significant the impact of the area on the feature matching performance.

[0069] S106: Setting an overall loss function, by combining the loss value of the feature pairing network and the loss value of the significance judgment network, and optimizing their performance at the same time.

[0070] Set the overall loss function, by combining the loss value of the feature pairing network and the loss value of the significance judgment network, and optimize their performance at the same time. The expression of the overall loss function is as follows: in, is a set of real matching pairs, which comes from the annotations of real point pairs in the metadata set, where each element is a matching pair if and is the pairing point, the position value is 1, otherwise it is 0, V(.,θ) is the saliency judgment network, is the loss value of the feature pairing network, L is the value of the total loss function, and V(.,θ) is the significance judgment network.

[0071] Specifically, the set of true matching pairs Contains all feature point pairs that should be correctly matched.

[0072] S107: Iteratively perform the two-layer optimization steps through different data sets until the approximate optimal parameter values ​​of the feature pairing network and the saliency judgment network no longer change significantly or reach a preset number of iterations, so as to obtain the corresponding optimal parameter values. The parameters of the feature pairing network are optimized through the coarsely annotated data set, and the parameters of the saliency judgment network are optimized through the metadata data set.

[0073] The two-layer optimization steps are iteratively performed through different data sets, including: using the parameters of the current saliency judgment network of the coarsely annotated data set, iteratively optimizing the parameters of the feature pairing network through the gradient descent method to obtain the approximate optimal parameter values ​​of the feature pairing network. Using the approximate optimal parameter values ​​of the meta data set and the feature pairing network, iteratively optimizing the parameters of the saliency judgment network through the gradient descent method to obtain the approximate optimal parameter values ​​of the saliency judgment network.

[0074] When the fixed optimal parameter value of the saliency judgment network is obtained, the expression of the optimization process of the parameters of the feature pairing network is:* (θ * ) = argmin w L C (w,θ * ). Among them, w is the parameter of the feature pairing network, θ * is the optimal parameter value of the saliency judgment network, L C (w,θ * ) is the optimal parameter value of the feature pairing network in the saliency judgment network, which is fixed to θ * The loss function for the coarsely labeled dataset when , argmin is the operator used to find the loss function that makes L C (w,θ * )The value of the parameter w of the minimum feature pairing network, w * (θ * ) is the result of the optimization process of the parameters of the feature pairing network. When the optimal parameter value of the saliency judgment network is fixed to θ * When , the optimal parameter value of the feature pairing network is w * .

[0075] When the fixed optimal parameter value of the feature pairing network is obtained, the expression of the optimization process of the parameters of the significance judgment network is: θ * =argmin θ L meta (w * (θ)). Where θ * When the optimal parameter value of the feature pairing network is fixed to w * When , the optimal parameter value of the saliency judgment network, θ is the parameter of the saliency judgment network, w * is the optimal parameter value of the feature pairing network, L meta (w * (θ)) is the optimal parameter value of the feature pairing network and is fixed to w * , the meta-loss function of the saliency judgment network on the meta-dataset, and argmin is the operator used to find the value that makes L meta (w * (θ)) is the value of the smallest parameter θ. The expression of the meta-loss function is: Among them, L meta (w * (θ)) is the optimal parameter value of the feature pairing network and is fixed to w * , the meta-loss function of the saliency judgment network on the metadata dataset, M is the total number of matching pairs in the metadata set, is a pair of matches in the metadata set, To match this pair And the optimal parameter value w of the feature pairing network *The meta-loss function value of , V(.,θ) is the saliency judgment network, and meta data is the meta data set.

[0076] Specifically, this application uses two NVIDIA Geforce RTX3090 GPUs for parallel computing. NVIDIA Geforce RTX3090 is a high-performance graphics processing unit (GPU), which is particularly suitable for training and reasoning of deep learning models. Its powerful computing power can significantly improve the speed of model training. The experiments of this application are carried out on a computer based on the Ubuntu 22.04 operating system. Ubuntu 22.04 is a widely used open source operating system that provides a stable and reliable environment to support the operation of deep learning frameworks and libraries. The training process of the entire model lasted 72 hours. This shows the amount of computation and time cost required to train the model, which is usually related to the complexity of the model, the size of the data set, and the performance of the hardware used.

[0077] Total number of epoch iterations: During the training process, the model traverses the entire training dataset 100 times, that is, the total number of epoch iterations is 100. Each epoch represents the process in which the model uses the complete training dataset for one forward propagation and one backward propagation. Through multiple iterations, the model can gradually learn and optimize its parameters. The initial learning rate of the training is set to 0.001. The learning rate is a hyperparameter that controls the step size of parameter updates during the training process. An appropriate learning rate helps the model converge to the optimal solution quickly and stably. In order to prevent the model from overfitting, the experiment adds a regularization term to the loss function and sets the λ coefficient to 0.0001. The regularization term limits the complexity of the model by penalizing the complexity of the model parameters, thereby avoiding the situation where the model performs too well on the training data but performs poorly on the test data. The Adam optimization algorithm is used to update the parameters of the model. Adam is an optimization algorithm based on gradient descent. It designs independent adaptive learning rates for different parameters by calculating the first-order moment estimate and the second-order moment estimate of the gradient. The Adam algorithm is widely used in deep learning because of its fast convergence speed and good effect.

[0078] This application enhances the performance of two mainstream backbone networks (SuperPoint+SuperGlue and Loftr) in remote sensing data feature matching tasks by integrating the constructed feature pairing network and saliency judgment network as meta-learning modules. In order to comprehensively evaluate the effect of this improvement, a dedicated remote sensing data test set was constructed, and a variety of existing feature extraction and matching methods including Delf, D2Net, Loftr, and SuperGlue were compared on this test set, as well as the performance changes of these methods after the addition of the meta-learning module. Table 1 shows the performance comparison results between the meta-learning module provided by this application and several existing mainstream matching methods (including Delf, D2Net, Loftr, and SuperGlue) on the constructed remote sensing data test set of the embodiment of this application.

[0079] Table 1

[0080]

[0081] It should be noted that AUC-RANSAC(5PX) and AUC-RANSAC(20PX) are indicators used to evaluate the performance of feature matching algorithms. 5PX and 20PX indicate that the threshold of pixel error is set to 5 pixels and 20 pixels respectively. Precision P (Precision) is another important indicator to measure the accuracy of matching results.

[0082] The results in Table 1 show that both the framework based on Loftr and SuperGlue have achieved significant performance improvements after incorporating the meta-learning module. Specifically, Loftr+ours (adding a meta-learning module to the Loftr framework) has reduced the number of matching points compared to the original Loftr (from 2231 to 1577), but has improved the AUC-RANSAC (5PX) and AUC-RANSAC (20PX) accuracy by 3.5% and 4.3%, respectively, and the precision P is as high as 94.2%, an increase of 4.4 percentage points over the original method. Similarly, SuperGlue+ours (adding a meta-learning module to the SuperGlue framework) has achieved steady improvements in AUC-RANSAC accuracy and P accuracy, reaching 37.1% and 91.8% respectively, while reducing the number of matching points (from 1794 to 1099), showing stronger matching accuracy and reliability.

[0083] The key to these performance improvements lies in the optimization of the feature point extraction strategy by the meta-learning module. This application can more accurately identify and retain significant feature points in areas with strong textures, while effectively removing unstable feature points in areas with weak textures. This strategy not only reduces noise interference, but also improves the overall credibility of feature points, thereby achieving more accurate and robust matching results in the subsequent registration process.

[0084] Figure 4 A schematic diagram of feature points extracted using the Loftr matching method and matching results provided in an embodiment of the present application. Figure 5 A schematic diagram of feature points and matching results extracted using the constructed feature pairing network and saliency judgment network provided in an embodiment of the present application. Figure 6 A schematic diagram of feature points and matching results extracted in mountainous and Gobi regions using the Loftr matching method provided in an embodiment of the present application. Figure 7 A schematic diagram of feature points and matching results extracted in mountainous and Gobi regions using the constructed feature pairing network and saliency judgment network provided in the embodiment of the present application. Figure 4 , Figure 5 , Figure 6 and Figure 7 As shown, the feature points extracted by the feature pairing network and the saliency judgment network constructed in the present application can be gathered in the feature salient area.

[0085] Table 2 is a comparison of the accuracy of the meta-learning module provided by this application and the existing model on the constructed remote sensing data test set implemented in this application.

[0086] Table 2

[0087]

[0088] This application has been improved on the basis of existing methods (such as Loftr, SP+SG, etc.) and introduced a meta-learning module. By testing in areas that have not appeared in the training set, such as mountainous areas and Gobi, it is verified that this application has strong generalization ability and wide adaptability. This shows that adding a meta-learning module to the existing method can not only handle common scenes, but also cope with complex and changeable actual environments. This application also combines the saliency judgment network with the Loftr and SP+SG networks to further improve the effect of feature point matching. The experimental results show that this application performs well on remote sensing datasets (such as the Fujian dataset) and MegaDepth datasets obtained during the actual flight process, especially in the AUC-RANSAC indicator. The specific performance is as follows: On the Fujian dataset, the AUC-RANSAC value of the meta-learning module (Loftr+ours and SP+SG+ours) added to this application is significantly higher than that of the existing method, especially at the 5PX threshold. The improvement is particularly obvious. On the MegaDepth dataset, the method also shows good performance, verifying its generalization ability and wide adaptability.

[0089] Figure 8 A schematic diagram of the distribution convergence process of feature points in the training process of the present application with the addition of a meta-learning module is provided for the embodiment of the present application. The changes in the distribution of feature points during the training process are shown by means of visualization. As the number of iterations increases, the feature points gradually change from being scattered to being concentrated, and cover areas in the image where semantic information is more obvious. This further demonstrates the effectiveness of the meta-learning module in extracting salient areas of the image.

[0090] In the field of remote sensing, faced with the challenges of significant changes in images and the difficulty in obtaining high-quality data sets, this application can effectively realize the function of feature extraction with only a small number of manually annotated samples. Its core advantage is that the extracted feature points are not only abundant in number, but also rich in semantic information, which is crucial for subsequent image analysis, understanding and application. The method of this application shows the characteristics of rapid convergence during the training process, which means that a higher performance level can be achieved in a shorter time, greatly improving the efficiency of research and application. In addition, the flexibility and scalability of this method also make it possible for its application in multiple fields. In addition to basic feature extraction, it can also be applied to scenarios such as sample self-selection and data annotation correction, bringing new solutions and technical means to these fields. By intelligently selecting representative samples and correcting annotation errors, this method can further improve the quality and reliability of remote sensing data processing.

[0091] The present application embodiment also provides a remote sensing image matching device 900 based on meta-learning, such as Fig. 9As shown, the device includes: a module 901 for constructing a metadata data set, a module 902 for constructing a coarsely labeled data set, a module 903 for constructing a network, a module 904 for setting a loss function, and a module 905 for iteratively executing a double-layer optimization step.

[0092] The metadata dataset construction module 901 is used to select a first preset number of image pairs from an existing dataset, and perform paired annotation of salient area points on each pair of images, so that each pair of images has a second preset number of matching pairs, so as to construct a metadata dataset for meta-learning. The image pairs include aerial images and satellite images of the same area.

[0093] The module 902 for constructing a coarsely annotated dataset is used to select all image pairs from the existing dataset, and perform a transformation operation on one image in each image pair to construct a coarsely annotated dataset.

[0094] The network construction module 903 is used to construct a feature pairing network and a saliency judgment network. The feature pairing network includes a convolutional feature extraction module, a position encoding module and an attention module. The network is used to extract similar features from the image pairs to be matched and pair them. The saliency judgment network includes a feature input, a multi-layer perceptron network and a saliency output. The network is used to receive two sets of features processed by the feature pairing network and output the saliency score of each feature point.

[0095] The loss function setting module 904 is used to set the loss function of the feature pairing network to guide it to predict the matching relationship between corresponding features in the image pair. The loss function of the saliency judgment network is set to obtain its ability to identify areas that have a significant impact on feature matching performance. The overall loss function is set to optimize the performance of the feature pairing network and the saliency judgment network by combining the loss value of the feature pairing network and the loss value of the saliency judgment network.

[0096] The iterative execution of the double-layer optimization step module 905 is used to iteratively execute the double-layer optimization step through different data sets until the approximate optimal parameter values ​​of the feature pairing network and the saliency judgment network no longer change significantly or reach a preset number of iterations, so as to obtain the corresponding optimal parameter values. Among them, the parameters of the feature pairing network are optimized through the coarsely annotated data set, and the parameters of the saliency judgment network are optimized through the metadata data set.

[0097] Some modules in the apparatus described in the present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0098] The devices or modules described in the above application embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described in various modules according to their functions. When implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, the module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0099] The methods, devices or modules described in this application can be implemented in a computer-readable program code mode. The controller can be implemented in any appropriate manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro) processor, a logic gate, a switch, an application-specific integrated circuit (English: Application Specific Integrated Circuit, referred to as: ASIC), a programmable logic controller and an embedded microcontroller. Examples of controllers include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a pure computer-readable program code mode, the controller can be completely implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included in it for implementing various functions can also be regarded as structures within the hardware component. Or even, the means for realizing various functions may be regarded as both a software module for realizing the method and a structure within a hardware component.

[0100] like Fig.10As shown, an embodiment of the present application also provides a remote sensing image matching server based on meta-learning, including a memory 1001 and a processor 1002; the memory 1001 is used to store computer-executable instructions; the processor 1002 is used to execute computer-executable instructions to implement a remote sensing image matching method based on meta-learning described above in an embodiment of the present application.

[0101] An embodiment of the present application also provides a computer-readable storage medium, which stores executable instructions. When a computer executes the executable instructions, it can implement the remote sensing image matching method based on meta-learning described above in the embodiment of the present application.

[0102] It can be known from the description of the above implementation mode that a person skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, or it can be reflected in the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the method described in the embodiment of the present application.

[0103] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations.

[0104] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. A remote sensing image matching method based on meta-learning, characterized in that: include: Selecting a first preset number of image pairs from an existing data set, and performing paired annotation of salient area points on each pair of images, so that each pair of images has a second preset number of matching pairs, so as to construct a meta-data set in meta-learning; wherein the image pairs include aerial images and satellite images of the same area; Select all image pairs from the existing dataset and perform transformation operations on one image in each image pair to construct a coarsely labeled dataset; Construct feature pairing network and saliency judgment network; The feature pairing network includes a convolutional feature extraction module, a position encoding module and an attention module. The network is used to extract similar features from the image pairs to be matched and pair them. The saliency judgment network includes feature input, multi-layer perceptron network and saliency output. The network is used to receive two sets of features processed by the feature pairing network and output the saliency score of each feature point. Setting the loss function of the feature pairing network to guide it to predict the matching relationship between corresponding features in the image pair; Set the loss function of the saliency judgment network to obtain its ability to identify regions that have a significant impact on feature matching performance; Set the overall loss function to optimize the performance of both the feature pairing network and the saliency judgment network by combining their loss values; The two-layer optimization steps are iteratively performed through different data sets until the approximate optimal parameter values ​​of the feature pairing network and the saliency judgment network no longer change significantly or reach the preset number of iterations, so as to obtain the corresponding optimal parameter values; among which, the parameters of the feature pairing network are optimized through the coarsely annotated data set, and the parameters of the saliency judgment network are optimized through the metadata data set.

2. The remote sensing image matching method based on meta-learning according to claim 1, characterized in that: The performing a transformation operation on one image in each image pair comprises: Perform a superposition alignment operation on the image pair to ensure that the two images correspond in spatial position to obtain a superimposed image; Select a small area in the overlay image, which covers the corresponding parts of the aerial image and the satellite image at the same time; Determine the center point of the small area of ​​the aerial image in the overlay image as the center, perform rotation and scaling operations on it to obtain a first transformation area, and calculate a first transformation matrix from the small area of ​​the aerial image to the first transformation area; Performing a random perturbation operation on the first transformation region to obtain a second transformation region, and calculating a second transformation matrix from the first transformation region to the second transformation region; Obtaining a minimum rectangle enclosing the second transformation region as a third transformation region, and calculating a third transformation matrix from the second transformation region to the third transformation region; Performing matrix multiplication on the first transformation matrix, the second transformation matrix, and the third transformation matrix to obtain a homography transformation matrix from a small area of ​​the aerial image in the overlay image to the third transformation area; wherein the homography transformation matrix is ​​used to map feature points on the satellite image to corresponding positions in the third transformation area; All transformed image pairs and the corresponding homography transformation matrices are organized into a coarsely labeled dataset.

3. The remote sensing image matching method based on meta-learning according to claim 1, characterized in that: In the convolutional feature extraction module of the feature pairing network, a convolutional neural network is used to extract multi-scale features from the input image pair, its inductive bias is used to capture the local features of the image, and a feature pyramid network is introduced to enhance the multi-scale feature extraction capability, thereby extracting a feature map; The extracted feature map is position-encoded in the position encoding module of the feature pairing network, and two-dimensional expansion is used to provide unique position information for each feature point in the feature map in a sinusoidal format. The position encoding is integrated with the feature map so that the feature point contains position information, thereby enhancing the ability of the feature pairing network to generate matches in indistinguishable areas; wherein the indistinguishable areas are areas with similar features but different positions; The attention module of the feature pairing network includes a self-attention layer and a cross-attention layer; different feature points on the same image are input into the self-attention layer to distinguish different areas within the image; corresponding feature points in the image pair are input into the cross-attention layer to explore the relationship between the feature points of the two images.

4. The remote sensing image matching method based on meta-learning according to claim 1, characterized in that: The input layer of the saliency judgment network is used to receive two sets of features processed by the feature pairing network and the loss value between the two sets of features; wherein the two sets of features are feature sets including multiple feature points extracted from the image pair; Design a fusion mechanism to fuse the two sets of features and the loss values ​​between them to generate fused features; Construct a multi-layer perceptron network to process the fusion features and obtain the perceptual features; The output layer converts the perceptual features into a saliency score for each pair of feature points for output.

5. The remote sensing image matching method based on meta-learning according to claim 1, characterized in that: The loss function of the feature pairing network is set to guide its prediction of the matching relationship between corresponding features in the image pair. The expression of the loss function of the feature pairing network is as follows: in, The feature pairing network matches pairs during training. The loss value, w is the parameter of the feature pairing network, Output confidence matrix P for feature pairing network c The value of the i-th row and j-th column on represents the confidence that the i-th point on the first image matches the j-th point on the second image. λ is a hyperparameter. is the confidence that the i-th point on the first image does not match the j-th point on the second image.

6. The remote sensing image matching method based on meta-learning according to claim 5, characterized in that: The loss function of the saliency judgment network is set to obtain its ability to identify regions that have a significant impact on feature matching performance. The expression of the loss function of the saliency judgment network is as follows: in, Matching pairs for feature pairing networks The loss value is used as the input of the saliency judgment network, θ is the parameter of the saliency judgment network, s ij The output of the saliency judgment network is a pair of matching pairs Saliency score, V(.,θ) is the saliency judgment network.

7. The remote sensing image matching method based on meta-learning according to claim 6, characterized in that: The overall loss function is set by combining the loss value of the feature pairing network and the loss value of the significance judgment network, and optimizing their performance at the same time. The expression of the overall loss function is as follows: in, is a set of real matching pairs, which comes from the annotations of real point pairs in the metadata set, where each element is a matching pair if and is the pairing point, the position value is 1, otherwise it is 0, V(.,θ) is the saliency judgment network, is the loss value of the feature pairing network, L is the value of the total loss function, and V(.,θ) is the significance judgment network.

8. The remote sensing image matching method based on meta-learning according to claim 1, characterized in that: The two-layer optimization step is iteratively performed through different data sets, including: Using the parameters of the current saliency judgment network of the coarsely annotated dataset, the parameters of the feature pairing network are iteratively optimized through the gradient descent method to obtain the approximate optimal parameter values ​​of the feature pairing network; Using the approximate optimal parameter values ​​of the metadata dataset and the feature pairing network, the parameters of the saliency judgment network are iteratively optimized through the gradient descent method to obtain the approximate optimal parameter values ​​of the saliency judgment network.

9. The remote sensing image matching method based on meta-learning according to claim 8, characterized in that: When the fixed optimal parameter value of the saliency judgment network is obtained, the expression of the optimization process of the parameters of the feature pairing network is: * (θ * ) = argmin w L C (w,θ * ), where w is the parameter of the feature pairing network, θ * is the optimal parameter value of the saliency judgment network, L c (w,θ * ) is the optimal parameter value of the feature pairing network in the saliency judgment network, which is fixed to θ * The loss function for the coarsely labeled dataset when , argmin is the operator used to find the loss function that makes L C (w,θ * )The value of the parameter w of the minimum feature pairing network, w * (θ * ) is the result of the optimization process of the parameters of the feature pairing network. When the optimal parameter value of the saliency judgment network is fixed to θ * When , the optimal parameter value of the feature pairing network is w * ; When the fixed optimal parameter value of the feature pairing network is obtained, the expression of the optimization process of the parameters of the significance judgment network is: θ * =argmin θ L meta (w * (θ)); where θ * When the optimal parameter value of the feature pairing network is fixed to w * When , the optimal parameter value of the saliency judgment network, θ is the parameter of the saliency judgment network, w * is the optimal parameter value of the feature pairing network, L meta (w * (θ)) is the optimal parameter value of the feature pairing network and is fixed to w * , the meta-loss function of the saliency judgment network on the meta-dataset, and argmin is the operator used to find the value that makes L meta (w * (θ)) is the value of the smallest parameter θ; where the expression of the meta-loss function is: Among them, L meta (w * (θ)) is the optimal parameter value of the feature pairing network and is fixed to w * , the meta-loss function of the saliency judgment network on the metadata dataset, M is the total number of matching pairs in the metadata set, is a pair of matches in the metadata set, To match this pair And the optimal parameter value w of the feature pairing network * The meta-loss function value of , V(.,θ) is the saliency judgment network, and meta data is the meta data set.

10. A remote sensing image matching device based on meta-learning, characterized in that: include: A metadata dataset construction module is used to select a first preset number of image pairs from an existing dataset, and perform paired annotation of salient area points on each pair of images, so that each pair of images has a second preset number of matching pairs, so as to construct a metadata dataset for meta-learning; wherein the image pairs include aerial images and satellite images of the same area; A module for constructing a coarsely annotated dataset is used to select all image pairs from the existing dataset and perform a transformation operation on one image in each image pair to construct a coarsely annotated dataset; A network module is constructed to construct a feature pairing network and a saliency judgment network; the feature pairing network includes a convolutional feature extraction module, a position encoding module and an attention module, and the network is used to extract similar features from the image pairs to be matched and pair them; the saliency judgment network includes a feature input, a multi-layer perceptron network and a saliency output, and the network is used to receive two sets of features processed by the feature pairing network and output a saliency score for each feature point; a loss function module is set to set a loss function of the feature pairing network to guide it to predict the matching relationship between corresponding features in the image pairs; a loss function of the saliency judgment network is set to obtain its ability to identify areas that have a significant impact on feature matching performance; an overall loss function is set to optimize the performance of the feature pairing network and the saliency judgment network by combining their loss values; The module for iteratively executing the two-layer optimization steps is used to iteratively execute the two-layer optimization steps through different data sets until the approximate optimal parameter values ​​of the feature pairing network and the saliency judgment network no longer change significantly or reach a preset number of iterations, so as to obtain the corresponding optimal parameter values; wherein the parameters of the feature pairing network are optimized through the coarsely annotated data set, and the parameters of the saliency judgment network are optimized through the metadata data set.

Images