An image local feature matching method and system based on a deep attention network

Abstract

The application provides a kind of based on deep attention network's image local feature matching method and system, comprising: S1, for given image pair, based on repeatability and reliability, random sampling image key point;S2, based on key point, the corresponding multi-scale local image block is extracted, and using neural network is reduced dimension;S3, the feature of local image block is matched, and local connection graph is constructed;S4, key point and its matching position information and local feature are merged, and the feature after merging is regarded as a basic unit of information processing;S5, on local connection graph, using basic unit is based on attention information aggregation, and local feature is updated;S6, local feature is input into deep attention network, and repeatedly executes prediction-a few attention layers, obtains the feature after final update, is used for final matching prediction.The application has efficient, flexible and easy-to-use image local feature matching characteristics.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically, it relates to an image local feature matching method and system based on deep attention networks. Background Technology

[0002] Image local feature matching, which involves finding correspondences between images, is an important component of many computer vision techniques, such as 3D reconstruction, SLAM, and SfM. A typical matching framework includes feature detection, feature description, and feature matching. Among these, the feature matching method has a significant impact on the final matching performance. Simple matching methods, such as nearest neighbor matching (NN), are easily affected by noise, so more complex matching and filtering methods are constantly being proposed.

[0003] Recent methods have attempted to use attention-based graph neural networks (GNNs) for matching (see Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In CVPR, 2020; Yan Shi, Jun-Xiong Cai, Yoli Shavit, Tai-Jiang Mu, Wensen Feng, and Kai Zhang. Clustergnn: Cluster-based coarse-to-fine graph neural network for efficient feature matching. In CVPR, 2022.). These methods often use cross-attention and cross-attention to model contextual relationships. However, since the complexity of the original attention is quadratic with the length of the input sequence, the computational complexity becomes very high when the number of keypoints is large. Therefore, some methods use more efficient attention mechanisms or reduce computational complexity through keypoint pruning, early stopping of inference, etc. Nevertheless, they still face a trade-off between matching complexity and matching performance.

[0004] Furthermore, existing methods are typically trained on specific keypoints and descriptors (see David GLowe. Distinctive image features from scale-invariant keypoints. IJCV, 2004; Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabi-novich. Superpoint: Self-supervised interest point detection and description. In CVPR, 2018). This presents two problems. First, training with specific keypoints and descriptors adapts the network to the distribution of specific keypoints, limiting the reusability of these methods when new, better keypoints emerge. Second, the existence of specific keypoints and descriptors may limit the network's performance. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this invention is to provide an image local feature matching method and system based on deep attention networks.

[0006] According to one aspect of the present invention, an image local feature matching method based on a deep attention network is provided, comprising:

[0007] S1, For a given pair of images, randomly sample key points from both images based on repeatability and reliability;

[0008] S2, Based on the key points, extract the corresponding multi-scale local image patches, and use a neural network to reduce the dimensionality to obtain the features of the local image patches;

[0009] S3, use the features of the extracted local image patch to perform feature matching, predict a matching item for each feature of the two images, and construct a local connectivity graph so that each key point is only connected to its nearest k matching key points.

[0010] S4, key points and their matching location information and local features are merged, and the merged features are regarded as a basic unit of information processing.

[0011] S5, on the local connectivity graph, use the basic units to perform attention-based information aggregation and update local features;

[0012] S6. Input the local features into a deep attention network and repeatedly perform prediction-several attention layers to obtain the final updated features, which are then used for final matching prediction.

[0013] Preferably, in S1, the reliability is defined as whether the sampling of the key points is meaningful; the repeatability is defined as whether the key points are co-visible in two images.

[0014] Preferably, in S2, based on the key points, corresponding multi-scale local image patches are extracted, and dimensionality reduction is performed using a neural network to extract the features of the local image patches, including:

[0015] For a given image and keypoints, set the scale number as W and the downsampling factor as r;

[0016] The image is downsampled W-1 times to obtain images at W scales, and the key points are also scaled proportionally.

[0017] At each scale, extract S×S local image patches around each keypoint, i.e., each keypoint corresponds to W S×S image patches. Flatten each image patch, and denote the i-th layer as p. i ;

[0018] Dimensionality reduction is performed using the same projection layer on S×S image patches of different scales:

[0019] d i =MLP1(p i )=L(LayerNorm(ReLU(L(p i ))))

[0020] Where L(·) represents a linear layer, ReLU(·) is the ReLU activation function, and LayerNorm(·) represents layer normalization;

[0021] Features at different scales are stitched together and projected using another network to obtain the features of local image patches corresponding to key points.

[0022] d = MLP2([d0,d1…d) W-1 ])=L(LaterNorm(ReLU(L([d0,d1…d W-1 ])))).

[0023] Preferably, in S3, feature matching is performed using the features of the extracted local image patches, predicting a matching term for each feature of the two images, including:

[0024] Define d A Let d be the set of all local feature points in graph A. B This is the set of all local feature points in Figure B;

[0025] d A and d B Perform pairwise related inner product operations:

[0026]

[0027] Applying the Softmax descriptor to M(i,j) in both directions yields the confidence matrix:

[0028] P(i,j)=softmax(M(i,·))j·softmax(M(·,j)) i

[0029] Find the maximum value in each column of the confidence matrix P(i,j) to obtain the predicted value of the element representing the row.

[0030] Find the maximum value in each row of the confidence matrix P(i,j) to obtain the predicted value of the column representative element.

[0031] Preferably, S3, constructing a local connectivity graph such that each keypoint is connected only to its k nearest matching keypoints, including:

[0032] For a given key point, calculate its distance to all other key points and remove points that are not confident in their matching.

[0033] Select the k nearest key points, including itself, as the objects of attention aggregation;

[0034] All key points are identified as objects of attention aggregation, and sparse local connectivity graphs are constructed in graphs A and B.

[0035] Preferably, in step S4, key points and their matching location information and local features are merged, including:

[0036] Normalize the keypoints, meaning that the mean of each dimension of the keypoint set is 0 and the variance is 1, to obtain the normalized position coordinates x. A ,x B ;

[0037] set up These are the local features of image A and the local features of image B, respectively. for If the predicted value is obtained, then feature merging is specifically as follows:

[0038]

[0039]

[0040]

[0041] in Represents the local features after fusion, concat(·) represents concatenation, MLP p (·) represents a projection layer:

[0042]

[0043] Where L(·) represents a linear layer, ReLU(·) is the ReLU activation function, and LayerNorm(·) represents layer normalization.

[0044] Preferably, in S5, on the local connectivity graph, attention-based information aggregation is performed using the basic units to update local features, including:

[0045] If we define the method for constructing the local graph as SA, then the local feature update is as follows:

[0046]

[0047] Where Attn(query, key, value) represents a standard attention aggregation, constructing a standard attention layer, specifically:

[0048]

[0049]

[0050]

[0051] in and For intermediate variables, L(·) represents a linear layer, LayerNorm(·) represents layer normalization, concat(·,·) represents concatenation, FFN represents a feedforward neural network, and MHA represents a multi-head attention mechanism. For updated local features.

[0052] Preferably, in S6, the deep attention network includes:

[0053] N p It consists of blocks, each block containing one prediction and multiple attention aggregations;

[0054] In each block, predictions are made and a local connectivity graph is constructed. The local connectivity graph is used in the next Na attention layers, and feature merging and attention aggregation are performed in each attention layer.

[0055] The features output by the last attention layer are used for the final matching, and the matches themselves are filtered using a threshold.

[0056] Preferably, it further includes supervising each match using logarithmic loss, including:

[0057] The logarithmic loss function is:

[0058]

[0059] Where M p and M g Let M be the set of positive labels and the set of negative labels, respectively. N is the total number of predictions, P represents the confidence matrix, and n represents the matrix obtained from the nth prediction. i, j represent the element in the i-th row and j-th column. Since there are fewer positive labels than negative labels, supervising all negative samples would increase gradient calculation. Therefore, M is dynamically generated... g This means that only the maximum mismatch value in each row is monitored.

[0060] According to a second aspect of the present invention, an image local feature matching system based on a deep attention network is provided, comprising:

[0061] The sampling module, for a given pair of images, randomly samples key points from both images based on repeatability and reliability.

[0062] The local feature extraction module extracts corresponding multi-scale local image patches based on the key points, and uses a neural network to reduce the dimensionality to obtain the features of the local image patches.

[0063] The prediction module uses the features of the extracted local image patches to perform feature matching, predicts a match for each feature of the two images, and constructs a local connectivity graph so that each keypoint is connected only to its nearest k most likely matching keypoints.

[0064] The merging module merges key points and their matching location information and local features, treating the merged features as a basic unit of information processing.

[0065] The attention aggregation module performs attention-based information aggregation on the local connectivity graph using the basic units to update local features;

[0066] The matching module inputs the local features into a deep attention network, repeatedly performs prediction-several attention layers, and obtains the final updated features for final matching prediction.

[0067] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0068] In embodiments of this invention, an image local feature matching method and system based on a deep attention network are proposed, which is an efficient, flexible, and easy-to-use image local feature matching approach. During training, randomly sampled keypoints and multi-scale local image patches are used to replace specific keypoints; during inference, random keypoints can be replaced with specific keypoints, thus ensuring the network's flexibility and repeatability for new keypoints. Furthermore, it guarantees generalization performance and matching effects for different keypoints, exhibiting greater flexibility. Simultaneously, an improved deep attention architecture is employed, resulting in lower complexity, which greatly improves the network's ease of use and wide applicability; using matching rather than local features as the basic unit for information aggregation, and aggregating features within a dynamic local connectivity graph, not only significantly improves the matching speed compared to global attention but also enhances matching performance. Attached Figure Description

[0069] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0070] Figure 1 This is a flowchart of an image local feature matching method based on a deep attention network in one embodiment of the present invention;

[0071] Figure 2 This is a block diagram of an image local feature matching method based on a deep attention network in a preferred embodiment of the present invention;

[0072] Figure 3 This is a schematic diagram of the sampling results of key points in a preferred embodiment of the present invention;

[0073] Figure 4 This is a schematic diagram of the matching results in a preferred embodiment of the present invention. Detailed Implementation

[0074] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings: This embodiment is implemented based on the technical solution of the present invention, and provides detailed implementation methods and specific operation processes, but the protection scope of the present invention is not limited to the following embodiments.

[0075] See Figure 1 and Figure 2 In one embodiment of the present invention, an image local feature matching method based on a deep attention network is provided, the steps of which include:

[0076] S1, For a given pair of images, randomly sample key points from both images based on repeatability and reliability;

[0077] S2, based on key points, extracts corresponding multi-scale local image patches, and uses a neural network to reduce dimensionality to obtain the features of the local image patches;

[0078] S3, use the features of the extracted local image patches to perform feature matching, predict a match for each feature of the two images, and construct a local connectivity graph so that each keypoint is only connected to its nearest k most confident matching keypoints;

[0079] S4, key points and their matching location information and local features are merged, and the merged features are regarded as a basic unit of information processing.

[0080] S5, on the local connectivity graph, use basic units to perform attention-based information aggregation and update local features;

[0081] S6. Input the local features into the deep attention network and repeatedly perform prediction-several attention layers to obtain the final updated features, which are then used for the final matching prediction.

[0082] The above embodiments leverage the advantages of existing deep attention networks, proposing to replace specific keypoints and descriptors with randomly sampled keypoints and multi-scale image patches for training the feature matching network; proposing feature merging, using the matching itself rather than local features as the basic unit of information processing to better model geometric relationships; and proposing to build a dynamic local connectivity graph to aggregate information between adjacent feature points with matching confidence. The matching performance is continuously improved through a successive prediction-attention module. This method can quickly match multiple feature points without retraining, provides a high-quality matching set, and performs excellently on various tasks.

[0083] In a preferred embodiment of the present invention, step S1 is implemented, which involves sampling key points.

[0084] During training, for a given image pair, keypoints are randomly sampled from both images to ensure repeatability and reliability. During inference, these random keypoints can be replaced with existing keypoints to ensure matching performance and stability. In this embodiment, the total number of keypoints sampled for each image during training is 4096. The specific steps are as follows:

[0085] Step S11, Sampling based on reliability: This property mainly defines whether the sampling of key points is meaningful, that is, whether the key points fall into scenes that cannot be matched at all, such as the sky, waves, etc. In this embodiment, the training data is MegaDepth (see Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In CVPR, 2018.), which has a depth map. It can be determined by whether the depth estimation results in the dataset exist.

[0086] Step S12: Sampling for repeatability: Define whether keypoints are co-visible between two images. If the number of keypoints sampled in an image is small, there may be fewer correspondences. In this case, the correspondences between the two images can be separated to obtain a portion of the keypoints, thus enhancing the repeatability of the keypoints. In this embodiment, we take a maximum of 10% of the keypoints for matching and apply a small amount of Gaussian noise to one side of the matching.

[0087] This embodiment proposes using randomly sampled keypoints and multi-scale local image patches to replace specific keypoints during training. During inference, random keypoints can be replaced with specific keypoints, which on the one hand allows for approximate end-to-end training and enhances matching performance; on the other hand, it enhances the flexibility of keypoint selection.

[0088] In a preferred embodiment of the present invention, step S2 is implemented by extracting the corresponding multi-scale local image patches from the key points obtained in the first step, and using a neural network to perform dimensionality reduction to extract the features of the local image patches.

[0089] The specific steps are as follows:

[0090] Step S21: For a given image and keypoints, let the number of scales be W and the downsampling coefficient be r. First, downsample the image W-1 times to obtain images at W scales. The keypoints are also scaled proportionally. At each scale, extract local image patches of size S×S around each keypoint, i.e., each keypoint corresponds to W S×S image patches. Flatten each image patch, and denote the i-th layer as p. i In some specific embodiments, W=4, downsampling coefficient r=2, and S=21.

[0091] Step S22: In order to fuse and extract features from images of different scales, firstly, dimensionality reduction is performed on S×S image patches of different scales using the same projection layer:

[0092] d i =MLP1(p i)=L(LayerNorm(ReLU(L(p i ))))

[0093] Where L(·) represents a linear layer, ReLU(·) is the ReLU activation function, and layerNorm(·) represents layer normalization. In some specific embodiments, d i The dimension is 256.

[0094] Step S23: Concatenate features at different scales and project them using another network:

[0095] d = MLP2([d0,d1…d) W-1 ])=L(LayerNorm(ReLU(L([d0,d1…d W-1 ]))))

[0096] In some specific embodiments, the projected feature dimension is 256.

[0097] In a preferred embodiment of the present invention, step S3 is implemented as follows: Feature matching is performed using the extracted features to predict a matching term for each feature of the two images; a local connectivity graph is constructed such that each keypoint is connected only to its k nearest matching keypoints. The specific steps are as follows:

[0098] Step S31: Assume the feature to be matched is d A and d B First, correlate these features pairwise:

[0099]

[0100] Here, <·> represents the inner product operation. The Softmax descriptor is applied to it in both directions:

[0101] P(i,j)=softmax(M(i,·)) j ·softmax(M(·,j)) i

[0102] Finding the maximum value in each column yields the predicted value for the row-representing element, and finding the maximum value in each row yields the predicted value for the column-representing element.

[0103] Step S32: The local connectivity graph is constructed as follows: For a given point, first calculate its distance to all feature points, then remove points that are not confident in their matching (confidence in matching is defined as whether the matching corresponding to a keypoint satisfies the nearest neighbor constraint). Then, take the k nearest keypoints as the objects of attention aggregation (including itself), and apply this operation to each point to construct a sparse local connectivity graph. In some specific embodiments, k is 8.

[0104] In a preferred embodiment of the present invention, step S4 is implemented by merging the key points and their matching location information and local features, and treating the matching as a basic unit of information processing.

[0105] The specific steps are as follows:

[0106] Step S41: First, normalize the feature points, that is, make the mean of each dimension of the feature point set 0 and the variance 1, to obtain the normalized position coordinates x. A ,x B

[0107] Step S42, set These are the local features of image A and the local features of image B, respectively. for The predicted value. Then the feature merging method is:

[0108]

[0109]

[0110]

[0111] in Represents the local features after fusion; concat represents splicing; MLP p For a projection layer:

[0112]

[0113] Where L(·) represents a linear layer, ReLU(·) is the ReLU activation function, and LayerNorm(·) represents layer normalization. In this embodiment, and The projected dimensions are all 256, and and Consistent.

[0114] The above embodiments propose using matching rather than local features as the basic unit for information aggregation, thereby enabling better modeling of matching geometry and achieving better matching results;

[0115] In a preferred embodiment of the present invention, step S5 involves performing attention-based information aggregation on the local connectivity graph to update local features. The specific steps are as follows:

[0116] Step S51: Assuming the local graph construction method is SA, the local feature update form is:

[0117]

[0118] Attn(query, key, value) represents standard attention aggregation.

[0119] Step S52: Construct a standard attention layer in the following form:

[0120]

[0121]

[0122]

[0123] in and For intermediate variables, L(·) represents a linear layer, LayerNorm(·) represents layer normalization, concat(·,·) represents concatenation, FFN represents a feedforward neural network, and MHA represents a multi-head form of attention.

[0124] In a preferred embodiment of the present invention, step S6 is implemented by repeatedly performing feature updates of prediction-multi-layer attention (without merging position information again) to construct a deep attention network, and using the finally updated features for matching prediction, and using threshold filtering for matching.

[0125] Among them, the deep attention network consists of N p It consists of blocks, each containing one prediction and multiple attention aggregations.

[0126] In each block, prediction is first performed and a local connectivity graph is constructed (same as S3 above). The local connectivity graph can be used in the following N... a The attention layers are used, and each attention layer performs feature merging (same as S4 above, but the merged content does not include location information) and attention aggregation (same as S5 above). In some specific embodiments, N p =3,N a =3.

[0127] Step S63: The features output from the last layer are used for the final matching. The matching method is the same as the prediction in step three, and the matching itself is filtered by a threshold.

[0128] The above embodiments improve the attention network by aggregating information within a dynamic local connectivity graph, which greatly improves the matching speed relative to global attention and also enhances the matching performance.

[0129] The above embodiments use a prediction-several attention layers as the basic network modules, and gradually improve the matching quality through continuous prediction and information aggregation.

[0130] To achieve better matching accuracy, in a preferred embodiment of this invention, logarithmic loss is used to supervise each prediction. The network's loss function is:

[0131]

[0132] Where M p and M g Let M be the set of positive labels and the set of negative labels, respectively, and N be the total number of predictions. It's worth noting that there are significantly fewer positive labels than negative labels. Supervising all negative samples would lead to a sharp increase in gradient computation. Therefore, we dynamically generate M... g That is, by only monitoring the maximum mismatch value in each row.

[0133] Based on the same inventive concept, other embodiments of the present invention provide an image local feature matching system based on a deep attention network, comprising:

[0134] The sampling module, for a given pair of images, randomly samples key points from both images based on repeatability and reliability.

[0135] The local feature extraction module extracts corresponding multi-scale local image patches based on key points and uses a neural network to reduce the dimensionality of the local image patches to obtain their features.

[0136] The prediction module uses the features of the extracted local image patches to perform feature matching, predicts a match for each feature of the two images, and constructs a local connectivity graph so that each keypoint is only connected to its nearest k most confident matching keypoints.

[0137] The merging module merges key points and their matching location information and local features, treating the merged features as a basic unit of information processing.

[0138] The attention aggregation module uses basic units on the local connectivity graph to perform attention-based information aggregation and update local features;

[0139] The matching module inputs local features into a deep attention network, repeatedly performs prediction through several attention layers, and obtains the final updated features for final matching prediction.

[0140] The specific implementation techniques of each module / unit in the above examples of the present invention can be referred to the steps of the image local feature matching method based on deep attention network in the above embodiments, and will not be repeated here.

[0141] To verify the technical effects of the above embodiments, in a specific embodiment of the present invention, the method in the above embodiments is applied to a specific dataset. The images used are from the datasets MegaDepth and YFCC100m (see Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internetphotos. In CVPR, 2018.; Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016.) as outdoor relative pose estimation datasets. ScanNet and SUN3D (see Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nieβner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.; Jianxiong Xiao, Andrew Owens, and Antonio Torralba. Sun3d: A database of big spaces reconstructed using SFM and object labels. In ICCV, pages 1625–1632, 2013.) as a dataset for indoor relative pose estimation.Meanwhile, the Aachen day-night dataset (see Torsten Sattler, Will Maddern, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, Josef Sivic, et al. Benchmarking 6d of outdoor visual localization in changing conditions. In CVPR, pages 8601–8610, 2018.) was used as the visual localization dataset.

[0142] Table 1 below shows the relative pose estimation results obtained on the MegaDepth1500 and YFCC100m datasets according to this embodiment. Both the MegaDepth1500 and YFCC100m datasets are outdoor test sets, containing significant viewpoint and illumination variations, and thus possess good representativeness. The method in this embodiment can report the relative pose estimation error AUC (see Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In CVPR, 2021). It can be seen that, as... Figure 3 As shown, the method of this embodiment of the invention can also provide high-quality matching results in complex scenarios and with different types of key points.

[0143] Table 1

[0144]

[0145] Table 2 below shows the relative pose estimation results obtained on the ScanNet and SUN3D datasets according to this embodiment. These two datasets are indoor scene datasets, which have problems such as sparse texture, increasing the difficulty of matching optimization. Figure 4 As shown, the method in this embodiment of the invention can still provide high-quality and stable matching results, demonstrating the powerful capabilities of the algorithm in this embodiment of the invention.

[0146] Table 2

[0147]

[0148] Table 3 below shows the visual localization results obtained on the Aachen day-night dataset according to this embodiment. This dataset contains considerable scale, viewpoint, and illumination variations, posing a significant challenge. The method in this embodiment also provides quite good results, demonstrating its powerful matching ability.

[0149] Table 3

[0150]

[0151] In summary, this invention uses randomly sampled keypoints and multi-scale local image patches to replace specific keypoints during training, and replaces random keypoints with specific keypoints during inference. This allows for approximate end-to-end training, enhancing matching performance, and also increases the flexibility of keypoint selection. Furthermore, this embodiment uses matching rather than local features as the basic unit for information aggregation, thereby better modeling the matching geometry and achieving better matching results. This embodiment also improves the attention network by aggregating features within a dynamic local connectivity graph, significantly improving matching speed and performance compared to global attention. This embodiment uses a prediction-several attention layers as the basic network module, gradually improving matching quality through continuous prediction and information aggregation. Experimental results on multiple datasets demonstrate that this method has powerful matching capabilities and broad practical value.

[0152] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A method for image local feature matching based on deep attention networks, characterized in that, include S1, For a given pair of images, randomly sample key points from both images based on repeatability and reliability; S2, Based on the key points, extract the corresponding multi-scale local image patches, and use a neural network to reduce the dimensionality to obtain the features of the local image patches; S3, use the features of the extracted local image patch to perform feature matching, predict a matching item for each feature of the two images, and construct a local connectivity graph so that each key point is only connected to its nearest k matching key points. S4, key points and their matching location information and local features are merged, and the merged features are regarded as a basic unit of information processing. S5, on the local connectivity graph, use the basic units to perform attention-based information aggregation and update local features; S6, input the local features into a deep attention network, repeatedly perform prediction-several attention layers to obtain the final updated features, which are used for final matching prediction; In S5, attention-based information aggregation is performed on the local connectivity graph using the basic units to update local features, including: The method for constructing a local graph is defined as follows: Then, the local features are updated, specifically as follows: ； in This represents the standard aggregation of attention, constructing a standard attention layer, specifically as follows: ； ； ； in and As an intermediate variable, Represents a linear layer. Representative level normalization, Represents splicing, It is a feedforward neural network. For a multi-headed form of attention, For updated local features; In S6, the deep attention network includes: It consists of blocks, each block containing one prediction and multiple attention aggregations; In each block, predictions are made and a local connectivity graph is constructed. The local connectivity graph is then used in the following... It is used in each attention layer, and each attention layer performs feature merging and attention aggregation; The features output by the last attention layer are used for the final matching, and the matches themselves are filtered using a threshold.

2. The image local feature matching method based on deep attention network according to claim 1, characterized in that, In S1, the reliability is defined as whether the sampling of key points is meaningful; the repeatability is defined as whether key points are co-visible between two images.

3. The image local feature matching method based on deep attention networks according to claim 1, characterized in that, In S2, based on the key points, corresponding multi-scale local image patches are extracted, and dimensionality reduction is performed using a neural network to extract the features of the local image patches, including: For a given image and keypoints, the scale is set to [number]. The downsampling coefficient is ; Downsampling the image Next, get The key points of the image are also scaled proportionally to the scale. Extract the area around each keypoint at each scale. A local image patch of a certain size, i.e., each keypoint corresponds to... indivual The image blocks are flattened, and the i-th layer is denoted as . ; For different scales Image patches are reduced in dimensionality using the same projection layer: ； in Represents a linear layer. It is the ReLU activation function. Representative layer normalization; Features at different scales are stitched together and projected using another network to obtain the features of local image patches corresponding to key points. 。 4. The image local feature matching method based on deep attention networks according to claim 1, characterized in that, In S3, feature matching is performed using the extracted features of the local image patch, predicting a matching term for each feature of the two images, including: definition Let A be the set of all local feature points in graph A. This is the set of all local feature points in Figure B; Will and Perform pairwise related inner product operations: ； right Applying the Softmax descriptor in both directions yields the confidence matrix: ； In the confidence matrix Find the maximum value in each column to obtain the predicted value of the element representing the row; In the confidence matrix Find the maximum value in each row to obtain the predicted value of the element represented by the column.

5. The image local feature matching method based on deep attention networks according to claim 4, characterized in that, S3, construct a local connectivity graph such that each keypoint is connected only to its k nearest matching keypoints, including: For a given key point, calculate its distance to all other key points and remove points that are not confident in their matching. Select the k nearest key points, including itself, as the objects of attention aggregation; All key points are identified as objects of attention aggregation, and sparse local connectivity graphs are constructed in graphs A and B.

6. The image local feature matching method based on deep attention network according to claim 1, characterized in that, S4, key points and their matching location information and local features are merged, including: Normalize the keypoints, meaning that the mean of each dimension of the keypoint set is 0 and the variance is 1, to obtain the normalized position coordinates. ; set up These are the local features of image A and the local features of image B, respectively. for If the predicted value is obtained, then feature merging is specifically as follows: ； ； ； in , Represents the local features after fusion. Represents splicing, For a projection layer: ； in Represents a linear layer. It is the ReLU activation function. Representative level normalization.

7. The image local feature matching method based on deep attention network according to claim 1, characterized in that, This also includes supervising each match using logarithmic loss, including: The logarithmic loss function is: ； in and They are respectively the set of positive labels and the set of negative labels. To predict the total number, Let represent the confidence matrix, n represent the matrix obtained from the nth prediction, and i, j represent the element in the i-th row and j-th column. Since there are fewer positive labels than negative labels, supervising all negative samples would increase gradient calculation; therefore, dynamic generation is used. This means that only the maximum mismatch value in each row is monitored.

8. An image local feature matching system based on a deep attention network, used to implement the method of claim 1, characterized in that, include The sampling module, for a given pair of images, randomly samples key points from both images based on repeatability and reliability. The local feature extraction module extracts corresponding multi-scale local image patches based on the key points, and uses a neural network to reduce the dimensionality to obtain the features of the local image patches. The prediction module uses the features of the extracted local image patches to perform feature matching, predicts a match for each feature of the two images, and constructs a local connectivity graph so that each keypoint is connected only to its nearest k most likely matching keypoints. The merging module merges key points and their matching location information and local features, treating the merged features as a basic unit of information processing. The attention aggregation module performs attention-based information aggregation on the local connectivity graph using the basic units to update local features; The matching module inputs the local features into a deep attention network, repeatedly performs prediction-several attention layers, and obtains the final updated features for final matching prediction.

Images