A local feature extraction method

Abstract

This invention belongs to the field of computer vision technology and discloses a local feature extraction method. A novel local feature learning network framework is established, using a fully convolutional network L2-Net as the backbone. Features from the spatial structure enhancement module are extracted and fused into the intermediate and deep layers of the backbone network. A dataset with depth information is used as the training set. The correspondence between two images is obtained through the inherent depth information of the dataset, serving as the ground truth for local feature discriminative and robust training. The novel local feature learning network framework outputs feature heatmaps for discriminative training and feature detection heatmaps and local feature descriptors for robust training. This invention balances robustness and discriminability; the network still contains rich structural information in its deep layers, thereby improving discriminability. A joint reinforcement learning method is used to ultimately obtain better local feature descriptors. Finally, this method was evaluated on a feature matching task and achieved state-of-the-art results.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for local feature extraction. Background Technology

[0002] Compared to global features, local feature descriptors are widely used in various vision tasks, such as Simultaneous Localization and Mapping (SLAM), visual localization, and feature matching, due to their stronger resistance to changes in lighting and viewpoint. Currently, local feature extraction methods are mainly divided into two categories: those based on traditional algorithms and those based on deep learning. Hand-designed feature extraction methods often fail to perform well when faced with drastic changes in the external environment due to their inherent limitations. In contrast, deep learning-based local feature extraction methods demonstrate a stronger advantage in handling appearance changes. Therefore, deep learning-based local feature extraction methods have become the mainstream research trend.

[0003] The paper "Advances in Neural Information Processing Systems, 33:14254–14265, 2020" proposes a feature extraction method based on reinforcement learning. This method abandons traditional supervised learning methods and introduces a reinforcement learning strategy to interactively optimize local features, achieving good performance. Although it has strong robustness against changes in lighting and viewpoint, discriminative ability is also a very important performance characteristic, which this method does not consider. The paper "In Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition, pages 15838–15848, 2022" presents a decoupled, weakly supervised feature extraction method. It utilizes a line-to-window search strategy to fully leverage the geometric information of camera pose, reducing the search space and thus improving feature accuracy. Although it does not require strong labels for training, it fails to achieve good performance when resisting environmental changes. Summary of the Invention

[0004] To address the problems existing in the background technology, this invention proposes a local feature extraction method that can simultaneously ensure the robustness and discriminative power of feature extraction.

[0005] The technical solution of this invention is as follows: A local feature extraction method, establishing a new local feature learning network framework; the new local feature learning network framework uses a fully convolutional network L2-Net as the backbone network, extracts features from the spatial structure enhancement module, and integrates them into the intermediate and deep layers of the backbone network; a dataset with depth information is used as the training set, and the correspondence between two images can be obtained through the depth information inherent in the dataset, the correspondence serving as the ground truth for robust and discriminative training of local features; the new local feature learning network framework outputs a feature heatmap for discriminative training and a feature detection heatmap and local feature descriptor for robust training; discriminative and robust training are performed respectively to optimize the parameters of the new local feature learning network framework.

[0006] The spatial structure enhancement module performs max pooling and average pooling operations on the shallow features extracted by the backbone network, then concatenates the pooling results, and finally performs convolution to obtain the final spatial structure enhancement features, which are then input into the intermediate and deep layers of the backbone network, respectively.

[0007] The discriminative training specifically involves performing a 1×1 convolution operation on both the intermediate layer features with added spatial structure enhancement features and the deep layer features with added spatial structure enhancement features, transforming the feature shapes from H×W×64 and H×W×128 to H×W×1 respectively. Then, a Softmax operation is performed on each of the two H×W×1 features to obtain S. l and S h The results are then weighted and summed to obtain the final confidence score map used for discriminative training. The specific calculation process is shown in Formula 1:

[0008]

[0009] Where, ω l and ω h Let ω represent the summation weights. l =1, ω h =2;

[0010] According to the confidence score graph Obtain the AP value and perform discriminative training based on the AP loss.

[0011] The AP value acquisition process is as follows: First, block-to-block matching is performed between the target image and the query image, and then the confidence score map is directly analyzed. The image is divided into blocks, and then matching is performed between the blocks; the size of the blocks depends on the window size. The matching order of the image blocks is obtained by calculating the Euclidean distance values. The set of correctly matched blocks is set as U. + Let the set of incorrect matches be U.- AP represents the average of the matching accuracy values ​​evaluated at the sorting position K of the match, as shown in the following formula:

[0012]

[0013]

[0014] Where 1[] represents a binary indicator function; x i Represents a set of blocks; if and only if U + Each image patch in the U is arranged in order. - AP reaches its optimal value before all image patches are processed;

[0015] To obtain discriminative local features, the final loss function is constructed using the confidence score map and AP value, as follows:

[0016]

[0017] In the formula, The confidence score map of the target image is represented by formula (1); to make the obtained features effective, we constrain the feature points through similarity loss, and the similarity calculation method is expressed as follows. in A feature confidence score map representing the target image. This represents the confidence score graph of the matched images, where C represents the set of correspondences between images I and I′, and q is a value in set C; f I and f I′ The feature descriptor is M(*,*), which represents the triplet margin ranking loss.

[0018] The image block is divided into 8×8 blocks.

[0019] The local feature descriptor is the final output of the deep features with added spatial structure enhancement features in the new local feature learning network framework, and then L2 regularization is applied. The feature detection heatmap is obtained by performing 1×1 convolution and then softmax processing on the final output of the deep features with added spatial structure enhancement features in the new local feature learning network framework.

[0020] The robust training specifically involves representing the detection and matching of feature points as probabilistic events, the outcome of which is determined by the output of the new local feature learning network framework; during the robust training process, the parameters are continuously updated through a reward function to obtain the final result.

[0021] The feature detection heatmap and local feature descriptors are used to roughly obtain the distribution of feature points through a non-maximum suppression strategy, denoted by P(x|θ):

[0022]

[0023] Based on the distribution of feature points, a matching distribution is obtained through cycle consistency matching. express:

[0024]

[0025] in, F represents the matching relationship between image A and image B. A F represents the location of the feature points extracted from image A. B m represents the location of the feature points extracted from image B. xy This indicates the position of the matching point between two images. The learning parameters used to represent the matching are shown; depth information is used as the truth value to determine whether the key points are correctly matched; a reward function is designed based on the matching result. If the distance between two points is less than the reprojection distance, a reward is given, indicating a correct match; otherwise, a penalty is imposed, indicating a failed match.

[0026] use Representing the reward function:

[0027]

[0028] Here, r includes 1 and -0.25, where r = 1 when a match is successful and r = -0.25 when a match fails;

[0029] Based on the probability distribution of feature points, the distribution of matching items, and the reward function, we can obtain the objective of robust training:

[0030]

[0031] in and This represents the expected value of feature matching and feature points;

[0032] The network parameters θ are updated by calculating the gradient of the above equation:

[0033]

[0034] in, This represents the probability distribution of the match. This represents the probability distribution of feature points in the original image. This represents the probability distribution of feature points in the matched image. Let R represent the gradient, R represent the reward function, and E represent the expectation.

[0035] The beneficial effects of this invention are as follows: Addressing the problems of poor robustness and weak discriminativeness of local features extracted in previous visual tasks, this invention proposes a novel network framework that enhances spatial structure information, enabling feature descriptors to maintain both robustness and discriminativeness. This invention introduces a spatial structure enhancement module, embedded into the network from shallow to deep layers, ensuring that the network retains rich structural information even at deeper levels, thereby improving discriminativeness. Furthermore, we design a novel loss function for discriminativeness, ultimately obtaining better local feature descriptors through joint reinforcement learning. Finally, our method was evaluated on a feature matching task and achieved state-of-the-art results. Attached Figure Description

[0036] Figure 1 This is the main flowchart of the technical solution of the present invention;

[0037] Figure 2 This is a flowchart of robust training. Detailed Implementation

[0038] Figure 1 This is the main flowchart of the technical solution of this invention. For example... Figure 1 As shown, the local feature extraction method proposed in this invention includes the following steps:

[0039] (1) Data preparation: The accuracy of local features depends on the selection of a suitable dataset. This invention uses the MegaDepth dataset with depth information as the training set. Depth information can help the model find the correspondence between two images, and this correspondence can be used as the ground truth for local feature training.

[0040] (2) Model Construction: This invention proposes a novel local feature learning network framework. This framework uses the fully convolutional network L2-Net as the backbone network. L2-Net is a network that does not require image downsampling, maintaining the same resolution throughout the network, which facilitates the learning of local features. By extracting features from the spatial structure enhancement module and fusing them into intermediate and deep layers, this ensures that the model's structural information is not lost as the number of convolutional layers increases. Finally, the output is a feature heatmap for discriminative training and a feature detection heatmap and feature descriptors for robust training. Details are as follows:

[0041] For local feature descriptors, they should not only be highly resistant to changes in appearance such as lighting and viewing angle, but also exhibit strong discriminative power when faced with images of similar structures. Based on the inherent characteristics of deep learning, shallow features contain more spatial structural information, while deeper features contain more semantic information. To combine the advantages of both, this invention utilizes the concept of spatial attention to enhance the network structure. Shallow features of the image are extracted to construct a spatial structure enhancement module, which is then inserted into the backbone network from shallow to deep. This module is constructed by performing max pooling and average pooling operations on the extracted shallow features, concatenating the pooling results, and finally performing a convolution operation to obtain the final spatial structure enhancement features.

[0042] The network output consists of two parts: a confidence score map used for discriminative training. This process involves performing a 1×1 convolution on the features for which spatial structure enhancement modules have been added, transforming the feature shapes from H×W×64 and H×W×128 to H×W×1. Then, a softmax operation is performed on each of the two H×W×1 features to obtain S... l and S h The results are then weighted and summed to obtain the final result. The specific calculation process is shown in Formula 1:

[0043]

[0044] Where, ω l and ω h The first two characters represent the summation weights, which we empirically set to 1 and 2 to allow deeper features to occupy a larger proportion. The second part consists of feature descriptors and detection heatmaps used for robust training. The feature descriptors are typically 128-dimensional; we directly apply L2 regularization to the network's final result to obtain the final feature descriptors. The feature detection heatmap is obtained by performing a 1×1 convolution on the network's final result followed by a softmax operation.

[0045] (3) Model Training

[0046] a) Discriminative Training. Unlike most learning-based local feature methods, the network in this invention outputs not only feature descriptors and detection heatmaps for robust training, but also a single-channel confidence map for learning discriminative power. The goal of this invention is to make the network pay more attention to regions with high confidence, meaning that smooth or featureless regions should receive less attention. This part is trained by obtaining the AP loss. To obtain the AP value, we first need to perform block-to-block matching between the target image and the query image. We only need to directly analyze the feature maps. The image is divided into blocks, and then matched between these blocks. The size of each block depends on the window size used for partitioning; in this invention, the blocks are divided into 8x8 blocks. The matching order of the image blocks is obtained by calculating the Euclidean distance. We define the set of correctly matched blocks as U. + Let U be the set of incorrect matches. - AP represents the average matching accuracy value evaluated at the sorting position K of the match, as shown in the following formula:

[0047]

[0048]

[0049] Where 1[] represents a binary indicator function. i Represents a set of blocks. U ∈ U. + Each image patch in the U is arranged in order. - AP reaches its optimal value before all image patches are processed.

[0050] To obtain discriminative local features, we construct the final loss function using the confidence score map and AP value output by the network, as follows:

[0051]

[0052] In the formula, The confidence score map represents the target image, calculated as shown in formula (1). We assume that a higher confidence score implies higher pixel saliency. Therefore, when AP and When both are relatively high, The smaller the value, the more discriminative the obtained features are. Furthermore, to ensure the effectiveness of the obtained features, we constrain the feature points using a similarity loss, the similarity calculation method of which is expressed as... in A feature confidence score map representing the target image. Let f represent the confidence score map of the matched images, C represent the set of correspondences between images I and I′, and q be the value in set C. I and f I′ The first part represents the feature descriptor, and the second part represents the triplet margin ranking loss. The above two parts constitute the entire discriminative loss.

[0053] b) Learning robustness.

[0054] This part of the invention introduces a reinforcement learning method, representing feature point detection and matching as probabilistic events, the outcome of which is determined by the network output. During network training, the parameters are continuously updated through a reward function to obtain the final result, which is the local feature we need. The entire process is as follows: Figure 2 As shown.

[0055] From the feature heatmap and descriptors output from the last layer of the network, we can roughly obtain the distribution of feature points using the non-maximum suppression (NMS) strategy, which we denote as P(x|θ):

[0056]

[0057] Based on the distribution of feature points, we can obtain the matching distribution through cycle consistency matching. express:

[0058]

[0059] in, F represents the matching relationship between image A and image B. A F represents the location of the feature points extracted from image A. B m represents the location of the feature points extracted from image B. xy This indicates the position of the matching point between two images. This represents the learning parameters used to represent the matching.

[0060] The network uses depth information as the ground truth, which allows us to determine whether keypoints are correctly matched. We design a reward function based on the matching results: if the distance between two points is less than the reprojection distance, we reward the match (correct matching); otherwise, we penalize the match (failure). Representing the reward function:

[0061]

[0062] Here, r includes 1 and -0.25. When a match is successful, r = 1, and when a match fails, r = -0.25.

[0063] Based on the probability distribution of feature points, the distribution of matching items, and the reward function, we can obtain the training objective:

[0064]

[0065] in and This represents the expected value of feature matching and feature points.

[0066] The network parameters θ are updated by calculating the gradient of the above equation:

[0067]

[0068] in, This represents the probability distribution of the match. This represents the probability distribution of feature points in the original image. This represents the probability distribution of feature points in the matched image. Let R represent the gradient, R represent the reward function, and E represent the expectation. The expectation cannot be calculated directly, but it can be approximated as a summation through sampling, ultimately yielding the training result.

[0069] The feature points of this invention balance distinguishability and repeatability. Furthermore, due to feature enhancement during the learning process, the learned features contain more structural information, resulting in stronger performance in subsequent visual tasks.

[0070] (3) Experiment: The local features of this invention are mainly applied in image matching and visual localization tasks. By extracting features from the original image and the query image, and then using a brute-force matching method to match the features of the two images, the performance of the extracted local features is verified by comparing them with the true values ​​of the two images.

[0071] To verify the effectiveness of the algorithm in long-term visual localization accuracy, this invention was tested on the Hpatches dataset. Images from the dataset were input into the network model to obtain the location and descriptor of keypoints in each image. Then, the descriptors of two randomly selected images were matched using the fastest brute-force matching method currently available. Finally, the matching results of all images were used to calculate the MMA value, where MMA represents the average matching accuracy. A higher MMA value indicates higher matching accuracy. According to the experimental results, the proposed algorithm achieved a matching accuracy MMA of 0.85 for multiple scenes under varying viewpoints and 0.71 for multiple scenes under varying illumination. The overall matching accuracy MMA reached 0.78.

Claims

1. A method for local feature extraction, characterized in that, A novel local feature learning network framework is established. This framework uses a fully convolutional network L2-Net as its backbone, extracting features from spatial structure enhancement modules and fusing them into the intermediate and deep layers of the backbone network. A dataset with depth information is used as the training set, and correspondences are obtained through the inherent depth information of the dataset. These correspondences serve as ground truth for local feature discriminative and robust training. The novel framework outputs a confidence score map for discriminative training and a feature detection heatmap and local feature descriptors for robust training. Discriminative and robust training are performed separately to optimize the parameters of the novel local feature learning network framework. The spatial structure enhancement module performs max pooling and average pooling operations on the shallow features extracted by the backbone network, then concatenates the pooling results, and finally performs convolution to obtain the final spatial structure enhancement features, which are then input into the middle and deep layers of the backbone network respectively. The discriminative training specifically involves performing a 1×1 convolution operation on both the intermediate layer features with added spatial structure enhancement features and the deep layer features with added spatial structure enhancement features, transforming the feature shapes from H×W×64 and H×W×128 to H×W×1 respectively. Then, a Softmax operation is performed on each of the two H×W×1 features to obtain... and The results are then weighted and summed to obtain the final confidence score map used for discriminative training. The specific calculation process is shown in Formula 1: (1) in, and They represent the summation weights, =1, =2; According to the confidence score graph Obtain the AP value and perform discriminative training based on the AP loss; The AP value acquisition process is as follows: First, block-to-block matching is performed between the target image and the query image, and then the confidence score map is directly analyzed. The image is divided into blocks, and then matching is performed between the blocks; the size of the blocks depends on the window size. The matching order of the image blocks is obtained by calculating the Euclidean distance values. The set of correctly matched blocks is set as... The set of incorrect matches is set as AP represents the average of the matching accuracy values ​​evaluated at the sorting position K of the match, as shown in the following formula: (2) (3) in, Represents a binary indicator function; Denotes a set of blocks; if and only if Each image patch in is arranged in AP reaches its optimal value before all image patches are processed; To obtain discriminative local features, the final loss function is constructed using the confidence score map and AP value, as follows: (4) In the formula, The confidence score map of the target image is represented by formula (1); to make the obtained features effective, the feature points are constrained by similarity loss, and the similarity calculation method is expressed as follows. ,in A feature confidence score map representing the target image. This represents the confidence score graph of the matched images, where C represents the image. and The set of correspondences between them q It is a value in set C; and Represents feature descriptors, This represents the sorting loss for triplet margins.

2. The local feature extraction method according to claim 1, characterized in that, The image block is divided into 8×8 blocks.

3. The local feature extraction method according to claim 1, characterized in that, The local feature descriptor is the final output of deep features with added spatial structure enhancement features in the new local feature learning network framework, and then... The feature detection heatmap is obtained by regularization; it is obtained by performing a 1×1 convolution and then softmax processing on the final output of the deep feature that adds spatial structure enhancement features to the new local feature learning network framework.

4. The local feature extraction method according to claim 3, characterized in that, The robust training specifically involves representing the detection and matching of feature points as probabilistic events, the outcome of which is determined by the output of the new local feature learning network framework; during the robust training process, the parameters are continuously updated through a reward function to obtain the final result. The feature detection heatmap and local feature descriptors roughly obtain the distribution of feature points using a non-maximum suppression strategy. express: (5) Based on the distribution of feature points, a matching distribution is obtained through cycle consistency matching. express: (6) in, This indicates the matching relationship between image A and image B. This indicates the location of the feature points extracted from image A. This indicates the location of the feature points extracted from image B. This indicates the position of the matching point between two images. The learning parameters used to represent the matching are shown; depth information is used as the truth value to determine whether the key points are correctly matched; a reward function is designed based on the matching result. If the distance between two points is less than the reprojection distance, a reward is given, indicating a correct match; otherwise, a penalty is imposed, indicating a failed match. use Representing the reward function: (7) in, Includes 1 and -0.25, when a match is successful. When a match fails ; Based on the probability distribution of feature points, the distribution of matching items, and the reward function, the objective of robust training can be obtained: (8) in and This represents the expected value of feature matching and feature points; The network parameters are updated by calculating the gradient of the above equation. : (9) in, This represents the probability distribution of the match. This represents the probability distribution of feature points in the original image. This represents the probability distribution of feature points in the matched image. Represents the gradient. R Represents the reward function. E It expresses expectation.

Images