Local feature extraction method

By introducing spatial structure enhancement modules and deep information training in the local feature extraction method, combining distinction and robustness training, the problem of insufficient local feature extraction robustness and distinction in the prior art is solved, and the visual task is realized The efficient feature extraction in the process achieved high matching accuracy under viewing angle and lighting changes.

CN119048770AActive Publication Date: 2024-11-29NORTHEASTERN UNIV CHINA

Patent Information

Application Number
CN202411143366.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-29
Estimated Expiration
2044-08-20

AI Technical Summary

Technical Problem

The existing local feature extraction methods show insufficient robustness when facing drastic changes in the external environment, and are not very differentiated, making it difficult to achieve excellent performance in visual tasks.

Method used

Using the full convolutional network L2-Net as the backbone network, the spatial structure enhancement module is introduced, and it is embedded in the network. Through deep information training features, combining distinctive and robust training, new loss functions and reward functions are designed to optimize the network. parameters to improve the distinction and robustness of features.

Benefits of technology

It realizes both robustness and distinction of feature extraction in visual tasks, improves the expressiveness of feature descriptors, and achieves high matching accuracy under viewing angle and lighting changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048770A_ABST
    Figure CN119048770A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of computer vision, and discloses a local feature extraction method. Establishing a new local feature learning network framework, taking a full convolutional network L2-Net as a backbone network, extracting features of a space structure enhancement module, and fusing the features to a middle layer and a deep layer of the backbone network; taking a data set with depth information as a training set, and obtaining a corresponding relation of the two images through the depth information of the data set, and taking the corresponding relation as a truth value of local feature distinguishing training and robustness training; the new local feature learning network framework outputs a feature heat map for discriminative training and a feature detection heat map and a local feature descriptor for robustness training. According to the method, robustness and discrimination are considered, and the network still contains rich structural information in a deep layer, so that the discrimination is improved, and a better local feature descriptor is finally obtained through a combined reinforcement learning method. Finally, according to the method, evaluation is carried out on a feature matching task, and an advanced result is obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a local feature extraction method. Background Art

[0002] Compared with global features, local feature descriptors are widely used in various visual tasks, such as simultaneous localization and mapping (SLAM), visual positioning, feature matching, etc., due to their stronger expressiveness against changes in illumination and perspective. At present, local feature extraction methods are mainly divided into two categories: one is the feature extraction method based on traditional algorithms, and the other is the feature extraction method based on deep learning. Due to its inherent limitations, the feature extraction method based on manual design often cannot show good performance when facing drastic changes in the external environment. The local feature extraction method based on deep learning shows stronger advantages in dealing with appearance changes. Therefore, the local feature method based on deep learning has become the mainstream research trend.

[0003] The paper "Advances in Neural Information Processing Systems, 33: 14254–14265, 2020" proposes a feature extraction method based on reinforcement learning. This method abandons the traditional supervised learning method, introduces a reinforcement learning strategy, optimizes local features in an interactive form, and achieves good performance. Although it has strong robustness in resisting changes in light and perspective, in addition to robustness, distinguishability is also a very important performance, which this method does not take into account. The paper "In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 15838–15848, 2022" is a weakly supervised feature extraction method based on decoupling, and through a line-to-window search strategy, it fully utilizes the geometric information of the camera pose, reduces the search space, and thus improves feature accuracy. Although strong labels are not required for training, good performance cannot be achieved when resisting environmental changes. Summary of the invention

[0004] In order to solve the problems existing in the above-mentioned background technology, the present invention proposes a local feature extraction method, which can simultaneously ensure the robustness and discrimination of feature extraction.

[0005] The technical scheme of the present invention is as follows: a local feature extraction method, establishing a new local feature learning network framework; the new local feature learning network framework uses a full convolutional network L2-Net as a backbone network, extracts the features of a spatial structure enhancement module, and fuses them into the middle layer and deep layer of the backbone network; a data set with depth information is used as a training set, and the correspondence between two images can be obtained through the depth information of the data set, and the correspondence is used as the true value of local feature robustness training and discriminative training; the new local feature learning network framework outputs a feature heat map for discriminative training and a feature detection heat map and a local feature descriptor for robust training; discriminative training and robust training are performed respectively, and the parameters of the new local feature learning network framework are optimized.

[0006] The spatial structure enhancement module performs maximum pooling and average pooling operations on the shallow features extracted by the backbone network, then concatenates the pooled results, and finally performs a convolution operation to obtain the final spatial structure enhancement features, which are input into the middle layer and deep layer of the backbone network respectively.

[0007] Specifically, the discriminative training is to perform a 1×1 convolution operation on the intermediate layer features with spatial structure enhancement features and the deep layer features with spatial structure enhancement features, respectively, so that the feature shapes are transformed from H×W×64 and H×W×128 to H×W×1, and then perform a Softmax operation on the two H×W×1 features to obtain S l and S h , and perform weighted summation on the results to obtain the final confidence score map for discriminative training The specific calculation process is shown in Formula 1:

[0008]

[0009] Among them, ω l and ω h Denote the summation weights, ω l =1,ω h =2;

[0010] According to the confidence score map Get the AP value and perform discriminative training based on the AP loss.

[0011] The AP value acquisition process is as follows: first, the target image and the query image are matched block by block, and the confidence score map is directly The blocks are divided and then matched; the size of the blocks is determined by the divided windows; the matching order of the image blocks is obtained by calculating the size of the Euclidean distance value; the set of correct matches is set as U + , the set of incorrect matches is set to U- , AP represents the average value of the matching accuracy value evaluated at the matching ranking position K, and the formula is as follows:

[0012]

[0013]

[0014] Among them, 1[] represents a binary indicator function; x i represents a set of blocks; if and only if U + Each image block in U - When all image blocks are in front of each other, AP reaches its optimal value;

[0015] In order to obtain discriminative local features, the final loss function is constructed through the confidence score map and AP value, which is expressed as follows:

[0016]

[0017] In the formula, Represents the confidence score map of the target image, and the calculation method is as shown in formula (1); in order to make the obtained features effective, we constrain the feature points through similarity loss, and the similarity calculation method is expressed as in Represents the feature confidence score map of the target image, represents the confidence score map of the matching image, C represents the set of correspondences between images I and I′, and q is the value in set C; f I and f I′ represents the feature descriptor and M(*,*) represents the triple margin ranking loss.

[0018] The image blocks are divided into 8×8 blocks.

[0019] The local feature descriptor is the final output result of the deep feature with the spatial structure enhancement feature added in the new local feature learning network framework, and then is obtained by performing l2 regularization processing; the feature detection heat map is the final output result of the deep feature with the spatial structure enhancement feature added in the new local feature learning network framework, and then is obtained by performing 1×1 convolution and softmax processing.

[0020] Specifically, the robustness training is to represent the detection and matching of feature points as probabilistic events, the results of which are determined by the output of the new local feature learning network framework; in the process of robustness training, the parameters are continuously updated through the reward function to obtain the final result;

[0021] The feature detection heat map and local feature descriptor roughly obtain the distribution of feature points through the strategy of non-maximum suppression, which is expressed as P(x|θ):

[0022]

[0023] According to the distribution results of feature points, the matching distribution is obtained through cycle consistency matching. express:

[0024]

[0025] in, represents the matching relationship between image A and image B, F A represents the position of the feature points extracted from image A, F B represents the position of the feature points extracted from image B, m xy represents the position of the matching points between the two images, Represents the learning parameters used to represent matching; uses depth information as the true value to determine whether the key points are correctly matched; designs a reward function based on the matching result. If the distance between the two points is less than the reprojection distance, a reward is given, indicating a correct match; otherwise, a penalty is given, indicating a matching failure;

[0026] use Represents the reward function:

[0027]

[0028] Wherein, r includes 1 and -0.25, when the match is successful, r = 1, when the match fails, r = -0.25;

[0029] Based on the probability distribution of feature points, the distribution of matching items, and the reward function, we can get the goal of robustness training:

[0030]

[0031] in and Represents the expected value of feature matching and feature points;

[0032] Update the network parameters θ by calculating the gradient of the above formula:

[0033]

[0034] in, represents the probability distribution of matching, Represents the probability distribution of the feature points of the original image, Represents the probability distribution of feature points matching the image; represents the gradient, R represents the reward function, and E represents the expectation.

[0035] Beneficial effects of the present invention: In order to solve the problems that local features extracted from previous visual tasks have poor robustness and weak distinguishability, the present invention proposes a new network framework that strengthens the spatial structural information so that the feature descriptor has both robustness and distinguishability. The invention introduces a spatial structure enhancement module, which is embedded into the network from shallow to deep, so that the network still contains rich structural information in the deep layer, thereby improving the distinguishability. In addition, we have designed a new loss function for distinguishability, and finally obtained a better local feature descriptor through the method of joint reinforcement learning. Finally, our method was evaluated on the feature matching task and achieved advanced results. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is the main flow chart of the technical solution of the present invention;

[0037] Figure 2 It is a flowchart of robust training. DETAILED DESCRIPTION

[0038] Figure 1 It is the main flow chart of the technical solution of the present invention. Figure 1 As shown, the local feature extraction method proposed by the present invention includes the following steps:

[0039] (1) Data preparation: It is very important to select a suitable data set to determine whether the local features are accurate. The present invention uses the MegaDepth data set with depth information as the training set. The depth information can help the model find the correspondence between two images, and the correspondence can be used as the true value of local feature training.

[0040] (2) Model construction: The present invention proposes a new local feature learning network framework. The framework uses the full convolutional network L2-Net as the backbone network. L2-Net is a network that does not require downsampling of images. The entire network always maintains the same resolution, which is conducive to the learning of local features. By extracting the features of the spatial structure enhancement module and merging them into the middle and deep layers, it is ensured that the structural information of the model will not be lost as the number of convolutional layers increases. Finally, the feature heat map for discriminative training and the feature detection heat map and feature descriptor for robust training are output. The specific contents are as follows:

[0041] For local feature descriptors, they should not only have strong resistance to appearance changes such as light and viewing angle, but also show strong distinguishability when facing similar structural images. According to the inherent characteristics of deep learning: shallow features contain more spatial structural information, and deep features have more semantic information. In order to have the common advantages of both, the present invention uses the idea of ​​spatial attention to enhance the structure of the network. The shallow features of the image are extracted, and a spatial structure enhancement module is constructed, which is then inserted into the backbone network from shallow to deep. The module is constructed by performing maximum pooling and average pooling operations on the extracted shallow features, then splicing the pooled results, and finally performing a convolution operation to obtain the final spatial structure enhancement features.

[0042] The output of the network consists of two parts. One part is the confidence score map used for discriminative training. The process is to perform a 1×1 convolution operation on the features with the spatial structure enhancement module added respectively, so that the feature shape is transformed from H×W×64 and H×W×128 to H×W×1, and then perform a Softmax operation on the two H×W×1 features respectively to obtain S l and S h , and perform weighted summation on the results to get the final result The specific calculation process is shown in Formula 1:

[0043]

[0044] Among them, ω l and ω h They represent the summation weights respectively. Based on experience, we set them to 1 and 2 to allow deep features to occupy a larger proportion. The other part is the feature descriptor and detection heat map used for robustness training. The feature descriptor is usually 128-dimensional. The final result of the network is directly subjected to l2 regularization to obtain the final feature descriptor. The feature detection heat map is a 1×1 convolution of the final result of the network followed by a softmax process.

[0045] (3) Model training

[0046] a) Discrimination training. Unlike most local feature learning-based methods, the network of the present invention not only outputs feature descriptors and detection heat maps for robustness training, but also outputs a single-channel confidence map for learning discrimination. The goal of this invention is to make the network pay more attention to areas with high confidence, that is, some smooth areas or areas without features should be paid less attention. This part is trained by obtaining the AP loss. To obtain the AP value, we first need to match the target image and the query image block by block. We only need to directly match the feature map. The blocks are divided and then matched. The size of the blocks can be determined according to the divided windows. The present invention divides them into 8×8 sizes. The matching order of the image blocks can be obtained by calculating the size of the Euclidean distance value. We set the set of correct matches as U + The set of incorrect matches is set to U - , AP represents the average value of the matching accuracy value evaluated at the matching ranking position K, and the formula is as follows:

[0047]

[0048]

[0049] Among them, 1[] represents a binary indicator function. i represents a set of blocks. + Each image block in U - When all image blocks are included in the image, AP reaches its optimal value.

[0050] In order to obtain discriminative local features, we use the confidence score map and AP value output by the network to construct the final loss function, which is expressed as follows:

[0051]

[0052] In the formula, represents the confidence score map of the target image, which is calculated as in formula (1). We assume that a higher confidence score means a higher pixel significance. Therefore, when AP and When both are higher, becomes smaller, which indicates that the obtained features are the most discriminative. At the same time, in order to make the obtained features effective, we constrain the feature points through similarity loss. The similarity calculation method is expressed as in Represents the feature confidence score map of the target image, represents the confidence score map of the matching image, C represents the set of correspondences between images I and I′, and q is the value in set C. I and f I′ represents the feature descriptor, M(*,*) represents the triple margin sorting loss, and the above two parts constitute the entire discriminative loss part.

[0053] b) Learning robustness.

[0054] This part of the invention introduces the reinforcement learning method, which represents the detection and matching of feature points as a probabilistic event, the result of which is determined by the output of the network. During the network training process, the reward function continuously updates the parameters to obtain the final result, which is the local feature we need. The whole process is as follows Figure 2 shown.

[0055] From the feature heat map and descriptor output from the last layer of the network, we can roughly get the distribution of feature points through the strategy of non-maximum suppression (NMS), which we express as P(x|θ):

[0056]

[0057] According to the distribution results of feature points, we can get the matching distribution through cycle consistency matching. We use express:

[0058]

[0059] in, represents the matching relationship between image A and image B, F A represents the position of the feature points extracted from image A, F B represents the position of the feature points extracted from image B, m xy represents the position of the matching points between the two images, represents the learned parameters used to represent the matching.

[0060] The network uses depth information as the true value, and based on the true value we can determine whether the key points are correctly matched. We design a reward function based on the matching result. If the distance between the two points is less than the reprojection distance, we give a reward, which means it is a correct match. Otherwise, we give a penalty, which means the match fails. Represents the reward function:

[0061]

[0062] Here, r includes 1 and -0.25. When the match is successful, r=1, and when the match fails, r=-0.25.

[0063] Based on the probability distribution of feature points, the distribution of matching items, and the reward function, we can get the training target:

[0064]

[0065] in and Represents the expected value of feature matching and feature points.

[0066] Update the network parameters θ by calculating the gradient of the above formula:

[0067]

[0068] in, represents the probability distribution of matching, Represents the probability distribution of the feature points of the original image, Represents the probability distribution of feature points matching the image. represents the gradient, R represents the reward function, and E represents the expectation. The expectation cannot be calculated directly, but it can be approximately expressed as a sum through sampling, and finally the training result is obtained.

[0069] The feature points of the present invention take into account both distinctiveness and repeatability. At the same time, because feature enhancement is performed during the learning process, it is ensured that the learned features contain more structural information and will have stronger expressiveness in the subsequent visual tasks.

[0070] (3) Experiment: The local features of the present invention are mainly applied in image matching and visual positioning tasks. The features of the original image and the query image are extracted, and then the features of the two images are matched using a brute force matching method. The performance of the extracted local features is verified by comparing them with the true matching values ​​of the two images.

[0071] In order to verify the effectiveness of the algorithm in long-term visual positioning accuracy, the present invention is tested on the Hpatches dataset, the images in the dataset are input into the network model, the positions and descriptors of the key points of each image are obtained, and then the descriptors of two images are randomly taken for matching. The matching method adopts the fastest brute force matching method at present, and finally the matching results of all images are used to calculate the MMA value, where MMA represents the average matching accuracy, and the larger the MMA value, the higher the matching accuracy. According to the experimental results, the matching accuracy MMA of the algorithm proposed in the present invention reaches 0.85 for multiple scenes under the change of viewing angle, and reaches 0.71 for multiple scenes under the change of illumination. The overall matching accuracy MMA reaches 0.78.

Claims

1. A local feature extraction method, characterized in that: A new local feature learning network framework is established; the new local feature learning network framework uses a fully convolutional network L2-Net as the backbone network, extracts features of a spatial structure enhancement module, and fuses them into the middle and deep layers of the backbone network; a dataset with depth information is used as a training set, and a correspondence is obtained through the depth information of the dataset, and the correspondence is used as the true value of local feature discriminative training and robust training; the new local feature learning network framework outputs a confidence score map for discriminative training and a feature detection heat map and local feature descriptors for robust training; discriminative training and robust training are performed separately to optimize the parameters of the new local feature learning network framework.

2. The local feature extraction method according to claim 1, characterized in that: The spatial structure enhancement module performs maximum pooling and average pooling operations on the shallow features extracted by the backbone network, then concatenates the pooled results, and finally performs a convolution operation to obtain the final spatial structure enhancement features, which are input into the middle layer and deep layer of the backbone network respectively.

3. The local feature extraction method according to claim 2, characterized in that: Specifically, the discriminative training is to perform a 1×1 convolution operation on the intermediate layer features with spatial structure enhancement features and the deep layer features with spatial structure enhancement features, respectively, so that the feature shapes are transformed from H×W×64 and H×W×128 to H×W×1, and then perform a Softmax operation on the two H×W×1 features to obtain S l and S h , and perform weighted summation on the results to obtain the final confidence score map for discriminative training The specific calculation process is shown in Formula 1: Among them, ω l and ω h Denote the summation weights, ω l =1,ω h =2; According to the confidence score map Get the AP value and perform discriminative training based on the AP loss.

4. The local feature extraction method according to claim 3, characterized in that: The AP value acquisition process is as follows: first, the target image and the query image are matched block by block, and the confidence score map is directly The blocks are divided and then matched; the size of the blocks is determined by the divided windows; the matching order of the image blocks is obtained by calculating the size of the Euclidean distance value; the set of correct matches is set as U + , the set of incorrect matches is set to U - , AP represents the average value of the matching accuracy value evaluated at the matching ranking position K, and the formula is as follows: Among them, 1[] represents a binary indicator function; x i represents a set of blocks; if and only if U + Each image block in U - When all image blocks are in front of each other, AP reaches its optimal value; In order to obtain discriminative local features, the final loss function is constructed through the confidence score map and AP value, which is expressed as follows: In the formula, Represents the confidence score map of the target image, and the calculation method is as shown in formula (1); in order to make the obtained features effective, we constrain the feature points through similarity loss, and the similarity calculation method is expressed as in Represents the feature confidence score map of the target image, represents the confidence score map of the matching image, C represents the set of correspondences between images I and I′, and q is the value in set C; f I and f I′ represents the feature descriptor and M(*,*) represents the triple margin ranking loss.

5. The local feature extraction method according to claim 4, characterized in that: The image blocks are divided into 8×8 blocks.

6. The local feature extraction method according to claim 1 or 2, characterized in that: The local feature descriptor is the final output result of the deep feature with the spatial structure enhancement feature added in the new local feature learning network framework, and then is obtained by performing l2 regularization processing; the feature detection heat map is obtained by performing 1×1 convolution and then softmax processing on the final output result of the deep feature with the spatial structure enhancement feature added in the new local feature learning network framework.

7. The local feature extraction method according to claim 6, characterized in that: Specifically, the robustness training is to represent the detection and matching of feature points as probabilistic events, the results of which are determined by the output of the new local feature learning network framework; in the process of robustness training, the parameters are continuously updated through the reward function to obtain the final result; The feature detection heat map and local feature descriptor roughly obtain the distribution of feature points through the strategy of non-maximum suppression, which is expressed as P(x|θ): According to the distribution results of feature points, the matching distribution is obtained through cycle consistency matching. express: in, represents the matching relationship between image A and image B, F A represents the position of the feature points extracted from image A, F B represents the position of the feature points extracted from image B, m xy represents the position of the matching points between the two images, Represents the learning parameters used to represent matching; uses depth information as the true value to determine whether the key points are correctly matched; designs a reward function based on the matching result. If the distance between the two points is less than the reprojection distance, a reward is given, indicating a correct match; otherwise, a penalty is given, indicating a matching failure; use Represents the reward function: Wherein, r includes 1 and -0.25, when the match is successful, r = 1, when the match fails, r = -0.25; Based on the probability distribution of feature points, the distribution of matching items, and the reward function, we can get the goal of robustness training: in and E P(A;θ)&P(B;θ) Represents the expected value of feature matching and feature points; Update the network parameters θ by calculating the gradient of the above formula: in, represents the probability distribution of matching, Represents the probability distribution of the feature points of the original image, represents the probability distribution of the feature points of the matching image; ▽ represents the gradient, R represents the reward function, and E represents the expectation.

Citation Information

Patent Citations

  • Chicken farm raised chicken recognition algorithm based on Yolov4

    CN113221853A

  • Complementary feature reliable description and matching method based on deep learning

    CN115294371A

  • Mine image enhancement method

    CN118314056A

  • Image sparse local feature extraction method based on improved PoSFeat network

    CN118379509A

  • Medical image segmentation method based on u-net

    US20220309674A1

Cited By

  • Panorama feature matching method based on position sensing thin plate spline transformation

    CN120318539A