A local feature learning method based on semantic guidance and task reward / punishment

Abstract

This invention belongs to the field of computer vision technology and discloses a local feature learning method based on a combination of semantic guidance and task reward / penalty. A local feature extraction network acquires feature detection heatmaps and feature descriptors from the original image; the original image generates semantic feature classes through a pre-trained semantic segmentation network; the semantic feature classes are combined with the feature detection heatmaps and feature descriptors, and the output is used to obtain the feature point detection distribution and semantic feature descriptors for the feature points; the matching result of the semantic feature descriptors of the original image and the target image serves as the training basis for whether the feature points are correct; the training method is reinforcement learning, in which a gradient update formula is used to optimize the local feature extraction network during reinforcement learning training. This invention considers the influence of semantic information and constrains the local feature extraction network to learn effective local features through semantic information constraints. Reinforcement learning training is used to reward or penalize feature points, making the network more discriminative and robust.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a local feature learning method based on a combination of semantic guidance and task rewards and punishments. Background Technology

[0002] In recent years, local feature learning methods have made significant progress in computer vision, especially with the introduction of deep learning, which has greatly improved the accuracy and robustness of feature extraction. Traditional methods such as SIFT and ORB have some robustness in handling changes in illumination, viewpoint, and scale, but their performance is limited in real-time scenarios and complex scenes. Recent research has applied deep neural networks to the feature detection and description stages, enabling the learning of more expressive features from large-scale data and adapting to complex real-world scenes. Furthermore, some novel methods integrate multi-task learning, attention mechanisms, and self-supervised learning in feature extraction, significantly improving the discriminative power and computational efficiency of features. These methods provide more accurate and robust visual matching capabilities for tasks such as 3D reconstruction, augmented reality, and autonomous localization.

[0003] The paper "Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, papers 5206-5216, 2023" proposes a semantically aware feature extraction method. This method extracts globally reliable features by implicitly embedding high-level semantics into the detection and description process, rather than relying on explicit semantic labels, making the method more generalizable. While it is more semantically discriminative when dealing with complex environmental changes, it lacks robustness, resulting in inaccurate detection of the extracted features. The paper "Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, papers 1306-1315, 2023" proposes a novel deformable sensing network for joint detection and description of key points to address the challenging problem of matching deformable surfaces. All network components work collaboratively through a feature fusion method that enforces the uniqueness and invariance of descriptors. It exhibits strong robustness in scenes with large viewpoint changes, but its performance is poor when facing complex lighting variations, and its high time complexity hinders practical applications. Summary of the Invention

[0004] The purpose of this invention is to propose a local feature learning method based on the combination of semantic guidance and task reward / penalty. By combining semantic guidance and task reward / penalty for local feature extraction, it aims to have strong robustness and discriminativeness against complex environmental changes.

[0005] The technical solution of this invention is as follows: a local feature learning method based on semantic guidance and task reward / penalty combination, which obtains the feature detection heatmap f of the original image through a local feature extraction network. det and feature descriptor f des The original image is processed by a pre-trained semantic segmentation network to generate semantic feature classes; the semantic feature classes obtained by the semantic segmentation network are compared with the feature detection heatmaps output by the local feature extraction network. det and feature descriptor f des The combined outputs are used to obtain the feature point detection distribution and the semantic feature descriptor of the feature points, respectively. The matching result of the semantic feature descriptor of the original image and the semantic feature descriptor of the target image serves as the training basis for whether the feature points are correct. The training method is reinforcement learning. During the reinforcement learning training process, a gradient update formula is used to optimize the detection and description of feature points in the local feature extraction network.

[0006] The gradient update formula is:

[0007]

[0008] in, Let P'(A|f) represent the gradient, r represent the reward value at each feature point, and E represent the expectation; det ) and P′(B|f det () represent the feature point detection distribution of the original image and the feature point detection distribution of the target image, respectively; For feature matching distribution;

[0009] The reward function is defined as follows: when feature points in two images have depth information and are within the ε-pixel range of their respective reprojections, the match is considered valid, and the matched feature points are rewarded. In this case, the reward function is defined as 1. If there is no depth information between two feature points, the outer polar distance from one feature point to the other is calculated. If the distance is less than 2, the feature points are considered to be matched reasonably, and the reward function is defined as 0.25. If neither of the above two conditions is met, the two feature points are not matched, and the reward function is defined as -0.001.

[0010] The feature point detection distribution P′(·|f detSpecifically, the detected feature points are classified by introducing semantic feature classes. For visual tasks in outdoor scenes, fixed objects such as buildings are defined as long-term effective features, and the value of these feature points is set to 4. Pedestrians are defined as invalid features, and the value of their feature points is set to 1. Grass and mountains are defined as short-term effective features, and the value of their feature points is set to 2. Other objects in the training set that involve some indoor scenes but are irrelevant to the test are set to a value of 3. Thus, a semantic reliability feature map S is constructed. mask :

[0011]

[0012] Among them, H s ′ eg This represents the final result after semantic classification of feature points; when H s ′ eg When S = 1, this feature is invalid, and S is in this case. mask =0; when H s ′ eg =2 or H s ′ eg When S = 3, mask =0.5; when H s ′ eg When S = 4, S mask =1; The feature detection heatmap f det Perform softmax(·) operation and connect with S mask Multiplying these yields the final feature point detection distribution, expressed as follows:

[0013] P′(x|f det ) = softmax(f det (x))·S mask (f det (x)) (2).

[0014] The feature matching distribution Specifically, based on the semantic reliability feature map S mask Feature descriptors are also divided into four categories, represented as follows: These are called semantic feature descriptors. During matching, the system first determines whether the semantic feature descriptors belong to the same category. If they do, a match is made; otherwise, no match is made. The formula for calculating the matching probability distribution is as follows:

[0015]

[0016] Among them, M c This represents the matching result between the original image A and the target image B, where c represents the category. and These represent the semantic feature descriptors of the feature points in the original image and the semantic feature descriptors of the feature points in the target image, respectively.

[0017] The semantic segmentation network employs the UpperNet segmentation framework for semantic segmentation. The network has an encoder-decoder structure; the encoder downsamples the outputs of the four network layers by 1 / 4, 1 / 8, 1 / 16, and 1 / 32, producing feature maps with [128, 256, 512, 1024] channels. The pyramid aggregation module acts as the decoder, generating feature maps for semantic segmentation. The segmentation head H... seg Process the feature map output by the decoder; the calculation process is as follows:

[0018] E seg =Encoder seg (I) (4)

[0019] H seg =Decoder seg (E seg (5)

[0020] Encoder seg Decoder represents the encoder of a semantic segmentation network. seg This represents the decoder of the semantic segmentation network.

[0021] The local feature extraction network adopts the Unet framework with an encoding and decoding structure; the original image I is input into the encoder for feature encoding to obtain the encoded features E. fea The encoded features are fed into the decoding section, resulting in two outputs: a feature detection heatmap f used for feature point detection. det and the feature descriptor f used for feature point description des ;

[0022] E fea =Encoder fea (I) (6)

[0023] f det ,f des =Decoder fea (E fea (7)

[0024] Encoder fea Decoder represents the encoder of a local feature extraction network. fea This represents the decoder of the local feature extraction network.

[0025] The beneficial effects of this invention are as follows: Addressing the issue that previous local feature extraction methods for visual tasks did not adequately consider semantic information, this invention proposes a local feature extraction method that combines semantic guidance and task reward. This invention utilizes a semantic segmentation network to obtain semantic priors, which constrain the local feature extraction network to learn effective local features. Furthermore, the final semantically constrained local feature detection network and descriptors are trained using reinforcement learning, rewarding or penalizing feature points according to the specific task, making the final learned network more discriminative and robust. Finally, our method was evaluated on a feature matching task and achieved state-of-the-art results. Attached Figure Description

[0026] Figure 1(a) is a flowchart of a local feature learning method based on the combination of semantic guidance and task reward and punishment;

[0027] Figure 1(b) is a schematic diagram of reinforcement learning;

[0028] Figure 2 This is a semantic reliability feature map for semantic categories. Detailed Implementation

[0029] Figures 1(a) and 1(b) are the main flowcharts of the technical solution of this invention. As shown in Figures 1(a) and 1(b), the local feature learning method proposed in this invention, based on a combination of semantic guidance and task reward / penalty, proposes an effective local feature extraction network that combines semantic segmentation and reinforcement learning methods to maximize the importance of semantic information in local feature extraction. From practical applications, we understand that different visual tasks have different requirements for local feature extraction. For example, this invention mainly targets long-term visual localization tasks, where we hope that features should focus on things that remain unchanged over time, such as houses and walls. Deformable objects like trees and pedestrians should receive less attention. Semantic information can help us distinguish which objects are the focus of local features. Reinforcement learning, as an interactive learning method, plays a crucial role in this invention by maximizing the reward for useful semantic features in complex and uncertain environments. In other words, reinforcement learning can continuously optimize the desired results based on semantic information.

[0030] Feature detection heatmaps of the original image are obtained through a local feature extraction network. det and feature descriptor f des The original image is processed by a pre-trained semantic segmentation network to generate semantic feature classes; the semantic feature classes obtained by the semantic segmentation network are compared with the feature detection heatmaps output by the local feature extraction network. det and feature descriptor f desThe combined outputs are used to obtain the feature point detection distribution and the semantic feature descriptor of the feature points, respectively. The matching result of the semantic feature descriptor of the original image and the semantic feature descriptor of the target image serves as the training basis for whether the feature points are correct. The training method is reinforcement learning. During the reinforcement learning training process, a gradient update formula is used to optimize the detection and description of feature points in the local feature extraction network.

[0031] The network framework of this invention mainly consists of two parts: a local feature extraction network and a semantic segmentation network. The local feature extraction network adopts the Unet framework. Unet is an encoding and decoding organization structure. The original image I is input into the encoder for feature encoding to obtain the encoded features E. fea The encoded features are then fed into the decoding section, yielding two outputs: one is a feature detection heatmap f used for feature point detection. det One is the feature descriptor f used to describe feature points. des .

[0032] E fea =Encoder fea (I) (1)

[0033] f det ,f des =Decoder fea (E fea (2)

[0034] Encoder fea and Decoder fea These represent the encoding and decoding of the feature extraction network, respectively.

[0035] For the semantic segmentation network, ConvNeXt is used as the backbone network, combined with the efficient UpperNet segmentation framework for the final semantic segmentation task. ConvNeXt, as one of the best backbone networks in recent years, exhibits excellent performance in semantic segmentation tasks. Furthermore, its CNN architecture is particularly suitable for the local feature extraction integration of this invention. This semantic segmentation network follows an encoder-decoder structure. Using the ConvNeXt base as the encoder, downsampling is performed in four stages at 1 / 4, 1 / 8, 1 / 16, and 1 / 32, outputting feature maps with [128, 256, 512, 1024] channels to capture local details and global semantic information of the image. The decoder utilizes a pyramid pooling module (PPM) to enhance the network's ability to capture global information. Finally, feature maps for semantic segmentation are generated. The segmentation head H... seg The feature maps output from the decoder are processed to generate a final segmentation map containing 150 categories, corresponding to all categories in the ADE20K dataset. The calculation process is as follows:

[0036] E seg =Encoder seg (I) (3)

[0037] H seg =Decoder seg (E seg (4)

[0038] Encoder seg and Decoder seg These represent the encoding and decoding of the semantic segmentation network, respectively. The segmentation result obtained by the semantic segmentation network is combined with the extraction result of the local feature extraction network, allowing the local feature learning network to selectively learn effective local features based on the semantic segmentation result. The two networks together constitute the main network structure of this invention, as shown in Figure 1(a).

[0039] This invention utilizes reinforcement learning to optimize local features, as shown in Figure 1(b). Compared to traditional methods, reinforcement learning-based methods enable more robust local features extracted by the local feature learning network. Furthermore, by introducing semantic information, this invention allows for semantic classification of local features, enabling reinforcement learning to selectively reward and penalize features based on semantic information, thus extracting more accurate local features. The key to applying reinforcement learning lies in representing the local feature extraction task in probabilistic form. This invention mainly comprises two parts: feature point detection distribution and feature matching distribution.

[0040] a) Feature point detection distribution. Feature heatmap f is obtained through the Unet network. det .

[0041] Feature detection heatmap f det Perform softmax(·) operation and connect with S mask Multiply them to obtain the final feature point detection distribution.

[0042] Preferably, the feature detection heatmap is divided into a grid, with each grid having a window size of w = [5×5]. The maximum value is extracted from each grid as a pre-selected feature point. Then, the pre-selected feature points are processed using soft max(·) to finally obtain the probability distribution of the image feature points. As shown in the formula:

[0043]

[0044] Not all pre-selected feature points are valid. Processing all feature points would not only take more time but also increase the risk of errors. By introducing semantic information to classify feature points, such as... Figure 2As shown. Specifically, regarding the 150 categories output by the semantic segmentation network, this invention reclassifies these 150 categories into four main categories. These categories include: valid features, such as buildings; invalid features, such as pedestrians and trees; features that are not important for the visual task, such as mountains and streets; and finally, some indoor scene features, such as curtains and carpets, which we also consider irrelevant to mountains and streets. Thus, a semantic reliability feature map S is constructed. mask :

[0045]

[0046] Among them, H seg This represents the result after semantic classification, when H seg When S = 1, we consider the feature invalid, and at this time S mask =0. When H seg =2 or H seg When S = 3, we consider this feature unimportant. mask =0.5. When H seg When S = 4, we consider the feature to be valid, and at this time S mask =1. Combining feature points with the semantic reliability feature map yields an effective semantic feature map. Finally, the final feature point detection distribution is obtained:

[0047]

[0048] According to the formula, we can obtain the feature probability distribution of image A. and the feature probability distribution of image B

[0049] b) Feature Matching Distribution. Unlike traditional algorithms that directly perform global matching between two images, this invention fully utilizes semantic information to optimize the matching algorithm. The 128-dimensional feature descriptors are semantically divided into four categories, represented as follows: Therefore, when performing a matching operation, we first determine whether the features belong to the same category. If the features belong to the same category, we perform a match; otherwise, we do not match. The matching probability can be calculated using a formula.

[0050]

[0051] Among them, M c represents the result of matching the semantic features of image A with the semantic features of image B, where c represents the category.

[0052] Based on the real ground pose and depth information provided by the dataset, the precise correspondence between the pixels of the original image A and the target image B can be obtained, and the correctness of the matching result output by the model can be determined by this correspondence.

[0053] a) Reward Function. The reward function is used to reward or penalize image matching results; we define it as follows: If two pixels have depth information and are within ε pixels of their respective reprojections, we consider the match valid and reward the matching feature point with a reward value of 1. If there is no depth information between the two pixels, we calculate the outer polar distance from one pixel to the other. If the distance is less than 2 pixels, we also consider the match reasonable, and in this case, we do not penalize the pixel, setting the reward value to -0.25. If neither of the above conditions is met, the two pixels do not match, and we penalize the feature point, setting the reward value to -0.001.

[0054] b) Enhancing reward maximization. Given the probability distribution of feature points, the distance distribution of feature matching, and the objective function, a gradient update formula is obtained to optimize the detection and description of feature points in the local feature extraction network:

[0055]

[0056] in, Let represent the gradient, r represent the reward function, and E represent the expectation. By rewarding correct matches, the gradient is continuously updated to achieve the optimal result.

[0057] The method of this invention is trained on the megadepth dataset, which contains various scenes from around the world and provides high-quality depth maps for each image, serving as ground truth for local feature training. This invention uses PyTorch 1.8 + CUDA 11.1 to build the runtime environment for the network model, employing the Adam optimizer with a learning rate of 1e-4 and a batch size of 4, training for a total of 40 epoches. The training images are uniformly set to 768×768 pixels. The network model is trained on two GenForce RTX 3090 GPUs, and all tests are performed on a GenForce RTX 1080Ti.

[0058] To verify the effectiveness of the algorithm, this invention was tested on the Hpatches dataset. According to the experimental results, the proposed algorithm achieved a matching accuracy (MMA) of 0.858 for multiple scenes under varying viewpoints and 0.75 for multiple scenes under varying illumination. The overall matching accuracy (MMA) reached 0.80.

Claims

1. A local feature learning method based on a combination of semantic guidance and task reward / punishment, characterized in that, Feature detection heatmaps of the original image are obtained using a local feature extraction network. and feature descriptors The original image is processed by a pre-trained semantic segmentation network to generate semantic feature classes; the semantic feature classes obtained by the semantic segmentation network are compared with the feature detection heatmap output by the local feature extraction network. and feature descriptors The combined outputs are used to obtain the feature point detection distribution and the semantic feature descriptors of the feature points, respectively. The matching result between the semantic feature descriptors of the original image and the target image serves as the training basis for determining whether the feature points are correct. The training method is reinforcement learning. During the reinforcement learning training process, a gradient update formula is used to optimize the detection and description of feature points in the local feature extraction network. The gradient update formula is: ； in, ; Let r represent the gradient, r represent the reward value at each feature point, and E represent the expectation. and These represent the feature point detection distributions of the original image and the target image, respectively. For feature matching distribution; The reward function is used when the feature points of two images have depth information and are in their respective reprojections. Within a pixel range, a match is considered valid, and the matching feature point is rewarded; in this case, the reward function is defined as 1. If there is no depth information between two feature points, the outer polar distance from one feature point to the other is calculated. If the distance is less than 2, the feature points are considered a valid match, and the reward function is defined as 0.

25. If neither of the above two conditions is met, the two feature points are not considered a match, and the reward function is defined as -0.

001. The feature point detection distribution... Specifically, the detected feature points are classified by introducing semantic feature classes. For visual tasks in outdoor scenes, fixed objects such as buildings are defined as long-term effective features, with a value of 4 for these feature points. Pedestrians are defined as invalid features, with a value of 1 for these feature points. Grass and mountains are defined as short-term effective features, with a value of 2 for these feature points. Other objects in the training set that are not relevant to the test are defined with a value of 3 for these feature points. Thus, a semantic reliability feature map is constructed. : (1) ； in, This represents the final result after semantic classification of feature points; when At that time, the feature is invalid. ;when or hour, ;when At this time Feature detection heatmap conduct Operation and Multiplying these yields the final feature point detection distribution, expressed as follows: (2)； The feature matching distribution Specifically, based on the semantic reliability feature map... Feature descriptors are also divided into four categories, represented as follows: These are called semantic feature descriptors. During matching, the system first determines whether the semantic feature descriptors belong to the same category. If they do, a match is made; otherwise, no match is made. The formula for calculating the matching probability distribution is as follows: (3) ； in, This represents the matching result between the original image A and the target image B, where c represents the category. and These represent the semantic feature descriptors of the feature points in the original image and the semantic feature descriptors of the feature points in the target image, respectively.

2. The local feature learning method based on semantic guidance and task reward / punishment combination according to claim 1, characterized in that, The semantic segmentation network employs the UpperNet segmentation framework for semantic segmentation. The network has an encoder-decoder structure; the encoder downsamples the outputs of the four network layers by 1 / 4, 1 / 8, 1 / 16, and 1 / 32, resulting in feature maps with 128, 256, 512, and 1024 output channels, respectively. The pyramid aggregation module acts as the decoder, generating feature maps for semantic segmentation. The segmentation head... Process the feature map output by the decoder; the calculation process is as follows: (4) ； (5) ； Represents the encoder of a semantic segmentation network; This represents the decoder of the semantic segmentation network.

3. The local feature learning method based on semantic guidance and task reward / punishment as described in claim 1 or 2, characterized in that, The local feature extraction network employs the Unet framework with an encoding and decoding structure; it processes the original image... Input the encoder to perform feature encoding to obtain the encoded features The encoded features are fed into the decoding section, resulting in two outputs: a feature detection heatmap for feature point detection. and feature descriptors used for feature point description ; (6) ； (7) ； The encoder represents the local feature extraction network. This represents the decoder of the local feature extraction network.

Images