Local feature learning method based on combination of semantic guidance and task reward and punishment

Abstract

The invention belongs to the technical field of computer vision, and discloses a local feature learning method based on combination of semantic guidance and task reward and punishment. The local feature extraction network obtains an original image feature detection heat map and a feature descriptor; generating semantic feature classes from the original image through a pre-training semantic segmentation network; the semantic feature class is combined with the feature detection heat map and the feature descriptors, and an output result is used for obtaining feature point detection distribution and obtaining semantic feature descriptors of feature points; a matching result of semantic feature descriptors of the original image and the target image is used as a training basis for judging whether feature points are correct or not; the training method is reinforcement learning, and in the reinforcement learning training process, the local feature extraction network is optimized by using a gradient updating formula. The semantic information influence is considered, and effective local features are learned through the semantic information constraint local feature extraction network. And feature points are rewarded or punished by utilizing reinforcement learning training, so that the network has better discrimination and robustness.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a local feature learning method based on the combination of semantic guidance and task rewards and penalties. Background Art

[0002] In recent years, local feature learning methods have made significant progress in computer vision, especially the introduction of deep learning has greatly improved the accuracy and robustness of feature extraction. Traditional methods such as SIFT and ORB have a certain degree of robustness in dealing with changes in illumination, perspective and scale, but their performance in real-time and complex scenes is limited. Recent research has applied deep neural networks to the feature detection and description stages, which can learn features with stronger expressive power from large-scale data and adapt to complex real-world scenes. In addition, some new methods integrate multi-task learning, attention mechanisms and self-supervised learning in feature extraction, which greatly improves the discriminative power and computational efficiency of features. These methods provide more accurate and robust visual matching capabilities for tasks such as 3D reconstruction, augmented reality and autonomous positioning.

[0003] The paper "Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, papers 5206-5216, 2023" proposes a semantic-aware feature extraction method that extracts globally reliable features by implicitly embedding high-level semantics into the detection and description process, rather than relying on explicit semantic labels, making the method more generalizable. Although it is more semantically discriminative when dealing with complex environmental changes, it is still lacking in robustness, resulting in the inability of the extracted features to be accurately detected. The paper "Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, papers 1306-1315, 2023" proposes a new deformation-aware network for joint detection and description of key points to deal with the challenging problem of matching deformable surfaces. All network components work together through a feature fusion method that enforces the uniqueness and invariance of descriptors. It shows strong robustness when dealing with scenes with large viewing angle changes, but the effect is poor when facing complex light changes, and the time complexity is high, which is not conducive to practical applications. Summary of the invention

[0004] The purpose of this invention is to propose a local feature learning method based on the combination of semantic guidance and task rewards and penalties, which combines semantic guidance and task rewards and penalties to extract local features, aiming to have strong robustness and discriminability when resisting complex environmental changes.

[0005] The technical solution of the present invention is as follows: a local feature learning method based on the combination of semantic guidance and task rewards and punishments, which obtains the feature detection heat map of the original image through the local feature extraction network det and feature descriptor f des ; The original image is passed through the pre-trained semantic segmentation network to generate semantic feature classes; The semantic feature classes obtained by the semantic segmentation network and the feature detection heat map f output by the local feature extraction network det and feature descriptor f des The output results are respectively used to obtain the feature point detection distribution and the semantic feature descriptor of the feature point; the matching result of the semantic feature descriptor of the original image and the semantic feature descriptor of the target image is used as the training basis for whether the feature point is correct; the training method is reinforcement learning, and during the reinforcement learning training process, a gradient update formula is used to optimize the detection and description of the feature points of the local feature extraction network.

[0006] The gradient update formula is:

[0007]

[0008] in, represents the gradient, r represents the reward value of each feature point, and E represents the expectation; P′(A|f det ) and P′(B|f det ) represent the feature point detection distribution of the original image and the feature point detection distribution of the target image respectively; is the feature matching distribution;

[0009] is the reward function; when the feature points of the two images have depth information and are within the ε pixel range of their respective reprojections, the match is considered valid and the matching feature points are rewarded. In this case, the reward function is defined as 1; there is no depth information between the two feature points, and the outer pole distance from the feature point to another feature point is calculated. If the distance is less than 2, the feature points are reasonably matched, and the reward function is defined as 0.25 at this time; if neither of the above two conditions is met, the two feature points do not match, and the reward function is defined as -0.001.

[0010] The feature point detection distribution P′(·|f det) Specifically, the detected feature points are classified by introducing semantic feature classes; for the visual task of outdoor scenes, fixed objects such as buildings are defined as long-term valid features, and the value of this type of feature points is set to 4, pedestrians are defined as invalid features, and the value of feature points is set to 1, grass and mountains are defined as short-term valid features, and the value of feature points is set to 2. For other objects in the training set involving some indoor scenes, but for features that are not important for testing, the value of feature points is set to 3; thus, the semantic reliability feature map S is constructed. mask :

[0011]

[0012] Among them, H s ' eg It represents the final result after semantic classification of feature points; when H s ' eg = 1, the feature is invalid, then S mask =0; when H s ' eg =2 or H s ' eg =3, S mask =0.5; when H s ' eg =4, then S mask =1; the feature detection heat map f det Perform softmax(·) operation and combine with S mask Multiply them together to get the final feature point detection distribution, which is expressed as follows:

[0013] P′(x|f det )=softmax(f det (x))·S mask (f det (x)) (2).

[0014] The feature matching distribution Specifically: According to the semantic reliability feature map S mask , feature descriptors are also divided into four categories, represented as It is called a semantic feature descriptor. When performing a matching operation, first determine whether the semantic feature descriptors belong to the same category. If they belong to the same category, they are matched, otherwise they are not matched. The matching probability distribution calculation formula is as follows:

[0015]

[0016] Among them, M c represents the matching result between the original image A and the target image B, and c represents the category. and They represent the semantic feature descriptors of the original image feature points and the target image feature points respectively.

[0017] The semantic segmentation network adopts the UperNet segmentation framework for semantic segmentation; the semantic segmentation network is an encoder-decoder structure, the encoder downsamples the output of the four network layers by 1 / 4, 1 / 8, 1 / 16 and 1 / 32, and the output channel is a feature map of [128, 256, 512, 1024], and the pyramid pooling module is used as a decoder to generate a feature map for semantic segmentation; the segmentation head H seg Process the feature map output by the decoder; the calculation process is as follows:

[0018] E seg =Encoder seg (I) (4)

[0019] H seg =Decoder seg (E seg ) (5)

[0020] Encoder seg Decoder represents the encoder of the semantic segmentation network; seg Represents the decoder of the semantic segmentation network.

[0021] The local feature extraction network adopts the Unet framework of the encoding and decoding organizational structure; the original image I is input into the encoder for feature encoding to obtain the encoded feature E fea The encoded features are sent to the decoding part to obtain two outputs: the feature detection heat map f for feature point detection det and the feature descriptor f used for feature point description des ;

[0022] E fea =Encoder fea (I) (6)

[0023] f det ,f des =Decoder fea (E fea ) (7)

[0024] Encoder fea Decoder represents the encoder of the local feature extraction network. fea Represents the decoder of the local feature extraction network.

[0025] Beneficial effects of the present invention: In view of the problem that local features extracted in previous visual tasks do not fully consider semantic information, the present invention proposes a local feature extraction method that combines semantic guidance and task rewards. The invention uses a semantic segmentation network to obtain semantic priors to constrain the local feature extraction network to learn effective local features. In addition, the local feature detection network and descriptor that finally contain semantic constraints are trained using reinforcement learning, and feature points are rewarded or punished according to specific tasks, so that the final learned network can be more discriminative and robust. Finally, our method was evaluated on feature matching tasks and achieved advanced results. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1(a) is a flow chart of a local feature learning method based on the combination of semantic guidance and task rewards and penalties;

[0027] Figure 1(b) is a schematic diagram of reinforcement learning;

[0028] Figure 2 It is the semantic reliability feature map of semantic category. DETAILED DESCRIPTION

[0029] Figures 1(a) and 1(b) are the main flow charts of the technical solution of the present invention. As shown in Figures 1(a) and 1(b), the local feature learning method based on the combination of semantic guidance and task rewards and penalties proposed in the present invention proposes an effective local feature extraction network, which combines semantic segmentation with reinforcement learning methods to maximize the importance of semantic information in local feature extraction. From practical applications, we know that different visual tasks have different requirements for local feature extraction. For example, the present invention is mainly aimed at long-term visual positioning tasks. We hope that the features should focus on things that remain unchanged for a long time, such as houses, walls, etc., while deformable objects such as trees and pedestrians should be given less attention. Semantic information can help us distinguish which objects are the local features that should be paid attention to. As an interactive learning method, reinforcement learning plays a role in the entire invention in that it can give the maximum reward to useful semantic features in a complex and uncertain environment. In other words, reinforcement learning can continuously optimize the results we want based on semantic information.

[0030] Obtain the feature detection heat map f of the original image through the local feature extraction network det and feature descriptor f des ; The original image is passed through the pre-trained semantic segmentation network to generate semantic feature classes; The semantic feature classes obtained by the semantic segmentation network and the feature detection heat map f output by the local feature extraction network det and feature descriptor f desThe output results are respectively used to obtain the feature point detection distribution and the semantic feature descriptor of the feature point; the matching result of the semantic feature descriptor of the original image and the semantic feature descriptor of the target image is used as the training basis for whether the feature point is correct; the training method is reinforcement learning, and during the reinforcement learning training process, a gradient update formula is used to optimize the detection and description of the feature points of the local feature extraction network.

[0031] The network framework of the present invention is mainly divided into two parts: local feature extraction network and semantic segmentation network. The local feature feature extraction network adopts the Unet framework. Unet is an organizational structure for encoding and decoding. The original image I is input into the encoder for feature encoding to obtain the encoded feature E fea , and then send the encoded features to the decoding part to get two outputs: one is the feature detection heat map f for feature point detection det , one is the feature descriptor f used to describe the feature points des .

[0032] E fea =Encoder fea (I) (1)

[0033] f det ,f des =Decoder fea (E fea ) (2)

[0034] Encoder fea and Decoder fea Represent the encoding and decoding of the feature extraction network respectively.

[0035] For the semantic segmentation network, ConvNeXt is used as the backbone network, combined with the efficient UperNet segmentation framework to perform the final semantic segmentation task. As one of the most outstanding backbone networks in recent years, ConvNeXt has shown excellent performance in semantic segmentation tasks. In addition, its CNN architecture is particularly suitable for the local feature extraction integration of the present invention and. The semantic segmentation network follows the encoder-decoder structure. Using the ConvNeXt base as the encoder, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 downsampling is performed in four stages, and the output channels are feature maps of [128, 256, 512, 1024] to capture local details and global semantic information of the image. The decoder uses a pyramid pooling module (PPM) to enhance the network's ability to capture global information. Finally, a feature map for semantic segmentation is generated. The segmentation head H seg Process the feature map output by the decoder to generate a final segmentation map containing 150 categories, corresponding to all categories in the ADE20K dataset. The calculation process is as follows:

[0036] E seg =Encoder seg (I) (3)

[0037] H seg =Decoder seg (E seg ) (4)

[0038] Encoder seg and Decoder seg They represent the encoding and decoding of the semantic segmentation network respectively. The segmentation result obtained by the semantic segmentation network is combined with the extraction result of the local feature extraction network, so that the local feature learning network can selectively learn effective local features according to the semantic segmentation result. The two networks together constitute the main network structure of the present invention, as shown in Figure 1(a).

[0039] The present invention uses reinforcement learning to optimize local features, as shown in Figure 1(b). Compared with traditional methods, the method based on reinforcement learning can make the local features extracted by the local feature learning network more robust. In addition, due to the introduction of semantic information, the present invention can semantically classify local features, so that reinforcement learning can reward and punish features in a targeted manner according to semantic information to extract more accurate local features. The key to the application of reinforcement learning is to represent the local feature extraction task in the form of probability. The present invention mainly includes two parts: feature point detection distribution and feature matching distribution.

[0040] a) Feature point detection distribution. The feature heat map f is obtained through the Unet network det .

[0041] The feature detection heat map f det Perform softmax(·) operation and combine with S mask Multiply them together to get the final feature point detection distribution.

[0042] Preferably, the feature detection heat map is divided into grids, and the window size of each grid is w = [5 × 5]. The maximum value is extracted from each grid as the pre-selected feature point. Then the pre-selected feature points are operated by soft max (·), and finally the probability distribution of the image feature points is obtained. As shown in the formula:

[0043]

[0044] Not all pre-selected feature points are valid. If all feature points are processed, it will not only take more time but also increase the risk of error. By introducing semantic information to classify feature points, such as Figure 2Specifically, regarding the 150 categories output by the semantic segmentation network, the present invention reclassifies these 150 categories into four categories. These categories include: valid features, such as buildings; invalid features, such as pedestrians and trees; features that are not important to the visual task, such as mountains and streets; and finally some indoor scene features, such as curtains and carpets, which we also believe are irrelevant to mountains and streets. Thus, the semantic reliability feature map S is constructed. mask :

[0045]

[0046] Among them, H seg Represents the result after semantic classification. When H seg = 1, we consider the feature to be invalid. mask = 0. When H seg =2 or H seg =3, we believe that this feature is not important. mask =0.5. When H seg =4, we believe that the feature is valid, then S mask = 1. Combine the feature points with the semantic reliability feature map to get an effective semantic feature map. Finally, the final feature point detection distribution is obtained:

[0047]

[0048] According to the formula, we can obtain the feature probability distribution of image A And the feature probability distribution of image B

[0049] b) Feature matching distribution. Different from the traditional algorithm that directly performs global matching on two images, the present invention makes full use of semantic information to optimize the matching algorithm, and semantically divides the 128-dimensional feature descriptors into four categories, which are represented as Therefore, when performing matching operations, we first determine whether the features belong to the same category. If the features belong to the same category, we match them, otherwise we do not match them. The matching probability can be calculated using the formula.

[0050]

[0051] Among them, M c It represents the result of matching the semantic features of image A with the semantic features of image B, and c represents the category.

[0052] Based on the real ground posture and depth information provided by the dataset, the precise correspondence between the pixels of the original image A and the target image B can be obtained, and this correspondence can be used to determine whether the matching result output by the model is correct.

[0053] a) Reward function. The reward function is used to reward or punish image matching results, which we define as If two pixels have depth information and are within the ε pixel range of their respective reprojections, we consider the match valid and reward the matching feature points, in which case the reward value is 1. If there is no depth information between the two pixels, we will calculate the outer pole distance from the pixel to the other pixel. If the distance is less than 2 pixels, we also consider the match to be reasonable. In this case, we do not penalize the pixel and set the reward value to -0.25. If neither of the above two conditions is met, the two pixels do not match, and we will penalize the feature points, setting the reward value to -0.001.

[0054] b) Strengthen reward maximization. Given the probability distribution of feature points, the distance distribution of feature matching, and the objective function, the gradient update formula is obtained to optimize the detection and description of feature points in the local feature extraction network:

[0055]

[0056] in, represents the gradient, r represents the reward function, and E represents the expectation. By rewarding the correct match and continuously updating the gradient, the final result is optimized.

[0057] The method of the present invention is trained on the megadepth dataset, which contains a variety of scenes from all over the world, and provides a high-quality depth map for each image, which can be used as the true value for local feature training. The present invention uses pytorch1.8+cuda11.1 to build the operating environment of the network model of the present invention, uses the Adam optimizer, sets the learning rate to 1e-4, the batch size to 4, and trains a total of 40 epoches. The training images are uniformly set to 768×768 size, the network model is trained on 2 GenForce RTX 3090s, and the tests are all performed on GenForce RTX 1080Ti.

[0058] In order to verify the effectiveness of the algorithm, the present invention is tested on the Hpatches dataset. According to the experimental results, the matching accuracy MMA of the algorithm proposed in the present invention reaches 0.858 for multiple scenes under the change of viewing angle, and reaches 0.75 for multiple scenes under the change of illumination. The overall matching accuracy MMA reaches 0.80.

Claims

1. A local feature learning method based on the combination of semantic guidance and task rewards and penalties, characterized in that: Obtain the feature detection heat map f of the original image through the local feature extraction network det and feature descriptor f des ; The original image is passed through the pre-trained semantic segmentation network to generate semantic feature classes; The semantic feature classes obtained by the semantic segmentation network and the feature detection heat map f output by the local feature extraction network det and feature descriptor f des The output results are respectively used to obtain the feature point detection distribution and the semantic feature descriptor of the feature point; the matching result of the semantic feature descriptor of the original image and the semantic feature descriptor of the target image is used as the training basis for whether the feature point is correct; the training method is reinforcement learning, and during the reinforcement learning training process, a gradient update formula is used to optimize the detection and description of the feature points of the local feature extraction network.

2. The local feature learning method based on the combination of semantic guidance and task rewards and penalties according to claim 1 is characterized in that: The gradient update formula is: in, represents the gradient, r represents the reward value of each feature point, and E represents the expectation; P′(A|f det ) and P′(B|f det ) represent the feature point detection distribution of the original image and the feature point detection distribution of the target image respectively; is the feature matching distribution; is the reward function; when the feature points of the two images have depth information and are within the ε pixel range of their respective reprojections, the match is considered valid and the matching feature points are rewarded. In this case, the reward function is defined as 1; there is no depth information between the two feature points, and the outer pole distance from the feature point to another feature point is calculated. If the distance is less than 2, the feature points are reasonably matched, and the reward function is defined as 0.25 at this time; if neither of the above two conditions is met, the two feature points do not match, and the reward function is defined as -0.

001.

3. The local feature learning method based on the combination of semantic guidance and task rewards and penalties according to claim 2 is characterized in that: The feature point detection distribution P′(·|f det ) Specifically, the detected feature points are classified by introducing semantic feature classes; for the visual task of outdoor scenes, fixed objects such as buildings are defined as long-term valid features, and the value of this type of feature points is set to 4, pedestrians are defined as invalid features, and the value of feature points is set to 1, grass and mountains are defined as short-term valid features, and the value of feature points is set to 2. For other objects in the training set involving some indoor scenes, but for features that are not important for testing, the value of feature points is set to 3; thus, the semantic reliability feature map S is constructed. mask : Among them, H s ' eg It represents the final result after semantic classification of feature points; when H s ' eg = 1, the feature is invalid, then S mask =0; when H s ' eg =2 or H s ' eg =3, S mask =0.5; when H s ' eg =4, then S mask =1; the feature detection heat map f det Perform softmax(·) operation and combine with S mask Multiply them together to get the final feature point detection distribution, which is expressed as follows: P′(x|f det )=softmax(f det (x))·S mask (f det (x)) (2)。 4. The local feature learning method based on the combination of semantic guidance and task rewards and penalties according to claim 3 is characterized in that: The feature matching distribution Specifically: According to the semantic reliability feature map S mask , feature descriptors are also divided into four categories, represented as It is called a semantic feature descriptor. When performing a matching operation, first determine whether the semantic feature descriptors belong to the same category. When the semantic feature descriptors belong to the same category, they are matched, otherwise they are not matched; the matching probability distribution calculation formula is as follows: Among them, M c represents the matching result between the original image A and the target image B, c represents the category, and They represent the semantic feature descriptors of the original image feature points and the target image feature points respectively.

5. The local feature learning method based on the combination of semantic guidance and task rewards and penalties according to any one of claims 1 to 4, characterized in that: The semantic segmentation network adopts the UperNet segmentation framework for semantic segmentation; the semantic segmentation network is an encoder-decoder structure, the encoder downsamples the output of the four network layers by 1 / 4, 1 / 8, 1 / 16 and 1 / 32, and the output channel is a feature map of [128, 256, 512, 1024], and the pyramid pooling module is used as a decoder to generate a feature map for semantic segmentation; the segmentation head H seg Process the feature map output by the decoder; The calculation process is as follows: E seg =Encoder seg (I) (4) H seg =Decoder seg (E seg ) (5) Encoder seg Decoder represents the encoder of the semantic segmentation network; seg Represents the decoder of the semantic segmentation network.

6. The local feature learning method based on the combination of semantic guidance and task rewards and penalties according to any one of claims 1 to 4, characterized in that: The local feature extraction network adopts the Unet framework of the encoding and decoding organizational structure; the original image I is input into the encoder for feature encoding to obtain the encoded feature E fea The encoded features are sent to the decoding part to obtain two outputs: the feature detection heat map f for feature point detection det and the feature descriptor f used for feature point description des ; E fea =Encoder fea (I) (6) f det ,f des =Decoder fea (E fea ) (7) Encoder fea Decoder represents the encoder of the local feature extraction network. fea Represents the decoder of the local feature extraction network.

Images