SuperPoint lightweight model for accelerating image feature extraction

Abstract

The invention discloses a SuperPoint lightweight model for accelerating image feature extraction, and belongs to the field of image processing. The method comprises the following steps: firstly, by adopting a knowledge distillation method, taking a SuperPoint original model as a teacher model, and carrying out weight reduction on the SuperPoint original model to obtain a student model; secondly, feature points of the original data set are amplified, and a hard tag of the data set is constructed; thirdly, constructing a soft label by using the output of the teacher model and the hard label information; meanwhile, an SE attention mechanism is added into the student model, so that the performance of the lightweight model is further improved; and finally, training the student model to obtain a SuperPoint lightweight model. According to the model, the parameter quantity is greatly reduced, and the original network performance is approached. And compared with the original network model, the parameter quantity is reduced to 105k from 1304k, and is reduced by 91.95%. In an actual test based on 100 pairs of images, the matching speed of the lightweight model is improved by 95.45%, and the matching accuracy and the matching score are only reduced by 1.74% and 1.73% respectively.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, is mainly aimed at image feature extraction and matching applications, and relates to a SuperPoint lightweight model for accelerating image feature extraction. Background Art

[0002] Image feature extraction has important applications in many fields such as image matching, image stitching, and visual navigation. In recent years, with the rapid development of deep learning, image feature extraction methods have evolved from manual feature extraction methods such as SIFT and ORB to deep learning methods such as KeyNet and SuperPoint models. Among them, the SuperPoint model, as a self-supervised feature extraction network, has shown potential to outperform traditional methods in a variety of visual tasks and has been widely used. However, because SuperPoint is a network model based on deep learning, the image feature extraction speed is slow by deepening the number of network layers to improve accuracy, and its application in practical scenarios is limited.

[0003] Model lightweighting is one of the effective solutions to the above problems. Typical methods include model quantization, model pruning, knowledge distillation, etc. Among them, model quantization converts the network weights, activation values, etc. from high precision to low precision; model pruning prunes unimportant neurons in the model to form a model with less computational complexity; the knowledge distillation method transfers the knowledge of a complex teacher model to a smaller student model, making it close to the teacher model in performance. In comparison, the knowledge distillation method has more advantages in ensuring lightweight performance. Summary of the invention

[0004] In order to obtain the feature extraction performance of SuperPoint and improve its feature extraction speed, the present invention adopts the knowledge distillation method, takes the original SuperPoint model as the teacher model, and lightweights it to obtain a SuperPoint lightweight model that accelerates image feature extraction. At the same time, the feature points of the original data set are amplified to construct the hard labels of the data set; the output of the teacher model and the hard label information are used to construct soft labels to further improve the performance of the lightweight model.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A SuperPoint lightweight model for accelerating image feature extraction includes the following steps:

[0007] Step 1: Dataset hard label expansion

[0008] Using the COCO2014 dataset, the self-supervised module MagicPoint in the original SuperPoint teacher model first extracts feature points from the dataset. Then, the traditional SIFT method is used to extract feature points from the dataset. Finally, the feature points of the two are fused to form a new dataset label, which is the hard label for training the SuperPoint lightweight model.

[0009] Step 2: Soft label construction

[0010] In the process of feature extraction of COCO2014 dataset by SuperPoint original model, the distribution probability of feature points is used as soft labels for training lightweight models. For the hard labels obtained in the first step, the intersection with the feature points extracted by SuperPoint original model is taken, and the distribution probability of the intersection feature points is set to 1, while the distribution probability of the remaining feature points remains unchanged.

[0011] Step 3: Student Model Construction

[0012] Modify the Encoder network of the original teacher model to construct a lightweight Encoder network of the student model, and reduce the dimension of the feature vector output by the original teacher model from the original 256 dimensions to 64 dimensions; add a channel attention mechanism SE (Squeeze-and-Excitation Networks) module ("J.Hu, L.Shen and G.Sun,"Squeeze-and-ExcitationNetworks,"2018IEEE / CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp.7132-7141") to enhance the feature representation between channels and improve the model performance; the specific lightweight Encoder network is shown in Table 1:

[0013] Table 1 Lightweight Encoder network with SE

[0014] Input Dimensions Convolution kernel size Output Dimensions Step Length SE Attention 1 3×3 4 1 4 1×1 8 1 8 3×3 8 2 8 3×3 8 1 8 1×1 24 1 24 3×3 24 2 24 3×3 24 1 √ 24 1×1 64 1 64 3×3 64 2 64 3×3 64 1 √

[0015] Among them, “√” represents adding the SE attention mechanism module after the convolution layer.

[0016] Step 4: Introduce knowledge distillation loss function.

[0017] The knowledge distillation loss function is defined as follows:

[0018]

[0019] Where N represents the total number of pixels in the image, M represents the valid mask in the teacher model, t represents the probability of the pixel in the teacher model, and s represents the probability of the pixel in the student model.

[0020] Finally, the total loss function is: L total =L p +L distillation +λL d ⑵

[0021] Where L p is the feature point loss function in the teacher model, L d is the feature descriptor loss function in the teacher model, and λ is the weight coefficient set in the teacher model.

[0022] Step 5: Train the model. The training framework is tensorflow. The training parameters mainly include: batch size, learning rate, number of training iterations, and feature point extraction threshold.

[0023] Step 6: Test the model. Test performance indicators such as matching time, matching accuracy, and matching score, which are defined as follows:

[0024] (1) Matching time

[0025] The overall matching process includes image preprocessing, feature point detection, feature point description, feature point matching, and false matching elimination. The sum of the time consumed by these processes is equal to the matching time, which is used to measure the efficiency of the image matching algorithm.

[0026] (2) Matching accuracy

[0027] The matching accuracy Acc refers to the ratio of the number of remaining correct matching point pairs to the total number of matching point pairs after removing the wrong matching point pairs. The higher the matching accuracy, the more accurate the feature description, which can indicate the effectiveness of the feature matching algorithm:

[0028] Among them, N right is the number of correct matching point pairs remaining after removing the wrong matching point pairs, N total is the number of all matching point pairs.

[0029] (3) Matching score

[0030] The matching score refers to the ratio of the number of correctly matched point pairs to the number of smaller feature points in the two images. The higher the matching score, the higher the quality of the detected feature points, which can characterize the effectiveness of the feature point detector:

[0031] Among them, N right is the number of correctly matched point pairs, N 1Represents the number of feature points detected in the first image, N 2 Represents the number of feature points detected in the second image, min(N 1 ,N 2 ) refers to the smaller number of feature points in the two images.

[0032] The present invention re-modifies the original SuperPoint network architecture to reduce dimension, expand hard labels, construct soft labels, add channel attention mechanism, and use knowledge distillation to train lightweight models. The SuperPoint lightweight model is designed to greatly reduce the number of parameters while approaching the original network performance. Compared with the original SuperPoint network model, the number of parameters is reduced from 1304k to 105k, a reduction of 91.95%. In the test based on GPU2080Ti, the average matching time of two images is reduced from the original 0.43s to 0.22s, and the average matching speed is increased by 95.45%. Figure 2 As shown in Figure 2, the average matching accuracy dropped from 54.12% to 52.38%, a decrease of only 1.74%. Figure 3 As shown in the figure, the average matching score dropped from 34.69% to 32.96%, a decrease of only 1.73%. Figure 4 shown. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a lightweight network model of SuperPoint;

[0034] Figure 2 It is a line graph of the matching time of 100 pairs of test images;

[0035] Figure 3 This is a line graph of the matching accuracy of 100 pairs of test images;

[0036] Figure 4 The figure is a line graph of the matching scores of 100 pairs of test images. DETAILED DESCRIPTION

[0037] An embodiment of the present invention is described in detail below in conjunction with the technical solution. The test images are 100 pairs of actual images taken by drones, including scenes of cities, deserts, rivers, mountains, etc., and the image resolution is 640*480.

[0038] Step 1: Dataset hard label expansion. The COCO2014 dataset with 82,783 images is used. First, the self-supervisory module MagicPoint in the original SuperPoint model extracts feature points from the dataset. Then, the SIFT method is used to extract feature points from the dataset, and the extraction threshold is set to 0.10. Finally, the feature points of the two are fused to form a new dataset label, which is the hard label for training the SuperPoint lightweight model.

[0039] Step 2: Soft label construction. In the process of feature extraction of COCO2014 dataset by SuperPoint original model, the distribution probability of feature points is used as soft labels for training lightweight models; for the hard labels obtained in the first step, the intersection with the feature points extracted by SuperPoint original model is taken, and the distribution probability of the intersection feature points is set to 1, while the distribution probability of the remaining feature points remains unchanged.

[0040] Step 3: Student model construction. Modify the Encoder network of the original SuperPoint model to construct a lightweight Encoder network of the student model, reduce the feature vector dimension of the original teacher model output from 256 dimensions to 64 dimensions; add a channel attention mechanism SE module; the specific lightweight Encoder network is shown in Table 1.

[0041] Step 4: Introduce knowledge distillation loss function L distillation , as shown in formula (1); the total loss function is shown in formula (2), where the weight coefficient λ is set to 0.0001.

[0042] Step 5: Train the model. The training framework is the tensorflow framework, and the hardware equipment is two GPU2080Ti. The main training parameters are set as follows: batch size is 2, learning rate is 0.0001, training iterations are 400000, and feature point extraction threshold is 0.01.

[0043] Step 6: Test the model. Test performance indicators such as matching time, matching accuracy, and matching score. The specific formulas for matching accuracy and matching score are shown in formulas ⑶ and ⑷ respectively. The specific process is as follows:

[0044] First, 100 pairs of images are input into the lightweight network in turn to extract feature points and generate 64-dimensional descriptors. Then, the fast nearest neighbor matching method is used to match the feature points of the two images, and the matching threshold is set to 0.7. Finally, the matching time, matching accuracy and matching score of the 100 pairs of images are recorded respectively. In the end, the SuperPoint lightweight network model designed by the present invention reduces the number of parameters from 1304k to 105k, a reduction of 91.95% compared to the original network model. In the actual test of 100 pairs of images based on GPU2080Ti, the average matching time of the two images was reduced from the original 0.43s to 0.22s, and the average matching speed was increased by 95.45%. Specifically, Figure 2 As shown in Figure 2, the average matching accuracy dropped from 54.12% to 52.38%, a decrease of only 1.74%. Figure 3 As shown in the figure, the average matching score dropped from 34.69% to 32.96%, a decrease of only 1.73%. Figure 4 shown.

Claims

1. A SuperPoint lightweight model for accelerating image feature extraction, characterized by Follow these steps: Step 1: Dataset hard label expansion Using the COCO2014 dataset, the self-supervised module MagicPoint in the original SuperPoint teacher model first extracts feature points from the dataset; then, the traditional SIFT method is used to extract feature points from the dataset; finally, the feature points of the two are fused to form a new dataset label, which is the hard label for training the SuperPoint lightweight model; Step 2: Soft label construction In the process of feature extraction of COCO2014 dataset by SuperPoint original model, the distribution probability of feature points is used as soft labels for training lightweight models; for the hard labels obtained in the first step, the intersection with the feature points extracted by SuperPoint original model is taken, and the distribution probability of the intersection feature points is set to 1, and the distribution probability of the remaining feature points remains unchanged; Step 3: Student Model Construction The Encoder network of the original teacher model is modified to construct a lightweight Encoder network of the student model, and the dimension of the feature vector output by the original teacher model is reduced from the original 256 dimensions to 64 dimensions; a channel attention mechanism SE module is added to enhance the feature representation between channels and improve the model performance; the lightweight Encoder network is shown in Table 1: Table 1 Lightweight Encoder network with SE Among them, "√" represents adding the SE attention mechanism module after the convolution layer; Step 4: Introduce knowledge distillation loss function; The knowledge distillation loss function is defined as follows: Where N represents the total number of pixels in the image, M represents the valid mask in the teacher model, t represents the probability of the pixel in the teacher model, and s represents the probability of the pixel in the student model; The total loss function is: L total =L p +L distillation +λL d (2) Among them, L p is the feature point loss function in the teacher model, L d is the feature descriptor loss function in the teacher model, and λ is the weight coefficient set in the teacher model; Step 5: Training the model The training framework is tensorflow framework; the training parameters mainly include: batch size, learning rate, number of training iterations, feature point extraction threshold; Step 6: Test the model Test matching time, matching accuracy, and matching score, which are defined as follows: (1) Matching time The overall matching process includes image preprocessing, feature point detection, feature point description, feature point matching, and false matching elimination. The sum of the time consumed by these processes is equal to the matching time, which is used to measure the efficiency of the image matching algorithm; (2) Matching accuracy The matching accuracy Acc refers to the ratio of the number of remaining correct matching point pairs to the total number of matching point pairs after removing the wrong matching point pairs. The higher the matching accuracy, the more accurate the feature description, indicating the effectiveness of the feature matching algorithm: Among them, N right is the number of correct matching point pairs remaining after removing the wrong matching point pairs, N total is the number of all matching point pairs; (3) Matching score The matching score refers to the ratio of the number of correctly matched point pairs to the number of smaller feature points in the two images. The higher the matching score, the higher the quality of the detected feature points, which can characterize the effectiveness of the feature point detector: Among them, N right is the number of correctly matched point pairs, N1 represents the number of feature points detected in the first image, N2 represents the number of feature points detected in the second image, and min(N1, N2) refers to the smaller number of feature points in the two images.

Images