A feature point detection and descriptor generation method based on deep learning

Abstract

A method for feature point detection and descriptor generation based on deep learning, comprising the following steps: S100: constructing a new convolutional neural network; S200: using the new convolutional neural network to predict the feature points of an input image and generate a descriptor vector; wherein the new convolutional neural network structure consists of two parts, the first part is an encoder for feature encoding of the input image; the second part is three decoders, the first decoder generates a descriptor feature map, the second decoder generates a unique feature map of feature points, and the third decoder generates an accuracy feature map by splicing neural network features from different layers as input. The feature points generated by this method have higher positioning accuracy and uniqueness, which helps to improve the performance of feature point matching and the accuracy of subsequent tasks.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for detecting feature points and generating descriptors based on deep learning. Background Art

[0002] In image processing, feature points refer to points where the grayscale value of an image changes dramatically or points with large curvature on the edge of an image. Feature points usually contain the location information of the point on the image, and the descriptor is usually a vector that describes the information of the pixels around the feature point. Feature point detection and description is one of the most basic and important research areas in the field of computer vision. The ability to accurately extract and describe feature points in an image is a prerequisite for many computer vision tasks, such as camera calibration, pose estimation, stereo matching, simultaneous localization and mapping (SLAM), 3D reconstruction, etc.

[0003] The design of feature point detectors needs to consider noise resistance, scale invariance, viewpoint invariance, illumination invariance, fast detection and description, etc. The design of traditional feature point detectors is complex, and generally only one or several of the above aspects can be taken into account. In particular, the robustness to illumination and scale changes is poor, and it is difficult to adapt to changing actual scenes. Due to the excellent performance of deep learning in current computer vision tasks, methods based on deep neural networks have gradually become a research hotspot in this field. At present, there are two representative general models of deep learning: 1) Methods such as SuperPoint and R2D2 share a feature encoding network at the front end, and the back end is divided into two branch decoding networks, which simultaneously detect feature points and generate descriptors. 2) Methods such as D2-Net and ASLFeat use a single network to extract image features, and the feature vector is directly used as the descriptor vector, and the position of the feature point is predicted by the feature vector. The above method has the following problems in practical applications: First, the positioning accuracy of feature points is insufficient. By visualizing the layers of the neural network, it is found that after multiple layers of convolution, the original detailed features of the image are lost and the edge positions are shifted, which leads to a decrease in the positioning accuracy of feature points; secondly, the uniqueness of feature points is insufficient, that is, there are a large number of points with similar descriptors in the extracted feature points, and similar points will cause a large number of mismatches in subsequent tasks, which in turn leads to a decrease in the accuracy of pose estimation. Summary of the invention

[0004] In view of this, the present disclosure provides a method for feature point detection and descriptor generation based on deep learning, comprising the following steps:

[0005] S100: Building a new convolutional neural network;

[0006] S200: using the new convolutional neural network to predict feature points of the input image and generate a descriptor vector;

[0007] Among them, the new convolutional neural network structure consists of two parts. The first part is the encoder, which is used to encode the features of the input image; the second part is 3 decoders. The first decoder generates a descriptor feature map, the second decoder generates a unique feature map of the feature points, and the third decoder uses the concatenation of neural network features from different layers as input to generate an accuracy feature map.

[0008] Through the above technical solution, this method uses a neural network to predict the feature points of the input image and generate a descriptor vector. Compared with other deep learning methods, the feature points generated by this method have higher positioning accuracy and uniqueness, which helps to improve the performance of feature point matching and the accuracy of subsequent tasks.

[0009] The beneficial effects of this method are as follows: First, to address the problem of insufficient positioning accuracy of feature points, this method uses the operation of fusion of shallow and deep features to introduce underlying detail features into the feature point detector, and uses the gradient of the image as the loss function, thereby improving the positioning accuracy of the detector. Secondly, to address the problem of insufficient uniqueness of feature points, a uniqueness loss function is designed to compare the descriptors of the assumed feature points with all other feature points in the image to measure their uniqueness. Finally, using the HPatches dataset and the Aachen-Day and Night dataset, the excellent performance of this method in specific tasks such as image matching and visual positioning is fully demonstrated. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flow chart of a method for feature point detection and descriptor generation based on deep learning provided in one embodiment of the present disclosure;

[0011] Figure 2 is a structural diagram of a method for feature point detection and descriptor generation based on deep learning in one embodiment of the present disclosure;

[0012] Figure 3(a) to Figure 3(c) is a visualization diagram of feature maps of different layers in one embodiment of the present disclosure;

[0013] FIG. 4( a ) and FIG. 4( b ) are schematic diagrams of a grid image and a simple geometric image in one embodiment of the present disclosure;

[0014] Figure 5(a) to Figure 5(c) It is a comparison diagram of the thermal diagrams of R2D2 and this method in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] See also Figure 1In one embodiment, a method for feature point detection and descriptor generation based on deep learning is disclosed, comprising the following steps:

[0016] S100: Building a new convolutional neural network;

[0017] S200: using the new convolutional neural network to predict feature points of the input image and generate a descriptor vector;

[0018] Among them, the new convolutional neural network structure consists of two parts. The first part is the encoder, which is used to encode the features of the input image; the second part is 3 decoders. The first decoder generates a descriptor feature map, the second decoder generates a unique feature map of the feature points, and the third decoder uses the concatenation of neural network features from different layers as input to generate an accuracy feature map.

[0019] In this embodiment, shallow features are introduced into the detector through multi-layer feature fusion to solve the problem of position offset of feature points; and a deep and shallow feature loss function is proposed to improve the positioning accuracy of feature points. The loss function is designed to evaluate the uniqueness of feature points, and only points with high uniqueness are retained to prevent subsequent mismatching. This method is evaluated on general image matching and visual positioning datasets. The experimental results show that it is better than existing methods. The average matching accuracy on the HPatches dataset is improved to 73.2%, and the high-precision positioning part on the Aachen day-night dataset is also significantly improved.

[0020] Determine the structure of the neural network as Figure 2 As shown in the figure, it consists of two parts. The first part is the decoder. The decoder is a feature extraction layer shared by the front end. It is composed of a fully convolutional L2-Net and is used for feature encoding of the input image. The second part consists of three decoder branches. The first decoder generates a feature map of the descriptor, the second decoder generates a unique feature map of the feature points, and the third decoder is concatenated with neural network features from different layers as input, and finally generates an accuracy feature map.

[0021] In another embodiment, the encoder is composed of 9 convolutional layers, the first 6 convolutional layers use a convolutional kernel of size 3, and the last three convolutional layers use a convolutional kernel of size 2.

[0022] In another embodiment, dilated convolution is used in the encoder to replace the general convolution and downsampling process.

[0023] In this embodiment, in order to ensure that the resolution of the feature layer remains unchanged while enhancing the receptive field, we use dilated convolution instead of the general convolution and downsampling process.

[0024] In another embodiment, the concatenation of neural network features from different layers as input specifically refers to concatenation of features from the first layer, the second layer, the third layer, and the ninth layer of the encoder as input.

[0025] For this embodiment, the second part of the neural network includes 3 decoder branches, which are decoder networks for different learning tasks. The first decoder generates a descriptor feature map, which describes each pixel of the image with a D-dimensional vector. The second decoder generates a unique feature map of the feature point, which can exclude points with similar descriptors and prevent mismatching of feature points. The third decoder generates an accuracy feature map of the feature point, which can provide accurate and repeatable locations of feature points. In order to improve the accuracy of feature point positioning and ensure that the feature map contains the perception of shallow detail features, this part uses feature fusion operations. Figure 2 The figure shows the feature fusion operation of concatenating the outputs of the first, second, third, and ninth convolutional layers.

[0026] In another embodiment, a deep-shallow layer feature loss function is used to train the accuracy feature map.

[0027] In this embodiment, FIG3(a) is a visualization of the feature map of the first layer of the neural network, FIG3(b) is a visualization of the feature map of the fifth layer of the neural network, and FIG3(c) is a visualization of the feature map of the ninth layer of the neural network. Usually, the feature point detection algorithm based on deep learning generates the feature point response map of the image through a multi-layer convolutional network, and then selects the points with larger response values ​​as feature points. However, if Figure 3(a) to Figure 3(c) As shown in Figure 1, the visualization results generated by different layers of the neural network show that after multiple layers of convolution, the detailed features of the image have become blurred and offset. Therefore, this shows that only detecting feature points on high-level features will lead to a decrease in the positioning accuracy of feature points.

[0028] And from Figure 3(a) to Figure 3(c) It can be found from the visualization results that the shallow neural network has a good retention of detail features. Therefore, this method introduces shallow features to constrain high-level features, thereby improving the position accuracy of feature points. Figure 2 As shown, the first, second, third and ninth layers of the neural network are subjected to feature concatenation operations, and the positions of feature points are predicted on this basis.

[0029] In addition, shallow features are introduced into the loss function to enhance the detail information of the feature map. This method helps the detector locate feature points without degrading the performance of the descriptor.

[0030] Specifically, this method designs a deep and shallow feature loss. We let I and I′ be two images of the same scene, and let is the true value correspondence between them, H and W are the length and width of the image respectively. Inspired by Key.Net, we calculate the first-order and second-order derivatives of images I and I′ in different directions, and obtain the following underlying features:

[0031]

[0032] I x , I y , I x ×I y , and Respectively represent the first-order derivatives of the image in x and y and their combinations, I xx , I yy , I xy , I xx ×I yy and It is the second-order derivative of the image in the x, y and xy directions and their combination, and the above gradient information is combined as the underlying feature LF. Then the loss function is designed to make the accuracy feature map A correspond to the underlying feature LF. We use the cosine similarity between A and LF to measure the deep and shallow feature loss. (hereinafter represented by coshl), that is, we define a set of overlapping images P = {p}, which contains all N×N image blocks {1,...,W}×{1,...,H}, and define the loss as:

[0033]

[0034] Among them, A represents the accuracy feature map, LF represents the underlying feature, is a set of partially overlapping images, A[p] and LF[p] represent the flat N×N features extracted from A and LF, and in this experiment, the value of N is 8. coshl represents the cosine similarity between A and LF. In addition, as in R2D2, we also maximize the cosine similarity between A and A′ (hereinafter referred to as cosim) to ensure that the feature points in the image can be detected repeatedly:

[0035]

[0036] where A[p] and A′[p] represent the flat N×N blocks extracted from A and A′. Finally, the total accuracy loss is as follows:

[0037]

[0038] In another embodiment, the uniqueness loss function is:

[0039]

[0040] Here, we use a pair of matched images I and I' as input, x is a pixel of image I, D(x) is the uniqueness response of point x, K is the number of points; nx is the uniqueness index calculated using the descriptor, n x =∑ n∈N，p∈P 1(cos(M x , M′ n )>max(cos(M x , M′ p ), x is the feature point in the input image I, M x is the descriptor vector of x; set N is the set of negative sample points of a feature point x in image I in the input image I', M' n is the descriptor vector of the negative sample point; set P is the set of positive sample points of feature point x, M′ p is the descriptor vector of the positive sample point; cos(·) means calculating the cosine similarity between the two, and 1(·) means outputting 1 when the internal inequality holds, otherwise outputting 0.

[0041] In this embodiment, in order to remove feature points with a large degree of similarity and thus improve the quality of feature point extraction by the detection algorithm, this method evaluates the uniqueness of each feature point and thus proposes a uniqueness loss function. x The more unique, the x The smaller the value of D x The closer it is to 1; on the contrary, if there are many descriptor vectors in image I′ and M x Similar, then n x The larger the D x Close to 0. x represents the uniqueness response of point x, and K represents the number of points.

[0042] In another embodiment, the average precision AP of a differentiable approximation is used as a loss function to train the descriptor feature map.

[0043] In this embodiment, in order to train the descriptor and optimize the distance between its positive and negative samples, the average precision AP of differentiable approximation is used as the loss function of the descriptor. Compared with the triplet loss function, the litewise optimization method used by the AP loss function is easier to optimize. This is because the listwise sorting only needs to ensure that the matched image blocks are before all the unmatched image blocks, while ignoring the sorting between the correct matches and the unmatched ones. Specifically, for image I, each D-dimensional descriptor M x They all describe a k×k image patch centered on x. In this experiment, k is set to 16. In image I′, there are x The corresponding descriptor M x′. We hope that M x With M x ′ is as similar as possible, so we need to maximize AP:

[0044]

[0045] Where I and I' represent the matched images, x represents a pixel in image I, and M x and M′ x Represents the image blocks in images I and I'. U represents the mapping relationship between I and I', K is the number of points, and AP(·) represents the approximate average precision value of the calculated image block.

[0046] In another embodiment, a uniqueness loss function is used to train the uniqueness feature map.

[0047] In this embodiment, in addition to the accuracy feature map, the neural network also outputs a unique feature map to solve the problem of similar feature points. The explanation of the unique feature map is as follows: For the grid image shown in Figure 4(a), the intersection of the grid lines is usually extracted as a feature point, but the images near the intersection are similar to each other. If they are extracted as feature points, they may cause mismatches in subsequent image matching tasks, resulting in a decrease in the accuracy of pose estimation. Therefore, we hope that the uniqueness response value at the grid intersection is relatively low, so that the extraction of the intersection as a feature point can be avoided. For the image shown in Figure 4(b), the three vertices of the triangle are usually extracted as feature points, and the similarity between the vertices is relatively low. We hope that the uniqueness response values ​​at these three points are relatively high, so that extracting these three points as feature points can avoid mismatching of images.

[0048] The overall loss function of the descriptor is as follows:

[0049]

[0050] Where I and I′ represent the matching image pairs, M and M′ represent the descriptor feature maps corresponding to the image pairs I and I′, U represents the correspondence between the image pairs, x represents a pixel in image I, x′ represents the position of the pixel in image I′, and D and D′ represent the unique feature maps corresponding to the image pairs I and I′.

[0051] In summary, the final overall loss function is shown in the following formula, which consists of two parts: feature point detection loss and descriptor loss:

[0052] In another embodiment, during the training of the convolutional neural network, a data enhancement method is used to preprocess the data set to generate a training set.

[0053] In this embodiment, the training data set used by the method is mainly composed of the Aachen data set, and the data set is expanded by using the data enhancement method. The test set uses the Hpatches data set, which contains a total of 108 scenes (including 52 lighting changes and 56 viewing angle changes).

[0054] This paper uses data enhancement methods to preprocess the Oxford and Paris retrieval dataset and the Aachen Day-Night dataset, a visual positioning dataset, to generate training sets. Among them, the Oxford and Paris retrieval dataset is subjected to homography transformation, which is mainly used to simulate the change of perspective. The generated new image and the original image are used as a pair of matching images for training; three data enhancement operations are performed on the Aachen Day-Night dataset. The first is homography transformation, which is also used to simulate the change of perspective. The second is image style transfer and homography transformation, which are mainly used to simulate the changes in lighting and perspective. The third is to use the SFM model and Epicflow method to obtain the best matching image pair from the same scene to simulate the lighting and perspective changes in actual scenes.

[0055] In another embodiment, during the experiment, we used the Adam optimizer to optimize the training process to 25 epochs, fixed the learning rate to 0.0001, the weight decay to 0.0005, the batch size to 4, and the image cropping to 192 × 192. In addition, our experiment was implemented by Pytorch, and a single NVIDIA RTX 2080Ti card was used to complete training and testing, and the training time was within 12 hours.

[0056] for and The image sampling problem involved is that in image I′, we define the positive sample position as within a 3-pixel radius from the true position, and the negative sample position as between 9 and 11 pixels from the true position. The randomly sampled position is outside 11 pixels.

[0057] During testing, we use a multi-scale detection method to detect feature points on the input image. We start with L = 1024 pixels on the input image and perform multiple detections on the trained network at different scales, performing 2 1 / 4 Downsample until L < 256 pixels, where L is the image size. We use non-maximum suppression (NMS) of size 3 to remove detections that are too close in space. For each scale, we find the local maximum in R and take the uniqueness of the feature point from the corresponding position in D and the descriptor M from the corresponding position in M. x , M xThe score is A x D x Finally, we retain the best K descriptors at all scales, that is, we get accurate and unique feature points.

[0058] Figure 5(a) to Figure 5(c) In Figure 5(a), we use a checkerboard image as input and visualize the location, accuracy, and uniqueness heatmaps of the feature points detected by R2D2 and our method. The lighter the color in the image, the more accurate or unique the point is. From the feature point location image in Figure 5(a), we can see that compared with R2D2, our method can obtain more accurate feature point locations, and the extracted feature points are all located in the edge area of ​​the image; from the uniqueness heatmap in Figure 5(b), we can see that the uniqueness of our method can better eliminate areas with the same texture; from Figure 5(c), we can see that the accuracy heatmap generated by our method can better determine the edge and intersection positions of the checkerboard, while R2D2 has a certain deviation. In addition, we also conducted image matching and visual positioning experiments to explore the performance of the detector.

[0059] In order to evaluate the experimental results, an image matching task was implemented on the Hpatches dataset to quantitatively evaluate the proposed method. The image matching task is a common standard for evaluating the performance of feature point detectors and is also an important application of feature point detection. The Hpatches dataset contains a total of 116 scenes. In order to compare with popular detectors such as R2D2 and D2-Net, we selected 108 scenes (including 52 illumination change and 56 viewpoint change scenes).

[0060] The evaluation indicators mainly include Mean Matching Accuracy (MMA), Repeatability and Matching score. Mean Matching Accuracy is an indicator used to comprehensively evaluate the position of feature points and the performance of descriptors. That is, the descriptors of the first image and the second image are matched, and the point with the closest matching distance is calculated to see if it is a feature point pair; Repeatability is an indicator used to evaluate the position performance of feature points. The distance between the points extracted from the two images is calculated to represent the repetition rate, that is, the probability that a feature point in the first image is detected in the second image; Matching score is the average ratio of the true value corresponding points that can be restored by the entire neural network to the number of feature points in the common field of view.

[0061] The method is compared with DELF, Superpoint, LF-Net, etc., and the results are shown in Table 1. It can be seen that the performance of this method is significantly better than that of existing methods under different error thresholds. For example, the MMA of this method under the error thresholds of 1 pixel and 3 pixels is better than R2D2, which shows that this method can effectively improve the positioning accuracy of feature points.

[0062]

[0063] Table 1

[0064] We also used the original R2D2 models (WSF_N16, WASF_N16) for comparison. The results are shown in Table 2. At an error threshold of 3 pixels, the repeatability of our method is 68.47%, compared with 61.83% of WSF_N16 and 63.56% of WASF_N16, which proves the advantage of our method. In addition, the matching score and repeatability of our method are also higher than those of R2D2.

[0065]

[0066] Table 2

[0067] Finally, we use the visual localization benchmark to evaluate the performance of our detector on practical computer vision tasks. Visual localization is the problem of estimating the 6-DOF camera pose of a given image relative to a reference scene representation. The evaluation dataset we use is the Aachen Day-Night dataset. This dataset contains 4328 reference images and 98 night query images. We estimate the corresponding poses of the 98 night query images based on the reference images and their poses. The results are the percentage of query images that are localized within three given translation and rotation error tolerances, namely (0.25m, 2°), (0.5m, 5°), and (5m, 10°).

[0068] The results are shown in Table 3. It can be found that our method outperforms other methods under high precision conditions and is significantly better than R2D2(WASF_N16).

[0069]

[0070] Table 3

[0071] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.

Claims

1. A method for feature point detection and descriptor generation based on deep learning, comprising the following steps: S100: Building a new convolutional neural network; S200: using the new convolutional neural network to predict feature points of the input image and generate a descriptor vector; in, The new convolutional neural network structure consists of two parts. The first part is an encoder, which is used to encode the features of the input image. The second part consists of three decoders. The first decoder generates a descriptor feature map, the second decoder generates a unique feature map of feature points, and the third decoder concatenates the neural network features of different layers as input to generate an accuracy feature map. Among them, shallow features are introduced into the detector through multi-layer feature fusion, and shallow features are introduced into the loss function; The concatenation of neural network features from different layers as input specifically refers to: concatenation of features from the first layer, the second layer, the third layer, and the ninth layer of the encoder as input; Using deep and shallow layer feature loss functions to train the accuracy feature map; The deep and shallow layer feature loss function is: ; in, represents the accuracy feature map, Represents the underlying features, is a set of partially overlapping images, and Indicates from and Flat Features, express and The cosine similarity between , N is an integer; Using a uniqueness loss function to train the uniqueness feature map; The uniqueness loss function is: ; Where I and I' represent a pair of matching input images, x is a pixel of input image I, D(x) is the uniqueness response of pixel x, and K is the number of pixels; A uniqueness metric calculated using descriptors.

2. According to the method of claim 1, the encoder is composed of 9 convolutional layers, the first 6 convolutional layers use a convolutional kernel of size 3, and the last three convolutional layers use a convolutional kernel of size 2.

3. According to the method of claim 1, dilated convolution is used in the encoder to replace general convolution and downsampling processes.

4. According to the method of claim 1, the average precision AP of differentiable approximation is used as the loss function to train the descriptor feature map.

5. According to the method of claim 1, during the training process of the convolutional neural network, a data enhancement method is used to preprocess the data set to generate a training set.

Images