A deep learning-based image feature point detection and matching network
Through the improved SuperPoint network and adaptive spatial constraint layer, the problem of insufficient feature point detection accuracy and lack of spatial constraints is solved, and higher accuracy and robustness in three-dimensional reconstruction tasks are achieved to adapt to feature point matching in complex scenarios. .
Patent Information
- Application Number
- CN202210856359.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-20
- Publication Date
- 2024-12-17
- Estimated Expiration
- 2042-07-20
AI Technical Summary
The prior art has problems of insufficient accuracy and lack of explicit spatial constraints in feature point detection, especially in high-precision tasks and complex scenarios, traditional methods such as SIFT and ORB perform poorly when illumination changes and large-view angle changes.
The improved SuperPoint network is used to perform subpixel precision feature point detection, and combined with the adaptive spatial constraint layer and dynamic progressive consistent sampling module, feature points are extracted through a full convolutional network, and the attention mechanism and Sinkhorn algorithm are used to simulate the human matching process, which enhances the Accuracy and robustness of feature point matching.
The accuracy and robustness of feature point detection are significantly improved in the three-dimensional reconstruction task, especially in strong lighting changes and large-view angle changes, and have higher matching accuracy and stability than traditional methods.
Smart Images
Figure CN115115860B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer image processing, performs feature point detection and matching of images based on a deep learning method, and outputs matching feature point pairs and a spatial relationship matrix between two images. Background Art
[0002] Feature points refer to key points in an image that contain texture structure information and their corresponding descriptors. Feature point detection is divided into two steps: key point detection and descriptor calculation. Commonly used key point detection methods include Laplacian method, Harris corner detection method, Gaussian difference detection method, and FAST corner detection method. The SIFT operator detects key points in the DOG scale pyramid and calculates its 128-dimensional descriptor containing gradient direction information. The descriptor has scale invariance, rotation invariance, and affine invariance, and has very excellent positioning accuracy, but the calculation process is relatively complicated, so the operation speed is slow. Rublee proposed using improved FAST as a key point detector, combined with the improved BRIEF descriptor ORB operator, which uses neighborhood pixels to encode binary descriptors. It has a very fast operation speed while ensuring scale invariance, and has been applied in many real-time tasks. With the vigorous development of deep learning technology, researchers have also made continuous attempts and explorations in the field of feature point detection. SuperPoint is a self-supervised deep learning feature point extraction algorithm. It has good image understanding and feature point description capabilities, and its computing speed on GPU is much faster than traditional feature extraction algorithms. SuperGlue is an attention-based GNN matching network adapted for SuperPoint. It uses the attention mechanism to simulate the image matching process of humans. The disadvantage of SuperPoint is that the feature point coordinates are integers rather than floating-point numbers, so it is subject to certain limitations in tasks with high precision requirements. The disadvantage of SuperGlue is that when using GNN to simulate human vision, there is a lack of explicit spatial constraints. At present, the mainstream feature point detection tasks with 3D reconstruction as downstream tasks still rely on manual feature detection methods represented by SIFT. Summary of the invention
[0003] In order to overcome the deficiencies of the above-mentioned prior art, the present invention discloses a feature point detection and matching method based on deep learning. The present invention designs a fusion network based on an improved SuperPoint network and an improved SuperGlue network, which uses a fully convolutional network to extract image feature points, uses a sub-pixelation module to use neighborhood window information to improve the coordinate accuracy of feature points, and uses an attention mechanism to simulate the process of human feature point matching after jointly encoding image feature points and feature vectors, and uses the Sinkhorn algorithm to solve the matching relationship. The present invention designs an adaptive spatial constraint layer, which uses spatial constraint relationships to perform parallel screening and calculation of coarse matching point pairs using multiple methods, can adaptively distinguish the spatial relationship between images, and extract well-matched feature point pairs for the input image.
[0004] The technical steps adopted by the present invention are as follows:
[0005] Step 1: Use the improved SuperPoint network with sub-pixel accuracy to extract the fine feature point coordinates of the image to be matched.
[0006] Step 2: Extract feature points and descriptor joint encoding vectors based on sub-pixel precision feature point coordinates
[0007] Step 3: For the two images to be matched, perform steps 1 and 2 respectively to obtain two sets of joint vectors of feature points. Input the two sets of vectors into the attention graph neural network and the optimal matching layer of SuperGlue to obtain the matching relationship matrix, which describes the rough matching relationship between feature points and the corresponding confidence. Sort the matching relationships from high to low according to the confidence.
[0008] Step 4: The coordinates and matching relationship matrix of all feature points in the two images are input into the spatial constraint layer designed by the present invention. The spatial constraint layer includes parallel basic matrix constraints and homography matrix constraints. The homography matrix score is calculated according to the bidirectional reprojection error, and the reprojected HF spatial constraint model is used.
[0009] Step 4.1: Using the basic matrix as the fitting model, perform minimum median fitting on all matching point pairs to obtain the fitting matrix F1 and the set of matching point pairs corresponding to F1 when the projection error threshold is 2 pixels.
[0010] Step 4.2: Using the basic matrix as the fitting model, perform progressive consistent sampling fitting on all matching point pairs. According to the average confidence of the top 30% matching point pairs, dynamically allocate the number of iterations between 150 and 300 to obtain the fitting matrix F2 and the set of matching point pairs corresponding to F1 when the projection error threshold is 2 pixels.
[0011] Step 4.3: Using the homography matrix as the fitting model, perform minimum median fitting on all matching point pairs to obtain the fitting matrix H1 and the set of matching point pairs corresponding to H1 when the projection error threshold is 2 pixels.
[0012] Step 4.4: Using the homography matrix as the fitting model, perform progressive consistent sampling fitting on all matching point pairs. According to the average confidence of the top 30% matching point pairs, dynamically allocate the number of iterations between 150 and 300 to obtain the fitting matrix H2 and the matching point pair set corresponding to H2 when the projection threshold is 2 pixels. Steps 4.1, 4.2, 4.3, and 4.4 are run in parallel to save time.
[0013] Step 4.5: The homography matrix represents the relationship between the point and the projection point, and the distance between the projection point and the matching point is regarded as the projection error. The average bidirectional reprojection error is calculated based on the matching point pairs corresponding to H1 and H2. Similarly, the average bidirectional reprojection error corresponding to H2 is calculated. The smaller one is selected and the corresponding matrix is retained as H, and the corresponding average bidirectional reprojection error is recorded as SH.
[0014] Step 4.6: The basic matrix represents the relationship between the point and the projection epipolar line. The point-line distance between the projection epipolar line and the matching point is regarded as the projection error. The average two-way reprojection error is calculated based on the matching point pairs corresponding to F1 and F2. Similarly, the average two-way reprojection error corresponding to F2 is calculated. The smaller one is taken, and the corresponding matrix is retained as F, and the corresponding average two-way reprojection error is recorded as SF.
[0015] Step 4.7: When SH / (SF+SH)>0.4, retain the matrix H and output the set of matching point pairs corresponding to H; otherwise, retain the matrix F and output the set of matching point pairs corresponding to F.
[0016] Compared with the prior art, the present invention has the following beneficial effects:
[0017] (1) In the feature detection and matching tasks for 3D reconstruction, the proposed network has better accuracy performance
[0018] (2) Compared with traditional feature point detection and matching methods such as SIFT and ORB, it has better robustness in scenes with strong lighting changes and large viewing angle changes. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 The network structure diagram of the present invention
[0020] Figure 2 This is a structural diagram of the sub-pixel module of the present invention.
[0021] Figure 3 This is the descriptor decoder structure diagram of the present invention
[0022] Figure 4 The spatial constraint layer structure diagram of the present invention
[0023] Specific implementation steps
[0024] The present invention is further described below in conjunction with the accompanying drawings.
[0025] The present invention designs a fast feature point detection and matching network framework, improves the sub-pixel accuracy of the SuperPoint network for feature point detection, and adds an adaptive spatial constraint dynamic progressive consistent sampling module to the SuperGlue network for feature point matching. Figure 1 Shown is a network structure diagram of the present invention.
[0026] First, the SuperPoint network with sub-pixel accuracy is used to extract the fine feature point coordinates of the image to be matched. It is mainly divided into the following three steps:
[0027] 1. Input the image to be matched into the encoder of SuperPoint's VGG structure to extract the feature map of the image. The encoder contains 8 convolutional layers, 3 pooling layers, multiple BN layers and activation function layers. The number of convolution kernels in the convolutional layers is 64, 64, 64, 64, 128, 128, 128, 128. The convolutional layers are used to extract features. Three 2×2 maximum pooling layers downsample the H×W image to a height of H / 8 and a width of W / 8. After a H×W image is encoded by the encoder, a feature map of H / 8×W / 8×128 is obtained;
[0028] 2. Input the feature map obtained in step 1 into the feature point decoder of SuperPoint. The decoder contains two convolutional layers with channels of 256 and 65 respectively, multiple BN layers and activation function layers, and outputs a tensor of W / 8×H / 8×65. Each 65-dimensional tensor represents 65 cases in which the i-th pixel in a non-overlapping 8×8 pixel window of the original image is a feature point or the pixel window does not contain a feature point. The normalized probability distribution of the 65 cases is obtained through classification by the Softmax layer, and then the size is restored to H×W×1 through the ReShape layer to obtain an H×W score map. The pixel values of the score map are distributed between 0 and 1, representing the probability that each pixel point on the input image I is a feature point;
[0029] 3. In the original SuperPoint, the strategy for extracting feature points is to apply non-maximum suppression (NMS) to each N×N window in the output score map, retaining only one maximum value in each window, and then perform threshold judgment on the entire image, and regard points above the threshold as feature points. This output method only has integer-level coordinate accuracy and is not differentiable. The present invention adds a sub-pixel correction module that combines the neighborhood information of feature points to SuperPoint. The main process is as follows: Figure 2As shown. The score map obtained in step 2 is input into the coordinate sub-pixelization module designed by the present invention. The process includes: using non-maximum suppression for each non-overlapping 4×4 pixel window, setting the pixel values other than the maximum value in each pixel window to 0, and obtaining a coarse feature point map. Points in the coarse feature point map that are greater than a certain threshold are identified as coarse feature points. For all M coarse feature points, a 5×5 pixel window is taken in the score map with the coordinates of each coarse feature point as the center. The Softargmax method is used to calculate the expected deviation of each pixel in the pixel window relative to the center in the x and y directions respectively. The expected deviation is added to the coarse feature point coordinates to obtain M sub-pixelized fine feature point coordinates.
[0030] Then, the feature point and descriptor joint encoding vector are extracted according to the sub-pixel precision feature point coordinates, such as Figure 3 As shown in the figure. The descriptor decoder accepts a feature map of H / 8×W / 8×128 and outputs an initial descriptor matrix of H / 8×W / 8×256 after multiple convolutions. The original SuperPoint network uses bicubic interpolation to interpolate the initial descriptor matrix to H×W×256, and then uses L2 regularization to normalize the 256 channels. A 256-dimensional descriptor is calculated for each pixel of the original image I. In fact, it is unnecessary to calculate the descriptor of non-feature point pixels. The improved descriptor decoder in this paper performs M bilinear interpolations in the initial descriptor matrix based on the M sub-pixel coordinates output by the sub-pixel module to obtain M 256-dimensional vectors. These vectors are then L2 regularized to calculate the final 256-dimensional descriptor.
[0031] Secondly, the core idea of SuperGlue is to transform the feature point matching problem into the optimal transportation problem of the joint encoding vector of the feature point and the descriptor, and use the Sinkhorn algorithm to iteratively solve it. In this process, the attention GNN is used to simulate the repeated browsing characteristics of the human eye during matching, and the cross-attention and self-attention mechanisms are used to enhance the joint matching performance of position coordinates and visual descriptors. SuperGlue does not have explicit and strict spatial relationship constraints in the matching process. This paper improves SuperGlue to address this problem. A spatial constraint layer is added to SuperGlue, and a dynamic progressive consistent sampling module with adaptive HF spatial constraints is used to improve the matching accuracy of SuperGlue. Figure 4As shown, it is a structural diagram of the spatial constraint layer of the present invention. For two pictures to be matched, the above two steps are respectively executed to obtain two sets of joint vectors of feature points, and the two sets of vectors are input into the attention graph neural network and the optimal matching layer of SuperGlue to obtain the matching relationship matrix, which describes the rough matching relationship and the corresponding confidence between the feature points, and the matching relationship is sorted from high to low according to the confidence; finally, the coordinates of all feature points in the two pictures and the matching relationship matrix are input into the spatial constraint layer designed by the present invention, which includes parallel basic matrix constraints and homography matrix constraints, and the homography matrix score is calculated according to the bidirectional reprojection error, and the reprojected HF spatial constraint model.
Claims
1. An image feature point detection and matching network based on deep learning, characterized in that: The following steps are involved: The technical steps are as follows: Step 1: Use the SuperPoint network improved with sub-pixel accuracy to extract the fine feature point coordinates of the image to be matched; Step 1.1: Input the image to be matched into the encoder of SuperPoint's VGG structure to extract the feature map of the image. The encoder contains 8 convolutional layers, 3 pooling layers, multiple BN layers and activation function layers; the convolutional layer is used to extract features, and the number of convolution kernels is 64, 64, 64, 64, 128, 128, 128, 128, and three 2×2 maximum pooling layers are used for downsampling; a H×W image is encoded by the encoder to obtain a feature map of H / 8×W / 8×128; Step 1.2: Input the feature map obtained in step 1.1 into the feature point decoder of SuperPoint. The decoder contains two convolutional layers with channels of 256 and 65 respectively, multiple BN layers and activation function layers, and outputs a tensor of W / 8×H / 8×65; the normalized probability distribution of 65 cases is obtained through the Softmax layer classification, and then the size is restored to H×W×1 through the ReShape layer to obtain an H×W score map; Step 1.3: Input the score map obtained in step 1.2 into the designed coordinate sub-pixelization module. The process includes: using non-maximum suppression for each non-overlapping 4×4 pixel window, setting the pixel values other than the maximum value in each pixel window to 0, and obtaining a coarse feature point map; for all M coarse feature points, taking the coordinates of each coarse feature point as the center, taking a 5×5 pixel window in the score map, using the Softargmax method to calculate the expected deviation of each pixel in the pixel window relative to the center in the x and y directions respectively, adding the expected deviation to the coarse feature point coordinates, and obtaining M sub-pixelized fine feature point coordinates; Step 2: Extract feature points and descriptor joint encoding vectors based on sub-pixel precision feature point coordinates Step 2.1: Input the feature map obtained in step 1.1 into the SuperPoint descriptor decoder, and after multiple convolutions, output the descriptor matrix of H / 8×W / 8×256; Step 2.2: According to the M sub-pixel coordinates outputted from step 1.3, perform M bicubic interpolations on the descriptor matrix obtained from step 2.1 to obtain M 256-dimensional vectors, perform L2 regularization on these vectors, and obtain the 256-dimensional descriptors corresponding to the sub-pixel feature points; Step 2.3: Combine the fine feature point coordinates obtained in step 1.2 with the descriptor vector obtained in step 2.2 to obtain a joint feature point vector; Step 3: For the two images to be matched, perform steps 1 and 2 respectively to obtain two sets of joint vectors of feature points. Input the two sets of vectors into the attention graph neural network and the optimal matching layer of SuperGlue to obtain the matching relationship matrix; sort the matching relationships from high to low according to the confidence level; Step 4: Input the coordinates and matching relationship matrix of all feature points in the two images into the designed spatial constraint layer, calculate the homography matrix score based on the bidirectional reprojection error, and reproject the HF spatial constraint model; Step 4.1: Using the basic matrix as the fitting model, perform minimum median fitting on all matching point pairs to obtain the fitting matrix F1 and the matching point pair set corresponding to F1 when the projection error threshold is 2 pixels; Step 4.2: Using the basic matrix as the fitting model, perform progressive consistent sampling fitting on all matching point pairs. According to the average confidence of the top 30% matching point pairs, dynamically allocate the number of iterations between 150 and 300 to obtain the fitting matrix F2 and the matching point pair set corresponding to F1 when the projection error threshold is 2 pixels. Step 4.3: Using the homography matrix as the fitting model, perform minimum median fitting on all matching point pairs to obtain the fitting matrix H1 and the matching point pair set corresponding to H1 when the projection error threshold is 2 pixels; Step 4.4: Using the homography matrix as the fitting model, perform progressive consistent sampling fitting on all matching point pairs. According to the average confidence of the top 30% matching point pairs, dynamically allocate the number of iterations between 150 and 300 to obtain the fitting matrix H2 and the matching point pair set corresponding to H2 when the projection threshold is 2 pixels. Steps 4.1, 4.2, 4.3, and 4.4 are run in parallel to save time. Step 4.5: Calculate the average bidirectional reprojection error based on the matching point pairs corresponding to H1 and H2, and similarly calculate the average bidirectional reprojection error corresponding to H2. Take the smaller one, keep the corresponding matrix and record it as H, and the corresponding average bidirectional reprojection error as SH; Step 4.6: Calculate the average bidirectional reprojection error based on the matching point pair set corresponding to F1 and F2, and similarly calculate the average bidirectional reprojection error corresponding to F2. Take the smaller one, keep the corresponding matrix and record it as F, and the corresponding average bidirectional reprojection error as SF; Step 4.7: When SH / (SF+SH)>0.4, retain the matrix H and output the set of matching point pairs corresponding to H; otherwise, retain the matrix F and output the set of matching point pairs corresponding to F.
2. The method according to claim 1, characterized in that: Step 1 uses a sub-pixel coordinate module to extract refined feature point coordinates, giving the network more accurate feature extraction capabilities.
3. The method according to claim 1, characterized in that: After obtaining the descriptor matrix in step 2, the refined coordinates obtained in step 1 are substituted in, and the corresponding sub-pixel descriptor is obtained through bilinear interpolation and L2 regularization. The coordinates and the descriptor are combined to obtain a joint coding vector.
4. The method according to claim 1, characterized in that: After obtaining the matching relationship matrix in step 3, the matching point pairs are sorted according to the confidence level.
5. The method according to claim 1, characterized in that: The spatial constraint layer designed in step 4 uses the basic matrix and the homography matrix as fitting models in parallel. First, the number of iterations of the progressive consistent sampling module is dynamically set according to the first 30% confidence level for fitting. Then, the minimum median method is used for fitting in parallel. The bidirectional reprojection error of the internal points is used to retain the optimal fitting result, and the discriminant formula is designed to adaptively retain the correct fitting model.