Priori knowledge assisted image feature point detection method

Abstract

The invention discloses a priori knowledge-assisted image feature point detection method, which comprises the following steps of: firstly, in order to extract multi-scale features, performing two times of down-sampling on an input picture to obtain three groups of images; the three groups of images respectively pass through a feature extraction module and a plurality of groups of feature fusion modules to obtain final response diagrams; the feature extraction module is composed of a main branch and a sub-branch; the main branch takes a Sobel edge detection result as input, partial parameters learn a DoG extreme value result, and depth features and priori knowledge are learned at the same time; a sub-branch uses Gaussian filtering to construct a scale pyramid and takes a DoG space as a label for learning; in order to realize the fusion of deep features and prior knowledge, the feature representation of the main branch and the feature representation of the sub-branch are fused through a proposed knowledge fusion module. After priori knowledge is introduced, the method has huge potential in the aspect of improving feature point detection performance and model robustness, and a new research direction is provided for feature point detection and matching tasks in future computer vision.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an image feature point detection method assisted by prior knowledge. Background Art

[0002] Image feature point detection and matching is one of the important tasks in computer vision, which aims to find matching point pairs in the illumination and perspective transformation of image sequences. As a kind of local feature, image feature points are widely used in structure from motion, SLAM and 3D reconstruction. Although feature point detection has been widely studied since 1980, various methods are still poorly robust to scale, illumination and affine transformation.

[0003] Early traditional image feature point detection methods were all based on manual design. The basic idea is to use the gradient, texture, color and other features of the local area of ​​the grayscale image to detect feature points. For example, the most widely used SIFT detects stable feature points in the image by constructing Gaussian pyramids and Gaussian difference pyramids (DoG) for local extreme value detection and scale space extreme point screening. With the development of deep learning, more and more learning-based feature point detection methods have emerged. Some methods convert feature point detection into a response map estimation problem. After an image passes through CNN, a response map is obtained. The score of each pixel in the response map after NMS represents the probability that the point is a feature point. These methods use the real response map or the similarity relationship between the corresponding positions of the two images to train the response map. Some methods also obtain the response map through the spatial or channel changes of multi-channel feature maps.

[0004] Data-driven methods require a large amount of training data to learn the patterns of feature points. The datasets used to train feature point detection models mostly come from motion structure recovery and depth estimation. The annotation workload of these datasets is large and difficult, so the training data is limited. In practical applications, the quality and scenes of images often change dramatically (such as day and night), which can cause the model to overfit the training dataset. The introduction of prior knowledge can use the knowledge of domain experts to provide additional supervision information for the model, which can guide the model to learn semantic feature expressions in images more accurately, avoid learning useless features, and reduce the model's dependence on a large amount of labeled data.

[0005] There are a few methods that have explored how to introduce prior knowledge into learning-based methods. Key.Net attempts to introduce manual features as prior knowledge into learning-based methods. The input image is filtered by the Sobel operator to obtain an edge image, and then the model performs feature point detection on the edge detection result. REKD, like Key.Net, also uses the result of manual feature filtering as the input of the model, but it differs from Key.Net only in the network structure and training process.

[0006] The above prior knowledge-assisted method only uses the result after Sobel filtering as the model input, and uses manual features to extract anchor points at the model input, but does not fully utilize the rich prior knowledge in the manual features. SIFT, as the most widely used manual feature, includes steps such as scale space construction, DoG space establishment, and DoG extreme value detection. Each step contains a lot of prior knowledge, which can assist the learning process of the deep learning model. Therefore, it is very important to design a model that incorporates SIFT prior knowledge. Summary of the invention

[0007] In order to overcome the shortcomings of the prior art, the present invention provides an image feature point detection method assisted by prior knowledge. First, in order to extract multi-scale features, the input image is downsampled twice to obtain three groups of images; the three groups of images are respectively passed through the feature extraction module and the multi-group feature fusion module to obtain the final response map; the feature extraction module is composed of a main branch and a sub-branch; in order to fully learn the prior knowledge generated by SIFT in the feature point detection process, the main branch uses the Sobel edge detection result as input, and some parameters learn the DoG extreme value result to achieve simultaneous learning of deep features and prior knowledge; the sub-branch uses Gaussian filtering to construct a scale pyramid and uses the DoG space as a label for learning; in order to achieve the fusion of deep features and prior knowledge, the feature representation of the main branch and the feature representation of the sub-branch are fused through the proposed knowledge fusion module. After introducing prior knowledge, the method of the present invention has great potential in improving the feature point detection performance and model robustness, and provides a new research direction for feature point detection and matching tasks in future computer vision.

[0008] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0009] Step 1: Multi-scale knowledge-assisted feature extraction;

[0010] Step 1-1: Construct three groups of pyramids by using downsampling, and the three groups of pyramids share weights;

[0011] Step 1-2: The feature extractor consists of a main part and an auxiliary part. The main part uses the first-order derivative and the second-order derivative of the image grayscale image to form a multi-channel as input; the gradient image is passed through a convolution block to extract the deep features, and then fused with the feature map of the auxiliary part through a knowledge fusion module; after two feature fusions, the feature map of H*W*C is obtained as the group feature;

[0012] Step 1-3: Let one channel of the group feature be learned with the extreme value detection result of the DoG space as the label; use binary cross entropy as the loss function L1, and the group features of the remaining channels use the window-based MIP Loss in Key.Net and REKD as the loss function L2;

[0013] Step 1-4: Use the Gaussian filter kernel to construct a 5-layer Gaussian scale space for the input grayscale image. After two convolution blocks, learn with the DoG space as the label, and use the mean square error function as the loss function L3 of this part; learn each channel in the DoG space;

[0014] Step 2: Knowledge fusion module;

[0015] Step 2-1: First, concatenate the prior knowledge and deep features, and further extract features through 2 convolution blocks and 1 convolution layer;

[0016] Step 2-2: Use channel attention and spatial attention to reconstruct the extracted features;

[0017] Step 2-3: Add residual connection to the knowledge fusion module;

[0018] Step 3: Multi-scale feature fusion;

[0019] Step 3-1: Fuse the feature maps of the three pyramids and enlarge the feature maps downsampled twice to the size of the input image;

[0020] Step 3-2: Use the bilinear interpolation algorithm to upsample the feature map. After cascading the feature maps of the same size, use 1×1 convolution to reduce the dimension to obtain the final response map.

[0021] Furthermore, the loss function used by the image feature point detection method is a balance of L1, L2 and L3. Therefore, the final loss function L is L=L1+L2+L3.

[0022] The beneficial effects of the present invention are as follows:

[0023] The method of the present invention has great potential in improving the feature point detection performance and model robustness after introducing prior knowledge, and provides a new research direction for feature point detection and matching tasks in future computer vision. In addition, the designed knowledge fusion module not only successfully realizes the organic combination of deep features and manual prior knowledge, but also provides a valuable reference for how to effectively fuse different types of features in other computer vision tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The convolution block and knowledge fusion module of the present invention; (a) convolution block, (b) knowledge fusion module. DETAILED DESCRIPTION

[0025] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0026] The present invention proposes a key point detection model based on the fusion of deep features and prior knowledge. First, in order to extract multi-scale features, the input image is downsampled twice to obtain three groups of images. The three groups of images are respectively passed through a feature extraction module and a multi-group feature fusion module to obtain the final response map. The feature extraction module consists of a main branch and a sub-branch. In order to fully learn the prior knowledge generated by SIFT in the feature point detection process, the main branch uses the Sobel edge detection result as input, and some parameters learn the DoG extreme value result to achieve simultaneous learning of deep features and prior knowledge. The sub-branch uses Gaussian filtering to construct a scale pyramid and uses the DoG space as a label for learning. In order to achieve the fusion of deep features and prior knowledge, the feature representation of the main branch and the feature representation of the sub-branch are fused through the proposed knowledge fusion module.

[0027] 1.1 Multi-scale knowledge-assisted feature extraction;

[0028] Multi-scale information can help the model detect feature points at different scales of the image and is robust to scale changes of the image. In order to enable the model to fully extract multi-scale information in the image and save multiple forward propagations, the network constructs three groups of pyramids by using downsampling. In order to save computing resources, the three groups of pyramids share weights.

[0029] In order to make full use of the prior knowledge in SIFT, the feature extractor consists of two parts: the backbone and the auxiliary. Inspired by the Harris and Hessian detectors, the backbone part uses the first-order derivative and the second-order derivative of the image grayscale image to form multiple channels as input. The gradient image allows the model to focus on learning areas with significant gradient changes. The gradient image is passed through a convolution block to extract deep features, and then fused with the feature map of the auxiliary part through a knowledge fusion module. After two feature fusions, the feature map of H*W*C is obtained as the group feature. The convolution block is as follows Figure 1 (a) shown.

[0030] As one of the core steps of the SIFT algorithm, constructing the DoG space can find scale-invariant feature points at different scales, and the response map after extreme value detection can more accurately locate the feature points. At the same time, since the group feature is directly related to the final predicted response map, one channel of the group feature is learned with the extreme value detection result of the DoG space as the label. The binary cross entropy is used as the loss function L1. The group features of the remaining channels use the window-based MIP Loss commonly used in Key.Net and REKD as the loss function L2. This loss function can efficiently find anchor-based feature points and can find affine-invariant feature points.

[0031] Similar to SIFT, the sub-network uses a Gaussian filter kernel to construct a 5-layer Gaussian scale space for the input grayscale image. After two convolution blocks, the DoG space is used as the label for learning, and the mean square error function is used as the loss function L3 of this part. Learn each channel in the DoG space. In addition, since the features learned by the sub-network will be integrated into the backbone network, and the final loss function is balanced by the backbone network and the sub-network, the backbone can not only learn the knowledge of the DoG space, but also adaptively discard unstable feature points.

[0032] The assistance of prior knowledge can reduce the amount of model training parameters, make model training more stable, and speed up the convergence of the model.

[0033] 1.2 Knowledge fusion module;

[0034] The features of the main part and the auxiliary part both carry a lot of prior knowledge and deep features. It is very important to fuse these two parts of features. Currently, there is no method to try to fuse manual features with deep features. Therefore, a module for fusing prior knowledge and deep features is designed. Figure 1 (b) as shown.

[0035] First, the two feature maps are concatenated, and features are further extracted through two convolution blocks and one convolution layer. Since the feature maps of different branches are directly concatenated in the channel dimension, how to extract the relationship between channels needs to be carefully designed. In addition, the positioning of feature points is also related to the different pixel information in the spatial neighborhood, so channel attention and spatial attention are used to reconstruct the extracted features. Channel attention can determine the different weight relationships between manual feature maps and deep feature maps, increase the weights of channels that are helpful for feature point detection, and suppress invalid channels. Spatial attention can increase the weights of pixels in significant areas, so that the model focuses on areas where both manual features and deep features are significant, and reduces attention to areas where neither feature is significant.

[0036] The input feature map contains a lot of prior knowledge. In order to reuse the prior knowledge, residual connections are added to the knowledge fusion module. Residual connections can enhance the model's feature extraction ability and improve the model's generalization ability, thereby making better use of prior knowledge.

[0037] 1.3 Multi-scale feature fusion;

[0038] In order to obtain the final response map containing multi-scale information, it is necessary to fuse the feature maps of the three pyramids and enlarge the feature maps that were downsampled twice to the size of the input image. Since the bilinear interpolation algorithm is fast and effective, the bilinear interpolation algorithm is used to upsample the feature map. After cascading the feature maps of the same size, 1×1 convolution is used to reduce the dimension to obtain the final response map.

[0039] 1.4 Loss function;

[0040] The loss function used in training is a balance of L1, L2, and L3. Therefore, the final loss function L is L = L1 + L2 + L3.

[0041] Example:

[0042] 2.1 Experimental setup;

[0043] 2.1.1 Training dataset;

[0044] Referring to Key.Net and REKD, a synthetic training dataset was generated from the validation set of the ImageNet ILSVRC 2012 dataset. A random homography transformation was applied to the image, and pairs of corresponding regions were extracted as the training set. The specific parameters of the homography transformation include: scaling range [0.5, 3.5], shearing range [-0.8, 0.8], and rotation angle range [-60, 60]. In order to increase the robustness of the network to changes in lighting and color, the contrast, brightness, and H value in the HSV space were randomly changed for one of the images in a pair. A total of 12,000 pairs of 192*192 image pairs were used as the dataset, of which 9,000 pairs were used as training sets and 3,000 pairs were used as validation sets.

[0045] 2.1.2 Implementation details;

[0046] All experiments were implemented using the Pytorch framework. Each convolutional layer uses a 5×5 convolution kernel with a padding size of 2 and no bias is added. All weights are randomly initialized. The batch size is set to 16. Adam is used as the optimizer, and the initial learning rate is set to 0.001. A total of 20 epochs are trained, and a cosine learning rate is used. For fair comparison, the MIP Loss related settings are exactly the same as [32-33]. The size of NMS is set to 5×5.

[0047] 2.1.3 Evaluation dataset;

[0048] Similar to REKD, two public datasets are used for comparison. The HPatches dataset is used to evaluate feature point detection and matching, and the IMC2021 dataset is used to evaluate the accuracy of 6DoF pose estimation.

[0049] The HPatches dataset consists of 116 scenes with 59 viewpoint variations and 57 illumination variations. Each scene consists of 5 image pairs with ground truth planar homographies, for a total of 696 image pairs. The model is compared with existing models using 1000 feature points for evaluation. The comparison is performed using repetition rate, number of predicted matches, and mean match accuracy (MMA). The repetition rate is the ratio between the number of repeatable feature points and the total number of feature points detected with a 3 pixel threshold. MMA is the average percentage of correct matches for each image pair. Correct matches are measured by 3 and 5 pixel thresholds of MMA.

[0050] IMC2021 is a large-scale wide baseline matching challenge dataset. IMC2021 consists of an unconstrained urban scene with large lighting and viewpoint variations. In this experiment, the proposed method is compared with existing keypoint detection methods. Experiments are performed on stereo track using the validation sets of Phototourism and PragueParks. The benchmark takes predicted matches as input and measures the accuracy of 6DoF pose estimation. The mean average accuracy (mAA) of 5° and 10° pose estimation as well as the number of inliers are measured.

[0051] 3.1HPatches;

[0052] Table 1 shows the results of feature point detection and matching in HPatches. As with REKD, 1000 feature points were counted. Hand-designed detectors and learning-based detectors were used as baselines for comparison with block-based descriptors. The joint detection and description method and the integrated method with rotation-invariant dense descriptors were also compared. In this experiment, the mutual nearest neighbor matching algorithm was used for all cases. Compared with existing key point detection methods, the model achieved the best feature point repetition rate, which means that the feature point detector of the present invention is robust to changes in viewing angle and lighting. Compared with the best performing key point detector REKD, the model of the present invention can achieve higher MMA scores when using the same descriptor.

[0053] Table 1 HPatches dataset

[0054]

[0055]

[0056] Note: The bold font is the optimal value for each row.

[0057] 3.2IMC2021;

[0058] Table 2 shows the results of 6DoF pose estimation in IMC2021 for evaluating complex tasks in general scenes. For this experiment, HardNet was used as a descriptor for image matching, and AdaLAM's DEGENSAC method was used for matching screening in all cases. For the AdaLAM stage, the direction values ​​estimated by the present invention and the scale values ​​inferred from the scale space were used. Four baselines, DoG+AN, Key.Net, and REKD, were compared. The results show that the model of the present invention consistently improves the camera pose estimation accuracy (mAA) and the number of internal points compared to REKD. The model of the present invention consistently outperforms the baseline REKD in all indicators.

[0059] Table 2 IMC2021 dataset

[0060]

[0061] 4. Ablation experiment;

[0062] 4.1 Knowledge fusion module;

[0063] In order to prove the effectiveness of the proposed knowledge fusion module, an ablation experiment was conducted on the two knowledge fusion modules in the model. The experiments were conducted by removing all knowledge fusion modules and retaining only one. The specific results are shown in Table 3. It can be clearly seen from the experiment that with the increase in the number of knowledge fusion modules, all evaluation indicators have been significantly improved, proving the effectiveness of the knowledge fusion module.

[0064] Table 3 Experimental results of different numbers of knowledge fusion modules

[0065]

[0066] 4.2 Prior knowledge labeling;

[0067] The framework of the model of the present invention is to learn the rich prior knowledge contained in the manual features as labels and finally fuse them with the deep features. Therefore, in order to prove that learning prior knowledge is necessary, an ablation experiment is conducted on whether to use prior knowledge as labels. The experiment shows that adding prior knowledge as labels for learning can improve various indicators, proving that learning with prior knowledge as labels is effective. The specific results are shown in Table 4.

[0068] Table 4 Experimental results of different prior knowledge

[0069]

[0070]

[0071] The present invention proposes an image feature point detection model based on the fusion of deep features and prior knowledge, aiming to solve the problem that the feature point detection method based on deep learning is highly dependent on a large amount of labeled data and is prone to overfitting.

[0072] By integrating the prior knowledge of manual features such as SIFT into the deep learning model, the model of the present invention can more accurately learn the semantic feature expression in the image, avoid learning useless features, and thus reduce the dependence on a large amount of annotated data. The experimental results achieved the most advanced performance on two classic datasets, HPatches and IMC2021, verifying the effectiveness and superiority of the method of the present invention.

Claims

1. A method for detecting image feature points assisted by prior knowledge, characterized in that: The steps include: Step 1: Multi-scale knowledge-assisted feature extraction; Step 1-1: Construct three groups of pyramids by using downsampling, and the three groups of pyramids share weights; Step 1-2: The feature extractor consists of a main part and an auxiliary part. The main part uses the first-order derivative and the second-order derivative of the image grayscale image to form a multi-channel as input; the gradient image is passed through a convolution block to extract the deep features, and then fused with the feature map of the auxiliary part through a knowledge fusion module; after two feature fusions, the feature map of H*W*C is obtained as the group feature; Step 1-3: Let one channel of the group feature be learned with the extreme value detection result of the DoG space as the label; use binary cross entropy as the loss function L1, and the group features of the remaining channels use the window-based MIP Loss in Key.Net and REKD as the loss function L2; Step 1-4: Use the Gaussian filter kernel to construct a 5-layer Gaussian scale space for the input grayscale image. After two convolution blocks, learn with the DoG space as the label, and use the mean square error function as the loss function L3 of this part; learn each channel in the DoG space; Step 2: Knowledge fusion module; Step 2-1: First, concatenate the prior knowledge and deep features, and further extract features through 2 convolution blocks and 1 convolution layer; Step 2-2: Use channel attention and spatial attention to reconstruct the extracted features; Step 2-3: Add residual connection to the knowledge fusion module; Step 3: Multi-scale feature fusion; Step 3-1: Fuse the feature maps of the three pyramids and enlarge the feature maps downsampled twice to the size of the input image; Step 3-2: Upsample the feature map using a bilinear interpolation algorithm; After cascading feature maps of the same size, 1×1 convolution is used to reduce the dimension to obtain the final response map.

2. The method for detecting image feature points assisted by prior knowledge according to claim 1, characterized in that: The loss function used by the image feature point detection method is a balance of L1, L2 and L3, so the final loss function L is L=L1+L2+L3.

Images