Method, device and computer readable medium for extracting features from an image

Abstract

The present invention provides a method, device and computer readable medium for extracting features from an image. Among them, a method for extracting features from an image includes: receiving an image including a plurality of pixels; generating a plurality of confidence values ​​corresponding to a plurality of positions of the plurality of pixels in the image by a feature extractor based on artificial intelligence (AI); selecting a first position from the plurality of positions of the plurality of pixels in the image, wherein a first confidence value at the first position among the generated plurality of confidence values ​​is higher than a first threshold; and generating a final key point-descriptor pair set based on the plurality of confidence values ​​corresponding to the plurality of positions of the plurality of pixels in the image, the final key point-descriptor pair set including at least two key point-descriptor pairs, the at least two key point-descriptor pairs corresponding to the first position. Implementation of the embodiments of the present invention can have a higher local key point density.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and more particularly to feature extraction in computer vision. Background Art

[0002] During the feature extraction process, interest points and corresponding local descriptors are generated from the input image. These point correspondences can be used to estimate the pose of a moving camera in various applications such as Visual Odometry (VO), Structure from Motion (SFM), and Visual-based Simultaneous Localization And Mapping (V-SLAM). Summary of the invention

[0003] The present invention provides a method, a device and a computer readable medium for extracting features from an image, which have a higher local key point density.

[0004] Specifically, a method for extracting features from an image provided by the present invention may include: receiving an image including multiple pixels; generating multiple confidence values ​​corresponding to multiple positions of the multiple pixels in the image through a feature extractor based on artificial intelligence (AI); selecting a first position from the multiple positions of the multiple pixels in the image, wherein a first confidence value at the first position among the generated multiple confidence values ​​is higher than a first threshold; and generating a final key point-descriptor pair set based on the multiple confidence values ​​corresponding to the multiple positions of the multiple pixels in the image, the final key point-descriptor pair set including at least two key point-descriptor pairs, and the at least two key point-descriptor pairs correspond to the first position.

[0005] The present invention provides a device for extracting features from an image, comprising: an AI-based feature extractor, configured to receive an image including multiple pixels, and generate multiple confidence values ​​corresponding to the positions of the multiple pixels in the image; an additional structure, configured to select a first position from the multiple positions of the multiple pixels in the image, wherein a first confidence value at the first position among the generated multiple confidence values ​​is higher than a first threshold; and generate a final key point-descriptor pair set based on the multiple confidence values ​​corresponding to the multiple positions of the multiple pixels in the image, the final key point-descriptor pair set comprising at least two key point-descriptor pairs, and the at least two key point-descriptor pairs correspond to the first position.

[0006] The present invention provides a computer-readable medium storing instructions, which, when executed by a processor, enables the processor to execute a method for extracting features from an image, the method comprising: receiving an image including a plurality of pixels; generating a plurality of confidence values ​​corresponding to the positions of the plurality of pixels in the image by a feature extractor based on artificial intelligence (AI); selecting a first position from a plurality of positions of the plurality of pixels in the image, wherein a first confidence value at the first position among the generated plurality of confidence values ​​is higher than a first threshold; and generating a final key point-descriptor pair set based on the plurality of confidence values ​​corresponding to the plurality of positions of the plurality of pixels in the image, the final key point-descriptor pair set comprising at least two key point-descriptor pairs, the at least two key point-descriptor pairs corresponding to the first position.

[0007] The present invention can have a higher local key point density by implementing the above method, device and computer readable medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 An example of an artificial intelligence (AI) based feature extraction process 100 is shown;

[0009] Figure 2 Two curves 210 and 220 showing homography matrix accuracy (viewpoints only) versus number of detections (detected interest points);

[0010] Figure 3 A feature extraction system 300 based on key point resampling and fusion (KRF) according to an embodiment of the present invention is shown;

[0011] Figure 4 Another KRF-based feature extraction system 400 according to an embodiment of the present invention is shown;

[0012] Figure 5 A KRF-based visual simultaneous localization and mapping (V-SLAM) system 500 according to an embodiment of the present invention is shown;

[0013] Figure 6 shows a KRF-based feature extraction process 600 according to an embodiment of the present invention; and

[0014] Figure 7 An exemplary apparatus 700 according to an embodiment of the invention is shown. DETAILED DESCRIPTION

[0015] Certain words are used in the specification and claims to refer to specific components. It should be understood by those skilled in the art that hardware manufacturers may use different terms to refer to the same component. This specification and claims do not use differences in names as a way to distinguish components, but use differences in the functions of components as the criteria for distinction. "Including" and "including" mentioned throughout the specification and claims are open-ended terms, so they should be interpreted as "including but not limited to". "Substantially" or "approximately" means that within an acceptable error range, those skilled in the art can solve the technical problem within a certain error range and basically achieve the technical effect. In addition, the term "coupled" or "coupled" herein includes any direct and indirect electrical connection means. Therefore, if the text describes a first device coupled to a second device, it means that the first device can be directly electrically connected to the second device, or indirectly electrically connected to the second device through other devices or connection means. The following is a preferred way to implement the present invention, the purpose of which is to illustrate the spirit of the present invention rather than to limit the scope of protection of the present invention. The scope of protection of the present invention shall be subject to the definition of the claims.

[0016] The following description is the best embodiment of the present invention. These descriptions are used to illustrate the general principles of the present invention and should not be used to limit the present invention. The scope of protection of the present invention should be determined based on the claims of the present invention.

[0017] I. Limitations of AI-based feature extraction

[0018] In various applications, feature extraction (FE) can be performed to generate interest points and corresponding local descriptors from the input image. The interest point correspondences between consecutive images can be used to estimate the pose of a moving camera to solve problems such as visual odometry (VO), structure from motion (SFM), vision-based simultaneous localization and mapping (V-SLAM), or similar applications.

[0019] For example, the robot relies on information from the camera to estimate the camera's pose and track its position accordingly. The detection engine (or feature detector) can detect points of interest based on feature distinctions (such as edges, spots, and corners in the input image). The description engine can describe each point of interest using a representation such as a 256-bit long binary vector. Detected features (keypoint-descriptor pairs) from consecutive images can be matched based on some similarity measure between their descriptors to establish point correspondences. Interest points, feature points, and key points are used interchangeably in the present invention.

[0020] In some examples, feature extractors that perform feature detection and feature description functions can be classified into traditional feature extractors and learned (Artificial Intelligence (AI) based) feature extractors.

[0021] Traditional feature extractors use handcrafted techniques to compute corner / edge detection. Detection is usually done by computing the contrast between a center pixel and its neighboring pixels, and the descriptor of the center pixel is usually generated by computing the gradient of multiple pixels within its local patch. Examples of hand-crafted techniques include ORB, rotated FAST, SIFT, rotated BRIEF, and ORB-SLAM2, which are described by the following references: (i) E. Rublee, V. Rabaud, K. Konolige and GRBradski, “ORB: an efficient alternative to SIFT or SURF”, ICCV, Spain, pp. 2564-2571, 2011; (ii) E. Rosten and T. Drummond, “Machine learning for high-speed corner detection,” ECCV, Austria, pp. 430–443, 2006; (iii) D. G. Love, “Distinctive image features from scale-invariant keypoints,” International Journal of Computer Vision, 60(2): 91–110, 2004; (iv) M. Calonder, V. Lepetit, C. Strecha, and P. Fua, “BRIEF: Binary Robust Independent Elementary Features,” ECCV Greece, pp. 778–792, 2010; and (v) R. Mur-Artal, JMM Montiel, and JD Tardos, “ORB-SLAM: a versatile and accurate monocular slam system,” IEEE Transactions on Robotics, 31(5): 1147–1163, 2015. These references are incorporated herein by reference in their entirety. In some examples, traditional feature extraction is calculated by a central processing unit (CPU).

[0022] The AI-based feature extractor can learn interest points and descriptors through an end-to-end training structure. For example, as an output of the AI-based feature extractor, a heatmap containing the extracted interest points can be generated. Each interest point in the heatmap can be associated with a confidence value or a confidence score (e.g., a number between 0 and 1, or a number in a range larger than 0-1). The interest points in the heatmap can be extracted by thresholding against a cutoff confidence threshold. However, this binary-style approach ignores the confidence information of each interest point and limits the accuracy of pose estimation.

[0023] Examples of AI-based feature extraction techniques include SuperPoint (2018), GCNv2 (2019), UnsuperPoint (2019), and ContextDesc (2019), which are described by the following references: (i) D. DeTone, T. Malisiewicz, A. Rabinovich, “SuperPoint: self-supervised interest point detection and description,” CVPRW, US, 2017; (ii) J. Tang, L. Ericson, J. Folkesson, and P. Jensfelt, “GCNv2: efficient correspondence prediction for real-time SLAM,” IEEE Robotics and Automation Letters, 2019; (iii) PH Christiansen, MF Kragh, Y. Brodskiy, and H. Karstoft, “UnsuperPoint: end-to-end unsupervised interest point detector and descriptor,” arXiv preprint arXiv:1907.04011, 2019; (iv) Zixin Luo, Tianwei Shen, Lei Zhou, Jiahui Zhang, Yao Yao, Shiwei Li, Tian Fang, and Long Quan, “Contextual Description: Local Descriptor Enhancement with Cross-Modal Context,” Computer Vision and Pattern Recognition (CVPR), 2019. These references are incorporated herein by reference in their entirety. In some examples, the AI-based feature extraction can be computed by a CPU, a Graphic Processing Unit (GPU), an Accelerated Processing Unit (APU), or a combination thereof.

[0024] Although more effective than traditional feature extraction, the state-of-the-art AI-based feature extraction suffers from the following limitations: multiple learned interest points with discriminative confidence values ​​are not distinguished due to binary thresholding, and are spatially agonistic due to nearby descriptors with high similarity scores.

[0025] Figure 1An example of an AI-based feature extraction process 100 is shown. In process 100, a binary threshold method is employed, and incorrect feature matching occurs due to the high similarity of descriptors between dense key points. In process 100, an image 110 including pixels 111 is input to an AI-based detector 121 and an AI-based descriptor 122.

[0026] The AI-based detector 121 performs key point detection to generate a detector heat map 131 (also referred to as a detection heat map or heat map). The detector heat map 131 includes a plurality of confidence values ​​(e.g., scores between 0-1) that correspond to individual pixels of the input image 110, respectively. By thresholding the confidence values ​​by a threshold value (e.g., 0.1), a set of key points 133 including key points AE can be extracted from the detector heat map 131. Each confidence value in the detector heat map 131 (corresponding to a pixel in the image 110) can indicate the eligibility of each position to become a point of interest. A higher confidence value may indicate that the corresponding position has a higher probability of becoming a high-quality point of interest.

[0027] Keypoints AC are close to each other in space and are therefore referred to as dense interest points, while keypoints DE are each far away from any other keypoint and are therefore each referred to as a sparse interest point.

[0028] The descriptor 122 performs feature description to generate a descriptor map 132. The descriptor map 132 may include a descriptor (e.g., a 256-bit feature vector) corresponding to each pixel 111 in the input image 110. Corresponding to the key points AE obtained from the detector heat map 131, by performing descriptor sampling, a set of descriptors 134 including descriptors AE can be sampled from the descriptor map 132. Therefore, a plurality of key point-descriptor pairs can be formed, each of which includes the coordinates of the key point and the corresponding descriptor. Due to the characteristics of the AI-based descriptor 122, the plurality of descriptors AC corresponding to the dense plurality of key points AC tend to be highly similar to each other, while each of the plurality of descriptors DE corresponding to the sparse plurality of key points DE tends to be different from other descriptors in the descriptor map 132.

[0029] As a result of the V-SLAM processing, for example, a point cloud 140 including map points 141 may be generated based on a series of images before the current image 110. In the map points 141, the map points AE correspond to pixels that are key points AE, respectively. A feature matching process may be performed based on, for example, a descriptor similarity measure between a key point in the current image 110 and a key point in a previous image to establish a corresponding key point correspondence. Figure 1 The correspondence between key points AE and image points AE is shown in FIG.

[0030] Since the descriptor scores of keypoints AC are highly similar, confusion occurs during the feature matching process, whereby keypoints A, B, and C are incorrectly matched with image points B, C, and A, respectively, resulting in incorrect correspondences. In contrast, since the descriptor scores of keypoint DE have low similarity, keypoints D and E are correctly matched with image points D and E, respectively, resulting in correct correspondences.

[0031] As can be seen from the above process 100, keypoint detection is non-discriminative. All threshold-qualified points are treated in the same manner and used as valid interest points with the same importance to sample the descriptor map 132, although the extracted features with lower confidence scores have a higher tendency to fail to match (false matches). Certain measures such as feature matching, prose optimization, or random sampling consensus (RANSAC) can be used to remove outliers (incorrect matches). However, the ratio of high-quality keypoints to low-quality keypoints determines how successfully or accurately these measures will work.

[0032] A large number of keypoints per image can facilitate bundle adjustment or pose graph optimization. However, using binary thresholding conflicts with this goal. Conditioning (using a larger detection threshold) can filter out reliable / repeated points, but reduces the number of points per image. Loose conditioning (low detection threshold) leads to a decrease in the ratio of good keypoint-descriptor pairs to total keypoint-descriptor pairs and generates too many noise points, resulting in matching failures.

[0033] In addition, in process 100, multiple descriptors 134 are spatially adversarial. Ideally, the extracted locally dense and globally sparse key points can optimize the performance of applications such as V-SLAM systems. However, some processing operations may not be able to handle dense points or concentrated points generated from AI-based feature extraction. For example, adjacent descriptors AE may conflict with each other during feature matching. In order to avoid conflicts, in some examples, before sampling the descriptor map 132, adjacent key points may have to be partially removed to convert dense key points into sparse key points. Therefore, due to the spatial adversarial characteristics, the dense feature points generated from the AI ​​extraction cannot be fully utilized. In addition, suppressing the extracted key points results in a reduction in the number of available key points, which reduces the accuracy of the feature matching operation.

[0034] It is expected that AI-based feature extraction methods can provide a large number of available keypoints for each image while maintaining a high percentage of good points and maintaining locally concentrated keypoints among the available keypoints.

[0035] Image Scaling Pyramid

[0036] In some examples, a scale pyramid (or image pyramid) of the input image is established to obtain dense key points. The scale pyramid may include multiple layers of images with different sizes. The upper layer image may be filtered out and subsampled from the adjacent lower layer image, so it has a smaller height and width. The same feature extraction scheme may be applied to each layer of the image to obtain key point-descriptor pairs. The key point positions on each scaled layer may be scaled back to the size of the original input image. As a result, multiple key points may be located at or near the same pixel of the input image. However, since each of those nearby key points at different layers is associated with a different receiving area due to scaling (e.g., a point at a higher layer has a larger receiving area than a point at a lower layer), the descriptors calculated from different receiving areas may be different. Therefore, dense key points with different descriptors may be obtained.

[0037] As an example, the ORB feature extraction method employs a feature scaling pyramid to improve the performance of real-time applications including object detection and patch tracking on smartphones.

[0038] II. AI-based Feature Extraction with Keypoint Resampling and Fusion (KRF)

[0039] Keypoint resampling and fusion (KRF) is an AI-based feature extraction scheme. In an embodiment using KRF, confidence information from the AI-based feature extraction process is used to improve the accuracy of pose optimization. For example, interest points can be resampled from an input image or a detection map based on different confidence thresholds to generate pixel-stacked high-confident interest points. Therefore, the ratio of high-quality interest points to low-quality interest points can be increased. In some cases, supplementary multiple descriptors are used to describe the interest points of the pixel stack. In some cases, KRF can be used in conjunction with a scaling pyramid scheme and applied to multiple layers of resized images to further improve tracking accuracy.

[0040] Compared with other AI-based or non-AI-based feature extraction schemes, when KRF is adopted, a higher ratio of correct correspondence to incorrect correspondence can be obtained due to the additional high-confidence key points. In addition, local dense key points can be maintained. These technical advantages can improve pose estimation / optimization performance.

[0041] Table 1 shows the comparison of sampling points for each pixel in different feature extraction schemes.

[0042] Table 1

[0043]

[0044] 1. Posture Optimization

[0045] In one example, for an input image of size H × W, the AI-based feature detector head predicts a heatmap X∈R H×W In addition, the descriptor head prediction vector map where d D YesR H×W The vector depth of each pixel in . Based on the confidence threshold t c , extracting interest points x from X using a binary thresholding approach while sampling descriptor vectors from D. Image-to-map registration is performed by matching the descriptors of 2D interest points x with three-dimensional (3D) object points using perspective-n-point (PnP) and / or bundle adjustment (BA). Therefore, pose optimization is a maximum likelihood problem:

[0046]

[0047] in Represents a 2D point x k ,R,t,is the rotation and translation in 3D, and π,is the 3D-2D reprojection function. Therefore, the pose optimization accuracy depends on the ratio of the exact correspondences to the total correspondences,χ,.

[0048] 2. Key point resampling

[0049] In one embodiment, a VGG-style encoder with two local heads (decoders) is used to generate the heatmap X and the descriptor map D. The two heads can be jointly trained using viewpoint-warped image pairs, where the detector head learns from pseudo-labels and the descriptor head learns to associate similar pixels between the warped images. Finally, for i,j∈R H×WFor each pixel of the heat map X, a confidence value, for example, between 0 and 1, is learned.

[0050] In one example, a binary thresholding method is used to sample keypoints from the heatmap. If a keypoint x ij The confidence is equal to or higher than the threshold t c , then the key point can be extracted:

[0051]

[0052] In one example, a resampling scheme (multiple rounds of thresholding) is used to generate the key points of the stack. A first thresholding method (or main thresholding method) is performed using a low confidence threshold t lc Sample keypoints x from the heatmap lc , while the second thresholding method (or auxiliary thresholding method) uses a high confidence threshold t hc Sample keypoints x from the heatmap hc :

[0053]

[0054] x hc ∈x lc ∈X (4)

[0055] Accordingly, due to being t lc and t hc All are considered qualified, and the high confidence key point at pixel (i, j) is extracted twice. Finally, the key point extracted at pixel (i, j) will present a stack because the point are located at the same pixel. Therefore, confidence information is encoded into the keypoints of the stack, resulting in a refined pose optimization with improved accuracy.

[0056] In one example, after extracting the keypoints of the stack, descriptors corresponding to different layers of the keypoints of the stack are not sampled from the same descriptor map D. During pose estimation, keypoints with similar or identical descriptors are likely to form incorrect matches and are therefore removed. In such a configuration, the descriptors for a pair of keypoints of the stack are different from each other.

[0057] Various schemes can be used to obtain descriptors that are different from the descriptors in the descriptor graph D for the keypoints of the stack. In one example, the descriptors are represented as binary vectors. Thus, for two keypoints stacked at the same pixel, the complementary descriptors corresponding to the original descriptors in the descriptor graph D can be determined by switching each bit in the binary vector from 0 to 1 or from 1 to 0.

[0058] 3. Impact of key points of the stack

[0059] Figure 2 Two plots 210 and 220 of homography accuracy (viewpoint-only) versus number of detections (detected points of interest) are shown. These plots are obtained from a homography dataset called HPatches. Figure 2 Also shown in FIG. 2 is a curve 230 representing a hypothetical trend.

[0060] Curve 210 (baseline) corresponds to the case where a binary threshold t c The first scenario. For the threshold t c Different values ​​of c When t = 0.05, the homography accuracy (HA) is the best, and when t c When t is too small or too large, the homography matrix accuracy deteriorates. You can tentatively select the threshold t c This allows the detection of a large number of interest points. If t c If t is too low, low-quality (e.g., noisy and low repeatability) interest points will be acquired. Low-quality interest points may be detrimental to pose estimation because non-repeatable points will generate incorrect correspondences. In PnP / BA, incorrect correspondences are less harmful when the geometric distance is large because they are easily identified as outliers. However, when the geometric distance is small, incorrect correspondences can be accepted as intrinsic values ​​and will lead to pose errors. On the other hand, if t c Too high and too many points will be blocked, causing the optimizer (performing PnP / BA) to become sensitive to wrong correspondences.

[0061] Curve 220 (keypoints of the stack) corresponds to the second scene with keypoint resampling. For each data point on curve 220, a pair of thresholds shown in parentheses are used for two rounds of thresholding operations to generate the keypoints of the stack. The average number of detections corresponding to each data point is the sum of the keypoints generated by the two rounds of sampling operations.

[0062] As shown, as the threshold value changes from a higher value to a lower value, more key points become available. Since there are additional high confidence key points in each round of key point resampling, the ratio of correct correspondences to incorrect correspondences can be maintained. Therefore, the HA corresponding to curve 220 can be maintained or increased. As shown in curve 230, as long as interest points are extracted without reducing the ratio of correct correspondences, the accuracy of pose estimation will not deteriorate.

[0063] In addition, due to the keypoint resampling scheme, more high-confidence keypoints are counted into the resulting keypoint set to increase the bias of high-quality keypoints. Therefore, for all data points on curve 220, the HA of curve 220 is higher than the HA of curve 230.

[0064] 4. Feature extraction and pose optimization based on KRF

[0065] Figure 3 According to an embodiment of the present invention, a KRF-based feature extraction system 300 is shown. The system 300 may include an AI-based feature extraction structure 301 and an additional structure 302. The AI-based feature extraction structure 301 may operate in a manner similar to an AI-based feature extraction scheme using a binary threshold method, while the additional structure 302 may operate based on resampled key points to generate additional key point-descriptor pairs associated with high confidence key points of the stack.

[0066] The AI-based feature extraction structure 301 may include an AI-based feature extractor 320. The AI-based feature extractor 320 may be configured to process the input image 310 to output a detector heat map 331 and a descriptor map 332. The detector heat map 331 may include a confidence value corresponding to each pixel of the input image 310. The detector heat map 331 may include extracted key points, each of which is associated with a numerical value (confidence value) that indicates the quality of each key point (e.g., how reliable and / or repeatable the key point is), or indicates the probability of the point being a good key point. For example, the extracted key point may be represented by a position coordinate (i, j) in the input image 310. The descriptor map 332 may include a descriptor corresponding to each pixel in the input image 310.

[0067] In various embodiments, the AI-based feature extractor 320 can use various neural network models and be trained in various ways. Typically, the AI-based feature extractor 320 can employ a convolutional neural network (CNN) trained in a supervised, unsupervised, or semi-supervised manner. For example, the AI-based feature extractor 320 can use the SuperPoint architecture, which is described by D. DeTone, T. Malisiewicz, A. Rabinovich in "self-supervised interest point detection and description," CVPRW, US, 2017.

[0068] In one embodiment, Figure 3As shown, the AI-based feature extractor 320 includes a shared backbone (shared encoder) 321, a detector head (detector decoder) 322, and a descriptor head (descriptor decoder) 323. The shared backbone 321 processes and reduces the dimensionality of the input image. For example, the shared backbone 321 can use a VGG-style encoder, and the shared backbone 321 includes multiple convolutional layers, spatial downsampling via pooling, and nonlinear activation functions. The detector head 322 and the descriptor head 323 learn task-specific weights.

[0069] In one embodiment, the AI-based feature extractor 320 includes independent pipelines for generating the detector heat map 331 and the descriptor map 332, respectively. For example, two separate neural networks corresponding to the detector heat map 331 and the descriptor map 332 may be used. Alternatively, an AI-based detector is used to generate the detector heat map 331, and a non-AI-based descriptor engine is used to generate the descriptor map 332.

[0070] The detector heat map 331 may include a confidence value for each pixel in the input image 310. For example, the confidence values ​​in the detector heat map may be output from the detector head 322. Then, a first thresholding method (relative to a second thresholding method performed subsequently) may be performed using a first threshold value to filter out key points extracted from the detector heat map 331.

[0071] In various embodiments, each of the confidence values ​​in the detector heat map 331 may be a value in the range of 0-1, which represents the probability of the point being a good keypoint. Alternatively, each confidence value may be in a wider range than 0-1. Corresponding to these two cases, a threshold may be appropriately defined so that keypoints with a quality above a certain level may be selected.

[0072] After the detector heat map 331 and the descriptor map 332 are available, a descriptor sampling operation can be performed to select descriptors corresponding to keypoints in the detector heat map 331 from the descriptor map 332. A first list of keypoint-descriptor pairs (abbreviated in the figure as: keypoint-descriptor pair list) 333 can thus be formed by combining keypoints with corresponding descriptors. For example, each keypoint-descriptor pair can include coordinates (i, j) and a descriptor vector.

[0073] In one embodiment, in the operation of the additional structure 302, another detector heat map (referred to as conditional detector heat map 341) may be generated by resampling (e.g., using a confidence filter) the detector heat map 331. For example, a second thresholding method may be performed with a second threshold value to filter out key points in the detector heat map 331 to be included in the conditional detector heat map 341. The second threshold value may correspond to a higher confidence value than the first threshold value used in the first thresholding method to generate key points in the detector heat map 331.

[0074] In one example, resampling of the detector heat map 331 is performed by comparing the confidence values ​​of the pixels in the detector heat map 331 with the second threshold. In another example, resampling of the detector heat map 331 is performed on key points extracted based on the first threshold. The confidence values ​​of the key points extracted based on the first threshold are compared with the second threshold.

[0075] In one embodiment, a complementary descriptor map 342 may be generated by the additional structure 302. For example, by switching each bit in the descriptor vector in the descriptor map 332 from 0 to 1 or from 1 to 0, a corresponding complementary vector in the complementary descriptor map 342 may be generated.

[0076] In other embodiments, a descriptor graph that replaces the supplemented descriptor graph 342 may be generated in various ways to obtain descriptors that are different from the descriptors in the descriptor graph 332. In one example, descriptors having different vector lengths than the descriptors in the descriptor graph 332 may be used. In one example, a descriptor may be generated using a descriptor generation scheme (e.g., non-AI based) that is different from the descriptor generation scheme (e.g., AI based) used to generate the descriptor graph 332. In further embodiments, the descriptor 332 may be used to replace the supplemented descriptor graph 342. Subsequent processing operations for posture estimation or posture optimization may be appropriately adjusted to accommodate such a configuration.

[0077] After the conditional detector heat map 341 (containing the keypoints resulting from the resampling) and the supplemented descriptor map 342 are available, another descriptor sampling process may be performed to select descriptors corresponding to the keypoints in the conditional detector heat map 341 from the supplemented descriptor map 342. A second list of keypoint-descriptor pairs (not shown) may accordingly be formed by combining the keypoints with the resulting descriptors.

[0078] Finally, the additional structure 302 can perform a fusion operation to combine the first list 333 and the second list of keypoint-descriptor pairs to generate a final list of keypoint-descriptor pairs (abbreviated in the figure as: keypoint-descriptor pair list) 343. The final list 343 of keypoint-descriptor pairs may include keypoint-descriptor pairs of pixel stack keypoints, which have a relatively high confidence value above the second threshold. Such a final list 343 of keypoint-descriptor pairs results in a larger number of total keypoints, a higher ratio of higher quality keypoints to lower quality keypoints (potentially resulting in a higher ratio of correct feature matches to incorrect feature matches), and a higher local keypoint density compared to the first list 333 of keypoint-descriptor pairs generated without the keypoint resampling scheme.

[0079] Although in Figure 3 In the example of , a resampling is performed once from the detector heat map 331 to generate a two-layer stack of key points, but in some embodiments, multiple resamplings can be performed to generate more than two layers of stacked key points. For example, 5 rounds of resampling can be performed on the detector heat map 331 using 5 different or the same confidence thresholds, resulting in 6 layers of stacked key points. Therefore, corresponding to each sampling, a unique or similar descriptor generation scheme can be used to generate a corresponding descriptor map. The final list of keypoint-descriptor pairs may include keypoint pairs generated based on each of the 5 layers of key points.

[0080] Figure 4 Another KRF-based feature extraction system 400 according to an embodiment of the present invention is shown. The system 400 may include a first AI-based feature extractor 420, a second AI-based feature extractor 430, and a KRF module 440.

[0081] In one embodiment, the first AI-based feature extractor 420 may include a backbone 421 (e.g., a VGG-style encoder) shared by a keypoint detector 422 and a keypoint descriptor 423. The backbone 421 processes the input image 410 to generate a spatially reduced representation of the input image 410. Based on the output of the backbone 421, the keypoint detector 422 generates a detection heat map 424 (also referred to as a detector heat map). The detection heat map 424 includes a plurality of detected keypoints 425 that are respectively associated with confidence values ​​(or confidence scores) above a first threshold. Based on the output of the backbone 421, the keypoint descriptor 423 may generate a descriptor map 426 including a plurality of descriptors corresponding to a plurality of pixels in the input image 410.

[0082] Based on the detected keypoints 425, corresponding descriptors 427 may be sampled from the descriptor map 426. The detected keypoints 425 and corresponding descriptors 427 may be combined to form a first keypoint-descriptor pair list 428.

[0083] In one embodiment, the KRF module 440 includes a conditioner 441. The conditioner 441 can perform a confidence threshold 442 on the detected key points 425 to determine the key points 445 that can be included in the conditional detection heat map 444. The confidence threshold 442 can be based on a second threshold value that is higher than the first threshold value. Therefore, the conditional detection heat map 444 includes key points 445 having a confidence value that is higher than the second threshold value. In addition, the conditioner 441 can perform a complementing operation 443 to generate a supplementary descriptor map 446 based on the descriptor map 426. The KRF module 440 can sample the supplementary descriptor map 446 based on the key points 445 in the conditional detection heat map 444 to select corresponding descriptors 447. By combining the key points 445 and the corresponding descriptors 447, a second key point-descriptor pair list 448 can be generated.

[0084] In one embodiment, the second AI-based feature extractor 430 may have a similar structure to the first AI-based feature extractor 420 and may perform similar functions to the first AI-based feature extractor 420. However, the input of the second AI-based feature extractor 430 may be an image 411 obtained by resizing (reducing) the input image 410. In some examples, the resizing may be based on various multi-scale signal representation image processing techniques, such as image pyramids, scale space representations, multi-resolution analysis, etc. For example, various different smoothing kernels may be used to generate pyramids, such as Gaussian pyramids, Laplacian pyramids, steerable pyramids, etc.

[0085] In one example, corresponding to the reduced size of the input image 411, the size of the neural network employed in the second AI-based feature extractor 430 may be reduced compared to the first AI-based feature extractor 420. By processing the input image 411, a third keypoint-descriptor pair list 431 may be generated.

[0086] Therefore, KRF 440 can perform a fusion operation 450 to combine the first keypoint-descriptor pair list 428, the second keypoint-descriptor pair list 448 and the third keypoint-descriptor pair list 431 to generate a final list of keypoint-descriptor pairs (abbreviated as: keypoint-descriptor pair list in the figure) 451.

[0087] In one embodiment, the KRF module 440 may include another conditioner (similar to the conditioner 441) to process the output from the second AI-based feature extractor 430 to generate a fourth list of keypoint-descriptor pairs. The fourth list may also be included in the final list 451 of keypoint-descriptor pairs to further increase the number of keypoint-descriptor pairs.

[0088] In other embodiments, one or more additional AI-based feature extractors similar to the first AI-based feature extractor 420 and the second AI-based feature extractor 430 can be added to the system 400 in parallel with the first AI-based feature extractor 420 and the second AI-based feature extractor 430. Additional resized images can be generated using multi-scale signal representation techniques and input to each added feature extractor. In this way, more lists of keypoint-descriptor pairs can be generated and included in the final list of keypoint-descriptor pairs 451.

[0089] Note that although the terms "thresholding", "threshold", "confidence threshold", "first threshold", "second threshold" are used to describe the operation of obtaining key points from the output of the AI-based extraction process, the present invention is not limited to operations related to these terms. Any operation other than the threshold operation described herein that selects key points based on confidence scores associated with locations corresponding to pixels of the input image and results in key points of a multi-level pixel stack falls within the spirit of the present invention. Any other method (not limited to thresholding), such as post-processing the input image with respect to its channel or spectrum, should fall within the scope of the present invention as long as the detection results are continuously extracted from the same pixel.

[0090] in addition, Figure 4 The confidence threshold 442 and the supplementary operation 443 in the module of the conditioner 441 in the example are simple choices for the implementation method. As long as the key points in the conditional detection heat map 444 can be effectively generated and the supplementary descriptor map 446 can be effectively generated, different methods can also be used. Figure 4 Other conditioners or methods described in the example.

[0091] Furthermore, in some embodiments, separate heatmaps (eg, heatmaps 331 , 341 , 424 , and 444 ) may not be used and resampling of a first heatmap to generate keypoints in a second heatmap may not be performed.

[0092] For example, after a set of confidence scores corresponding to pixels of an input image can be obtained from an AI-based feature extraction process (e.g., in the form of a heat map), these confidence scores can be checked one by one using a first threshold. When a confidence score higher than or equal to the first threshold is found, the position corresponding to the confidence score can be determined as the position of the key point of the pixel stack to be extracted. Therefore, two or more pairs of key point-descriptor pairs can be formed corresponding to the position (with coordinates of (i, j) in the input image). Each pair of key point-descriptor pairs can have different descriptors sampled from different descriptor maps (e.g., supplementary descriptor maps).

[0093] In a similar manner, multiple locations of keypoints with pixel stacks may be determined. A corresponding keypoint-descriptor pair may be formed for each of the multiple locations. These keypoint-descriptor pairs may be included in a final list of keypoint-descriptor pairs.

[0094] In addition, when checking the confidence score of a location in the heat map, if the confidence score is less than the first threshold, the confidence score can be further compared to a third threshold. The third threshold can be lower than the first threshold. If the confidence score is higher than or equal to the third threshold, a non-stacked keypoint can be extracted from the location. Subsequently, the keypoint-descriptor pair at this keypoint can be added to the final list of keypoint-descriptor pairs.

[0095] In some embodiments, the extraction of stack key points is performed by ranking the confidence values ​​in the heat map. For example, the confidence values ​​of different positions in the heat map can be first classified. Then, the first number of positions with the highest confidence values ​​can be selected as the key points of the pixel stack. The second number of positions with the highest confidence values ​​can be selected as the key points of the non-pixel stack. Using the ranking scheme, stack key points greater than two layers can also be determined.

[0096] The KRF scheme disclosed in this article can be combined with a wide range of computer vision related technologies or applications to improve performance. For example, the final list of keypoint-descriptor pairs generated using the KRF technique can be used as input to various algorithms and applications, such as Perspective-N-Point (PNP), feature / interest point matching, package adjustment, graph optimization, image retrieval, iterative closest point (ICP), SLAM, visual odometry, relocalization, loop closure, 3D reconstruction, robot / drone navigation, self-driving cars, augmented reality (AR), virtual reality (VR).

[0097] Figure 5A V-SLAM system 500 based on KRF according to an embodiment of the present invention is shown. System 500 can be configured to track a camera moving in a scene of an environment without a priori map while building a map of the environment based on images output from a camera. The camera can be fixed to a robot or vehicle moving in the environment. System 500 may include a per-frame domain for a tracking process and a key frame domain for a map construction process. In one example, the tracking process and the map construction process are separate and run in two parallel threads. System 500 may include a feature extraction module 510, a tracking module 520, a local optimization module 530, and a key frame determination module 540 in a per-frame domain, as well as a global optimization module 550 and a loop closure module 570 in a key frame domain.

[0098] The feature extraction module 510 may be configured to receive image frames output from a camera and perform AI-based feature extraction frame by frame based on a KRF scheme. The features extracted for each frame in the form of a keypoint-descriptor pair list may be input to the tracking module 520.

[0099] The tracking module 520 can be configured to perform a pose estimation process to determine the camera pose (e.g., including a translation portion and a rotation portion) relative to the 3D map for each frame based on the features extracted in the corresponding frame. In one example, a feature matching process can be performed to find the correspondence between the key points of the current frame and the previous frame. Based on the correspondence and a previously constructed 3D point map (e.g., using a triangulation based on the extracted key points), the transformation (pose update) of the current frame relative to the previous frame pose (the pose of the previous frame) can be determined (e.g., using a PnP technique). By concatenating the pose update with the previous frame pose, the current frame pose can be determined and output from the tracking module 520. In addition, a trajectory estimate of the camera can be obtained.

[0100] The local optimization module 530 can perform a local optimization process to optimize the frame poses corresponding to the frames within the sliding time window received from the tracking module 520. The local optimization process can reduce the pose estimation error accumulated during the pose concatenation process. Various local optimization techniques can be used. Generally, a local bundle adjustment (BA) optimization technique can be used based on the estimated poses and key points of the most recent frames (e.g., 5 frames). As a result, an optimized frame pose (based on the time order) can be output from the local optimization module 530. In addition, a more accurate estimate of the trajectory of the camera can be determined.

[0101] The key frame determination module 540 may select key frames based on some criteria, for example, the image quality should be good; the time since the last key frame was added should exceed a certain number of frames; and / or the difference between two adjacent key frames should be above a threshold.

[0102] The global optimization module 550 can be configured to perform global optimization based on a large number of key frames, for example. For example, the 3D point graph can be initialized and expanded when a new key frame becomes available. When a new key frame is available, 3D points of key points not included in the 3D point graph can be added (e.g., by triangulation). In addition, the 3D point graph can include data of all or part of the selected key frames (e.g., key points and camera poses of each key frame generated from modules 510 and 520). After selecting a new key frame and adding relevant information to the 3D point graph, global BA can be performed on the 3D point graph to output an optimized 3D point graph (e.g., optimized 3D point positions and camera poses of each frame). The 3D point graph can be stored in the database 560. In some examples, the local optimization module 530 can use the 3D point graph to perform a local optimization process.

[0103] The loop closure module 570 can be configured to detect loops in the path traveled by the camera and close the loop. In one example, by measuring the similarity between a new keyframe and a previously available keyframe, and if two similar keyframes are found, a loop can be detected. To close the loop, a pose graph optimization with loop constraints can be performed to distribute loop closure errors along various graphs.

[0104] The local or global optimization techniques (e.g., local or global BA) used at module 530 or 550 in the V-SLAM system typically rely on a large number of key points (or 3D points) in the scene to achieve the desired optimization performance. More accurate key points in the scene can guide the optimizer to make predictions more accurately. Therefore, since the KRF-based feature extractor can add high-confidence key points of the stack to increase the number of available key points and increase the deviation of high-confidence points in the scene, the performance of the KRF-based V-SLAM can be effectively improved.

[0105] III. Feature extraction process based on KRF

[0106] Figure 6 FIG. 6 is a diagram showing a KRF-based feature extraction process 600 according to an embodiment of the present invention. Note that not all steps in process 600 are performed in some cases, and may be performed in parallel or in parallel. Figure 6 The steps in process 600 may be performed in a different order than the order shown. Process 600 may start from S601 and then proceed to S610.

[0107] At S610, an input image including a plurality of pixels may be received at an AI-based feature extractor. For example, the input image may be output from a camera moving in an environment.

[0108] At S620, a confidence value corresponding to the position of the pixel in the image may be generated by an AI-based feature detector in an AI-based feature extractor. In one example, the confidence value is in the form of a detection heat map.

[0109] In S630, a first set of positions may be selected from the positions of pixels in the image as a first set of key points based on the confidence values ​​generated in the detection heat map. For example, a first thresholding method may be performed on the confidence values ​​generated at S620 with a first threshold to extract the first set of key points. Each of the first set of positions may correspond to a confidence value in the generated confidence values ​​that is higher than the first threshold.

[0110] At S640 , a first keypoint descriptor map may be generated by a feature descriptor engine in the AI-based feature extractor.

[0111] At S650, a first list of keypoint-descriptor pairs may be generated. For example, based on the positions of the first set of keypoints, a first keypoint descriptor may be selected from the first keypoint descriptor graph. The first set of keypoints may be combined with the first keypoint descriptor selected from the first keypoint descriptor graph to form a first list of keypoint-descriptor pairs.

[0112] In S660, a second group of positions can be selected from the positions of pixels in the image as a second group of key points based on the generated confidence values. The second group of positions can each correspond to a confidence value in the generated confidence values ​​that is higher than a second threshold. In one example, the second threshold is higher than the first threshold. Therefore, the first group of positions includes the second group of positions. In one example, the second group of key points is included in the conditional heat map.

[0113] In one example, a second set of key points is selected from the first set of key points. For example, a second thresholding may be performed on the confidence values ​​of the first set of key points relative to a second threshold to extract the second set of key points. In one example, the second set of key points are selected from the locations of pixels in the input image or the locations in the detection map. For example, the confidence values ​​at the pixel locations in the input image may be thresholded relative to a second threshold to extract the second set of key points.

[0114] At S670, a second key point descriptor map may be generated. In one example, the second key point descriptor map is generated based on the first key point descriptor map. For example, each key point descriptor in the second key point descriptor map is a complementary binary vector to a corresponding binary vector in the first key point descriptor map.

[0115] At S680, a second list of keypoint-descriptor pairs may be generated. For example, based on the positions of the second set of keypoints, a second keypoint descriptor may be selected from the second keypoint descriptor graph. The second set of keypoints may be combined with the second keypoint descriptor selected from the second keypoint descriptor graph to form a second list of keypoint-descriptor pairs.

[0116] At S690, a final list of keypoint-descriptor pairs can be generated, which can include the first and second lists of keypoint-descriptor pairs. Thereafter, the final list of keypoint descriptors can be input to, for example, a pose estimation process, a pose optimization process, or a pose graph optimization process. Process 600 can proceed to S699 and terminate at S699.

[0117] In an alternative KRF-based feature extraction process, instead of determining a first set of key points and a second set of key points separately, the key points of a pixel stack may be determined as follows. For example, the position of a pixel in an input image may be checked sequentially or in parallel to see if the confidence value at the position is above a second threshold used, for example, in process 600. If a value above the second threshold is detected, it may be determined that a key point of the pixel stack may be employed at that position. Subsequently, two key point descriptors may be selected from the first key point descriptor map and the second key point descriptor map used in process 600 based on the detected position. Thus, two key point-descriptor pairs corresponding to the key points of the two pixel stacks at the detected positions may be formed and included in a final list of key point-descriptor pairs.

[0118] In addition, for positions where the confidence value is not higher than the second threshold, each confidence value can be compared to the first threshold. If the corresponding confidence value is higher than the first threshold, it can be determined that a single keypoint can be used for the position. Therefore, a keypoint-descriptor pair corresponding to the position can be generated and included in the final list of keypoint-descriptor pairs.

[0119] IV. Instrument and Computer Readable Media

[0120] Figure 7An exemplary device 700 according to an embodiment of the present invention is shown. The device 700 may be configured to perform various functions according to one or more embodiments or examples described herein. Therefore, the device 700 may provide a device for implementing the mechanisms, techniques, processes, functions, components, and systems described herein. For example, the device 700 may be used to implement an AI-based feature extractor, a non-AI-based feature extractor, a key point detector, a key point descriptor, a KRF module, an AI-based feature extraction module, additional structures, and functions of modules in the V-SLAM system in the various embodiments and examples described herein. The device 700 may include a general-purpose processor or a specially designed circuit to implement the various functions, components, or processes described herein in various embodiments. The device 700 may include a processing circuit 710 and a memory 720.

[0121] In various examples, the processing circuit 710 may include circuits configured to perform the functions and processes described herein in conjunction with or without software. In various examples, the processing circuit 710 may be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), a digital enhancement circuit or the like, or a combination thereof.

[0122] In some other examples, the processing circuit 710 may be a central processing unit (CPU) or an accelerated processing unit (APU) configured to execute program instructions to perform the various functions and processes described herein. Therefore, the memory 720 may be configured to store program instructions. When executing the program instructions, the processing circuit 710 may perform functions and processes. The memory 720 may further store other programs or data, such as an operating system, an application program, etc. The memory 720 may include non-temporary storage media, such as read-only memory (ROM), random access memory (RAM), flash memory, solid-state memory, hard disk drive, optical disk drive, and similar products.

[0123] The apparatus 700 may optionally include other components, such as input and output devices, additional or signal processing circuits, etc. Therefore, the apparatus 700 may be able to perform other additional functions, such as executing application programs, image processing algorithms, inputting or outputting data, etc.

[0124] The processes and functions described herein can be implemented as a kind of computer program, which, when executed by one or more processors, can cause one or more processors to perform respective processes and functions. The computer program can be stored or distributed on a suitable medium, such as an optical storage medium or solid-state medium provided together with other hardware or provided as a part of other hardware. The computer program can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. For example, a computer program can be obtained and loaded into a device, such as by obtaining a computer program through a physical medium or a distributed system (including, for example, from a server connected to the Internet).

[0125] A computer program may be accessed from a computer readable medium that provides program instructions for use by or in conjunction with a computer or any instruction execution system. A computer readable medium may include any device that stores, communicates, propagates, or transmits a computer program for use by or in conjunction with an instruction execution system, device, or apparatus. The computer readable medium may be a magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or device or apparatus) or propagation medium. The computer readable medium may include computer readable non-transitory storage media, such as semiconductor or solid-state memory, magnetic tape, removable computer disk, random access memory (RAM), read-only memory (ROM), magnetic disks and optical disks, etc. Computer readable non-transitory storage media may include all types of computer readable media, such as magnetic storage media, optical storage media, flash memory media, and solid-state storage media.

[0126] V. Experiment

[0127] 1. Dataset

[0128] The experiments of the KRF scheme described in this paper are conducted using 2 datasets, HPatches and TUM, which are benchmarks for homography matrix estimation and SLAM indoor tracking accuracy. The HPatches sequence dataset contains 580 image pairs divided into 59 viewpoints and 57 lighting scenarios. The KRF-based model is evaluated on HPatches HA, which transforms the ground truth warped image C j The average distance error of the four image corners and the distorted image estimated by the network The average distance error of the four image corners is compared. This distance error is then binary thresholded with a tolerance threshold and averaged across all N image pairs.

[0129]

[0130] In addition, the KRF-based model is integrated into the ORB-SLAM2 structure in order to evaluate the Absolute Trajectory Error (ATE) on the TUM RGB-D SLAM dataset. This dataset contains RGB image clips captured from 23 to 173 seconds using Microsoft Kinect at a resolution of 480×640 and 30FPS, with corresponding depth maps and ground truth trajectories.

[0131] 2. HPatches results

[0132] In Table 2, the HA scores for tolerance thresholds ε of 1, 3, and 5 are shown. Reference: PH Christiansen, MF Kragh, Y. Brodskiy, and H. Karstoft, “UnsuperPoint: End-to-end unsupervised interest point detector and descriptor,” arXiv preprint arXiv:1907.04011, 2019 reports the results obtained from classic FE, LFnet, and UnsuperPoint. For AI-FE, non-maximum suppression (NMS) is used because they are trained with spatial suppression to maintain the saliency of key points. In addition, NMS also improves the HA score of AI-FE.

[0133] Table 2

[0134]

[0135] The SuperPoint released by the authors is used as the baseline for the KRF model. In the implementation of the KRF model, the confidence threshold t c = 0.015 for the vanilla SuperPoint, and t hc = 0.05, and t hc = 0.1 for KRF resampling without using the scaling pipeline (e.g. Figure 4 KRF performs best at larger tolerance ε of 3 and 5, improving SuperPoint by 12-17%. However, since only high-quality images are used, this benchmark cannot fully demonstrate the full potential of KRF.

[0136] 3. SLAM Results

[0137] In Table 3, a comparison of the absolute trajectory error (ATE) between ORB-SLAM2, SuperPoint-SLAM, and SuperPoint-KRF-SLAM is given. To evaluate the tracking accuracy, all experiments were run without loop closure and tracked using the same method as in ORB-SLAM2. Due to motion blur, SuperPoint-SLAM uses a low threshold t c =0.0005, while SuperPoint-KRF-SLAM uses t c =[0.0005,0.1005]. In this way, at least 200 keypoints can be extracted per frame. In Table 3, the measurement results are the median / average of 10 runs. Experiments with more than 3 tracking losses are called "tracking loss".

[0138] Table 3

[0139]

[0140] When the tracked features are robust, SuperPoint outperforms ORB-SLAM2: when fr1_room_short contains the first 800 frames of fr1_room with robust features, SuperPoint-SLAM improves ATE by 0.010 over ORB-SLAM2. However, SuperPoint-SLAM performs poorly in the second half of fr1_room. For all other scenes except fr1_floor, SuperPoint-SLAM suffers from tracking loss.

[0141] SuperPoint-KRF-SLAM outperforms ORB-SLAM2 and SuperPoint-SLAM, but its room_short is slightly worse than SuperPoint-SLAM, and it also suffers from tracking loss in fr1_360.

[0142] SuperPoint-KRF-SLAM is the only model that can complete room without trajectory degradation in the second half, achieving a 40% improvement over SuperPoint-SLAM at a median ATE of 0.072. It can also accurately complete fr1_desk and fr3_large_cabinet, and our model outperforms ORB-SLAM2 by a wide margin on fr3_long_office and fr3_nstf, where SuperPoint-SLAM also suffers from tracking loss.

[0143] Although the present invention is disclosed as above with preferred embodiments, it is not intended to limit the scope of the present invention. Any person skilled in the art may make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the definition of the claims.

Claims

1. A method for extracting features from an image, characterized in that: include: receiving an image comprising a plurality of pixels; generating, by an artificial intelligence-based feature extractor, a plurality of confidence values ​​corresponding to a plurality of locations of the plurality of pixels in the image; selecting a first position among the plurality of positions of the plurality of pixels in the image, wherein a first confidence value at the first position among the plurality of generated confidence values ​​is above a threshold; and generating a final set of keypoint-descriptor pairs based on the plurality of confidence values ​​corresponding to the plurality of positions of the plurality of pixels in the image, the final set of keypoint-descriptor pairs comprising at least two keypoint-descriptor pairs, the at least two keypoint-descriptor pairs corresponding to the first position; The method further comprises: generating a first descriptor graph by the artificial intelligence based feature extractor; and Generate a second descriptor graph, The at least two key point-descriptor pairs include two key point-descriptor pairs corresponding to the first key point descriptor in the first descriptor graph and the second key point descriptor in the second descriptor graph, respectively.

2. The method for extracting features from an image according to claim 1, characterized in that: The threshold is a second threshold, and the method further includes: Selecting a first group of positions from the plurality of positions of the plurality of pixels in the image as a first group of key points based on the plurality of confidence values ​​generated, each of the first group of positions corresponding to a confidence value in the plurality of confidence values ​​generated that is higher than a first threshold; and selecting a second group of positions from the multiple positions of the multiple pixels in the image as a second group of key points based on the multiple confidence values ​​generated, each of the second group of positions corresponding to a confidence value higher than the second threshold value in the multiple confidence values ​​generated, and the first group of positions including the second group of positions, wherein selecting the first set of positions and / or the second set of positions from a plurality of positions of a plurality of pixels in the image comprises selecting the first position from the plurality of positions of the plurality of pixels in the image, and The final set of keypoint-descriptor pairs includes a first set of keypoint-descriptor pairs corresponding to members of the first set of keypoints and a second set of keypoint-descriptor pairs corresponding to members of the second set of keypoints.

3. The method for extracting features from an image according to claim 1, characterized in that: The threshold is a second threshold, and the method further includes: Selecting a first group of positions from the plurality of positions of the plurality of pixels in the image as a first group of key points based on the plurality of confidence values ​​generated, each of the first group of positions corresponding to a confidence value in the plurality of confidence values ​​generated that is higher than a first threshold; and selecting a second group of positions from the plurality of positions of the plurality of pixels in the image as a second group of key points based on the plurality of confidence values ​​generated, each of the second group of positions corresponding to a confidence value in the plurality of confidence values ​​generated that is higher than the second threshold, wherein selecting the first set of positions and / or the second set of positions from a plurality of positions of a plurality of pixels in the image comprises selecting the first position from the plurality of positions of the plurality of pixels in the image, and The final set of keypoint-descriptor pairs includes a first set of keypoint-descriptor pairs corresponding to members of the first set of keypoints and a second set of keypoint-descriptor pairs corresponding to members of the second set of keypoints.

4. The method for extracting features from an image according to claim 1, wherein: The threshold is a second threshold, and the method further includes: determining whether the first confidence value at the first position among the generated plurality of confidence values ​​is above a first threshold; and In response to the first confidence value at the first position in the generated plurality of confidence values ​​being above the first threshold, selecting a first keypoint descriptor from the first position of a first descriptor map, the first descriptor map comprising a plurality of keypoint descriptors corresponding to the plurality of positions of the plurality of pixels in the image, and determining whether the first confidence value at the first position among the generated plurality of confidence values ​​is higher than the second threshold; In response to the first confidence value at the first position in the generated plurality of confidence values ​​being above the second threshold, selecting a second keypoint descriptor from the first position of a second descriptor map, the second descriptor map comprising a plurality of keypoint descriptors corresponding to the plurality of positions of the plurality of pixels in the image, The at least two key point-descriptor pairs include two key point-descriptor pairs corresponding to the first key point descriptor and the second key point descriptor.

5. The method for extracting features from an image according to claim 1, characterized in that: The first key point descriptor in the first descriptor graph is a first binary vector, and the second key point descriptor in the second descriptor graph is a second binary vector complementary to the first binary vector, wherein 1 or 0 in the first binary vector is switched to 0 and 1 in the second binary vector, respectively.

6. The method for extracting features from an image according to claim 1, characterized in that: Also includes: resizing an image to generate a resized image; determining keypoint-descriptor pairs corresponding to the resized image; and The keypoint-descriptor pair corresponding to the resized image is included in the final keypoint-descriptor pair set.

7. A device for extracting features from an image, characterized in that: include: an artificial intelligence-based feature extractor configured to receive an image comprising a plurality of pixels and generate a plurality of confidence values ​​corresponding to locations of the plurality of pixels in the image; additional structure configured to select a first position among a plurality of positions of the plurality of pixels in the image, wherein a first confidence value at the first position among the plurality of confidence values ​​generated is above a threshold; and generating a final set of keypoint-descriptor pairs based on the plurality of confidence values ​​corresponding to the plurality of positions of the plurality of pixels in the image, the final set of keypoint-descriptor pairs comprising at least two keypoint-descriptor pairs, the at least two keypoint-descriptor pairs corresponding to the first position; wherein the artificial intelligence based feature extractor is further configured to generate a first descriptor graph; and The additional structure is further configured to generate a second descriptor graph, The at least two key point-descriptor pairs include two key point-descriptor pairs corresponding to the first key point descriptor in the first descriptor graph and the second key point descriptor in the second descriptor graph, respectively.

8. The device for extracting features from an image according to claim 7, characterized in that: The threshold is a second threshold, and the artificial intelligence-based feature extractor is further configured as follows: Selecting a first group of positions from the plurality of positions of the plurality of pixels in the image as a first group of key points based on the plurality of confidence values ​​generated, each of the first group of positions corresponding to a confidence value in the plurality of confidence values ​​generated that is higher than a first threshold; and The additional structure is further configured to select a second group of positions from the plurality of positions of the plurality of pixels in the image as a second group of key points based on the plurality of confidence values ​​generated, each of the second group of positions corresponding to a confidence value in the plurality of confidence values ​​generated that is higher than the second threshold, the first group of positions including the second group of positions, wherein the artificial intelligence-based feature extractor selects a first set of positions from a plurality of positions of a plurality of pixels in the image and / or the additional structure selects a second set of positions from a plurality of positions of a plurality of pixels in the image comprises selecting the first position from the plurality of positions of the plurality of pixels in the image, and The final set of keypoint-descriptor pairs includes a first set of keypoint-descriptor pairs corresponding to members of the first set of keypoints and a second set of keypoint-descriptor pairs corresponding to members of the second set of keypoints.

9. The device for extracting features from an image according to claim 7, characterized in that: The threshold is a second threshold, and the artificial intelligence-based feature extractor is further configured as follows: Selecting a first group of positions from the plurality of positions of the plurality of pixels in the image as a first group of key points based on the plurality of confidence values ​​generated, each of the first group of positions corresponding to a confidence value in the plurality of confidence values ​​generated that is higher than a first threshold; and The additional structure is further configured to select a second group of positions from the plurality of positions of the plurality of pixels in the image as a second group of key points based on the plurality of confidence values ​​generated, each of the second group of positions corresponding to a confidence value in the plurality of confidence values ​​generated that is higher than the second threshold, wherein the artificial intelligence-based feature extractor selects a first set of positions from a plurality of positions of a plurality of pixels in the image and / or the additional structure selects a second set of positions from a plurality of positions of a plurality of pixels in the image comprises selecting the first position from the plurality of positions of the plurality of pixels in the image, and The final set of keypoint-descriptor pairs includes a first set of keypoint-descriptor pairs corresponding to members of the first set of keypoints and a second set of keypoint-descriptor pairs corresponding to members of the second set of keypoints.

10. The device for extracting features from an image according to claim 7, characterized in that: The threshold is a second threshold, and the additional structure is further configured as: determining whether the first confidence value at the first position among the generated plurality of confidence values ​​is higher than a first threshold; and In response to the first confidence value at the first position in the generated plurality of confidence values ​​being above the first threshold, selecting a first keypoint descriptor from the first position of a first descriptor map, the first descriptor map comprising a plurality of keypoint descriptors corresponding to the plurality of positions of the plurality of pixels in the image, and determining whether the first confidence value at the first position among the generated plurality of confidence values ​​is higher than the second threshold; In response to the first confidence value at the first position in the generated plurality of confidence values ​​being above the second threshold, selecting a second keypoint descriptor from the first position of a second descriptor map, the second descriptor map comprising a plurality of keypoint descriptors corresponding to the plurality of positions of the plurality of pixels in the image, The at least two key point-descriptor pairs include two key point-descriptor pairs corresponding to the first key point descriptor and the second key point descriptor.

11. The device for extracting features from an image according to claim 7, characterized in that: The first key point descriptor in the first descriptor graph is a first binary vector, and the second key point descriptor in the second descriptor graph is a second binary vector complementary to the first binary vector, wherein 1 or 0 in the first binary vector is switched to 0 and 1 in the second binary vector, respectively.

12. The device for extracting features from an image according to claim 7, characterized in that: The artificial intelligence-based feature extractor is a first artificial intelligence-based feature extractor, and the device further comprises: A second artificial intelligence-based feature extractor configured to: receive a resized image generated after resizing the image, and determine keypoint-descriptor pairs corresponding to the resized image; and The final key point-descriptor pair set includes the key point-descriptor pair corresponding to the resized image.

13. A computer readable medium storing instructions, characterized in that: When the instruction is executed by a processor, the processor executes the method for extracting features from an image as claimed in any one of claims 1 to 6.

Images