Image feature extraction method

Abstract

The invention relates to the technical field of image processing, in particular to an image feature extraction method, device and equipment and a computer program product. The image feature extraction method comprises the following steps: determining a score chart of confidence of each pixel point of an input image as a key point; determining key points of the input image based on the score graph; the input image is partitioned according to the key points, each input image comprises one key point, and each image comprises feature information; and generating feature descriptors corresponding to the key points according to the image blocks where the key points are located.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and in particular, to a feature extraction method and device for an image, an image processing apparatus, and a computer program product. BACKGROUND

[0002] Structure from Motion (sfm) is a technology in the field of computer vision and three-dimensional reconstruction, which can restore the three-dimensional structure of a scene (such as camera pose and three-dimensional point cloud) by inputting two-dimensional feature points of an image through processes such as feature extraction, feature matching, geometric verification, camera pose estimation, triangulation, and bundle adjustment. The technology can be used for reconstruction of an autonomous driving scene and surveying and mapping by a drone.

[0003] In related technologies, an sfm method relies on a manually designed feature extraction algorithm (such as Scale-Invariant Feature Transform (SIFT)). Since the feature extraction algorithm in related technologies is usually used to extract features for all pixel points in an input image, the efficiency is low when processing large-scale image data, and it takes more time, and the stability of the extracted features is insufficient in a complex scene (such as light changes and occlusions). SUMMARY

[0004] To overcome the problems in related technologies, the present disclosure provides a feature extraction method and device for an image, an image processing apparatus, and a computer readable storage medium, which can solve the above problems.

[0005] According to a first aspect of an embodiment of the present disclosure, a feature extraction method for an image is provided, the method comprising: determining a score map of confidence of each pixel point of an input image as a key point; determining a key point of the input image based on the score map; dividing the input image according to the key point, wherein each block of the input image contains one key point, and each block of the image contains feature information; and generating a feature descriptor of the corresponding key point according to the image block where the key point is located.

[0006] According to a second aspect of an embodiment of the present disclosure, a feature extraction device for an image is provided, the device comprising: a score map determination unit configured to determine a score map of confidence of each pixel point of an input image as a key point; a key point determination unit configured to determine a key point of the input image based on the score map; an image division unit configured to divide the input image according to the key point, wherein each block of the input image contains one key point, and each block of the image contains feature information; and a descriptor generation unit configured to generate a feature descriptor of the corresponding key point according to the image block where the key point is located.

[0007] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, comprising a processor, a memory, the memory is configured to store a computer program, and the processor is configured to execute the feature extraction method of the image according to the first aspect by invoking the computer program.

[0008] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which stores a computer program, and the program is executed by a processor to implement the feature extraction method of the image according to the first aspect.

[0009] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, comprising a computer program, and the computer program is executed by a processor to implement the method according to the first aspect.

[0010] The technical solutions provided by the embodiments of the present disclosure can include the following beneficial effects:

[0011] The present disclosure can determine a score map corresponding to an input image, determine a plurality of key points from the input image based on the score map, and divide the image into blocks according to the key points, so that each block of the input image contains a key point, and each block of the image contains complete feature information. Then, the feature descriptor of the corresponding key point is generated according to the image block, so as to realize feature extraction.

[0012] Compared with the related art, the technical solutions of the present disclosure determine the key points based on the score map, and do not need to detect and compare all pixel points in the image one by one, so that the data processing amount is relatively less, and the processing efficiency is higher. Secondly, after determining the key points, the present disclosure does not extract features based on the key points, but divides the image into blocks so that the key points are contained in the image blocks with complete feature information, and then extracts features from the image blocks. In this case, the feature extraction is not based on the pixel points where the key points are located, but based on the image blocks where the key points are located. This way makes the features extracted by the present disclosure better reflect the semantic information in the image. In the face of complex scenes such as changes in light, scene occlusion, and repeated textures, the extracted features are more accurate and have higher precision.

[0013] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0014] The accompanying drawings, which are incorporated into the specification and constitute a part of the present disclosure, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure.

[0015] Figure 1 is a schematic flow chart of a feature extraction method of an image according to an exemplary embodiment of the present disclosure.

[0016] Figure 2 This is a schematic flowchart illustrating an image feature extraction method according to an exemplary embodiment of the present disclosure.

[0017] Figure 3 This is a block diagram of an image feature extraction apparatus according to an exemplary embodiment of the present disclosure.

[0018] Figure 4 This is a schematic block diagram illustrating a feature extraction apparatus for an image according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0019] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0020] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0021] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0022] In related technologies, SFM feature extraction relies on algorithms such as SIFT, which has low computational efficiency when processing large-scale image data and insufficient stability when dealing with complex scenes.

[0023] Sfm is a technology in the field of computer vision and three-dimensional reconstruction. Through the two-dimensional feature points of multiple images, the three-dimensional structure of the scene is recovered through feature extraction, feature matching, geometric verification, camera pose estimation, triangulation, bundle adjustment and other processes. Compared with traditional sfm, incremental sfm can be based on the initial reconstruction result, and new images can be added one by one, so that the three-dimensional reconstruction result is iteratively updated on the basis of the initial three-dimensional reconstruction result, which is more suitable for large-scale and dynamic scene three-dimensional construction.

[0024] The process of sfm is briefly introduced as follows:

[0025] Firstly, the SIFT algorithm can be used to detect extreme points of all pixel points of the input image by constructing a multi-scale Gaussian pyramid, and then generate a 128-dimensional feature descriptor. Based on this method, hand-crafted features with scale or rotation invariance can be extracted.

[0026] Secondly, the extracted feature descriptors can be matched by calculating the Euclidean distance between any two descriptors in the input image, and then the candidate matching pairs are selected based on the calculation results. The RANSAC algorithm is used to estimate the geometric model (such as F matrix) and count the number of inliers to remove the mismatched descriptor pairs.

[0027] The geometric verification can be H or E or F matrix selected according to the scene type, such as H matrix for planar scene and F matrix for uncalibrated camera. The RANSAC iterative optimization model parameters are combined to verify the geometric consistency of the matching pairs.

[0028] The image pair with the most matching pairs is selected, and the initial three-dimensional point cloud is generated by triangulation, so as to realize the initialization of the three-dimensional reconstruction result.

[0029] Then, new images can be added one by one and the global error is optimized by bundle adjustment (BA) to incrementally reconstruct the initial three-dimensional reconstruction result.

[0030] However, the related art needs to perform multi-scale convolution operation and extreme value detection for all pixel points in the feature extraction stage, which has a large amount of calculation and takes a long time. The brute force matching needs to calculate the distance between pixel points of images pair by pair, and then match, which has an exponential growth of calculation amount and time consumption when the size of image data is large, so it is difficult to meet the real-time or quasi-real-time three-dimensional reconstruction demand.

[0031] The related art is to extract feature descriptors for the key points selected based on extreme point detection from the pixel points when extracting features. The features represented by a single key point are limited, and it is difficult to accurately reflect the semantic features of the image region where the key point is located in a complex scene. Therefore, in the face of a complex scene, such as light changes, obstacle occlusions, and repeated textures (such as multiple windows on a building, vegetation groups), the robustness is insufficient, and it is difficult to remove the feature descriptors of the false matching pairs caused by the complex scene during feature matching, resulting in a deviation in the pose of the initialized three-dimensional reconstruction result or a distortion of the three-dimensional point cloud.

[0032] Finally, the three-dimensional reconstruction process of the related art has strong coupling, resulting in a low fault tolerance. The initialization of three-dimensional reconstruction depends on the geometric verification result. Therefore, if the initial image pair selection is improper or the focal length estimation is incorrect (for example, the fixed focal length is assumed in an uncalibrated camera scene), the error will be transmitted and accumulated in the subsequent incremental reconstruction, making it difficult to converge through optimization methods, and the reconstruction failure rate is high.

[0033] To solve the above technical problems, the present disclosure provides a feature extraction method of an image.

[0034] Figure 1 is a schematic flowchart of a feature extraction method of an image according to an embodiment of the present disclosure.

[0035] As shown in Figure 1 , the feature extraction method of an image includes:

[0036] In step S101, a score map of the confidence of each pixel point of the input image as a key point is determined;

[0037] In step S102, the key points of the input image are determined based on the score map;

[0038] In step S103, the input image is divided into blocks according to the key points, wherein each block of the input image contains one key point, and each block of the image contains feature information;

[0039] In step S104, the feature descriptors of the corresponding key points are generated according to the image blocks where the key points are located.

[0040] It should be noted that the feature extraction method of an image provided by the present disclosure can be used for three-dimensional reconstruction, and the feature descriptors extracted from the input image can be used for feature matching, geometric verification, and three-dimensional modeling required by three-dimensional reconstruction.

[0041] In some embodiments, the number of input images can include at least one image.

[0042] The number of input images can be one or multiple. For example, a first image and a depth image corresponding to the first image can be included.

[0043] In some embodiments, a score map of confidence of each pixel of the input image as a key point is determined.

[0044] The score map of confidence of each pixel of the input image as a key point can be determined based on a multi-layer convolution operation, and in the score map, the value of each pixel corresponds to the possibility of the pixel being a key point.

[0045] In some embodiments, the key points of the input image are determined based on the score map.

[0046] Since the score of each pixel is used to represent the confidence of the pixel as a key point, the higher the score of a pixel, the more suitable it is as a key point. On this basis, the key points can be determined based on certain rules.

[0047] For example, a predetermined number of pixels with high scores can be determined as key points based on the score map; all pixels with scores greater than a predetermined threshold can be determined as key points regardless of the number; or the input image can be divided into multiple regions, and the pixel with the highest score in each region can be determined as a key point.

[0048] In some embodiments, the input image is divided into blocks according to the key points, wherein each block of the input image contains one key point, and each block of the image contains feature information.

[0049] After determining the key points of the input image, the information of the key points can be obtained, and the input image is divided into blocks based on the key points.

[0050] By dividing the input image into blocks and ensuring that each block of the input image contains and only contains one key point, each key point can be used to represent the features of a block of the input image, which is more conducive to representing a block of the image that can express semantic information of the image through the features of the key points.

[0051] In some embodiments, a feature descriptor corresponding to a key point is generated according to the image block where the key point is located.

[0052] After completing the division of the input image, a feature descriptor corresponding to a key point can be generated for each block of the input image.

[0053] For example, the image patch can be input into a learning network (e.g., a descriptor network), which can encode the image patch through convolutional layers and fully connected layers, and output a fixed-dimension (e.g., 128-dimension) feature descriptor.

[0054] The feature extraction method of the image according to the present disclosure can determine a score map of the input image, and determine the key points based on the score map. Compared with the related art, which needs to construct a multi-scale Gaussian pyramid and then perform extreme point detection on all pixel points to determine the key points, the present disclosure needs less computation, consumes less resources, and takes less time to determine the key points.

[0055] After determining the key points, the present disclosure can divide the input image based on the key points, ensure that each patch of the input image contains a key point, and ensure that each patch of the image contains key feature information, and then generate a feature descriptor corresponding to the key point based on the image patches. Based on the image patch containing the key point, the overall features of the image in a certain region around the key point can be reflected, so that the semantic information of the image can be more easily extracted, and the feature descriptor extracted is more accurate, and a feature descriptor with better robustness can be obtained in a complex scene.

[0056] The feature extraction method of the image according to the present disclosure will be further described below through a specific example.

[0057] Figure 2 is a schematic flowchart of a feature extraction method of an image according to an embodiment of the present disclosure.

[0058] As shown in Figure 2 , the feature extraction method of the image can include inputting an image, processing by a learning network, generating a score map, deforming the score map, extracting key points, dividing the image, and generating a descriptor.

[0059] In some embodiments, the input image can include input image information and device parameters.

[0060] At least one image can be input into the learning network, and device parameters can be input, for example, camera intrinsic and extrinsic parameters (intrinsic matrix K, rotation matrix R, and translation vector t).

[0061] The input image can be preprocessed by standardization, and the size of the image can be unified, for example, the image can be adjusted to a fixed resolution size according to the grid requirements. In the case where the input image includes a first image and a corresponding depth image, the depth image and the camera parameters can also be combined to preliminarily screen the effective area in the first image, so as to exclude the obvious noise area, which includes the area where the pixel with abnormal depth value is located.

[0062] In some embodiments, after inputting the image into the learning network, the input image can be processed based on the learning network.

[0063] For example, the LF-Net learning network (Learning Local Features from Images) can be used instead of SIFT.

[0064] The LF-Net learning network is a deep architecture for learning local features. The network includes an image-level branch and a patch-level branch. The image-level branch is a dense, multi-scale, fully convolutional detector network for returning keypoint locations, scales, and orientations. The patch-level branch is a descriptor network that outputs local feature descriptors from image patches around the keypoints generated by the detector.

[0065] In some embodiments, the pre-processed input image can be processed by the image-level branch of the LF-Net network to generate a score map.

[0066] The image-level branch network can extract multi-scale features through multi-layer convolution operations, such as convolution kernels of different scales, and output a score map that can be used to represent the confidence of each pixel in the input image as a keypoint. The numerical value of each pixel corresponds to the probability of the pixel becoming a keypoint.

[0067] In some embodiments, after generating the score map, the score map can be deformed.

[0068] The score map can be geometrically deformed using an SE(2) module (Special Euclidean group in 2D) to simulate feature distribution under different viewing angles, thereby enhancing the adaptability of key points to image rotation and translation changes. After deformation, the score map can be smoothed by a Gaussian kernel to reduce local noise interference.

[0069] In some embodiments, determining the key points of the input image based on the score map includes dividing the score map into multiple regions, determining a first pixel point with the highest score in each region, and determining a second pixel point with a score greater than a preset threshold in the first pixel point as the key point.

[0070] The score map can be divided into multiple regions, and a non-maximum suppression algorithm is used to determine the key points. For example, the score map is divided in a 3x3 pixel range. In the divided regions, the first pixel point with the highest score in each region is retained, and the remaining redundant pixel points are removed. On this basis, in combination with a preset threshold (which can be determined according to training data statistics), the second pixel point with a score greater than the preset threshold in the first pixel point is determined as a key point. According to the score, a preset number of second pixel points can also be selected from the first pixel point as key points.

[0071] After determining the key points, the subsequent process will block the image based on the key points and determine the feature descriptor based on the image block. Therefore, it is necessary to avoid selecting pixel points close to each other as key points, which will cause the image block containing complete feature information to be further split when the image is blocked based on the key points, so that the obtained image block is too small to contain complete feature information, resulting in unreliable feature descriptors extracted subsequently.

[0072] Moreover, based on dividing the score map into multiple regions, determining the maximum value as the first pixel point from the multiple regions, and then screening the key points from the first pixel point, the key point screening is not based on all pixel points, and determining the first pixel point in the region only needs to be based on the comparison of a limited number of pixel points, which can effectively reduce the data calculation amount, reduce the time consumption, and speed up the processing efficiency.

[0073] In some embodiments, after determining the key points, the key point information can be recorded.

[0074] The information of the key points includes the position (pixel point coordinates) of the key points, the scale (feature scale determined based on the multi-scale convolution result) and the direction (main direction determined by the gradient direction statistics of the pixels around the key points).

[0075] In some embodiments, the blocking the input image according to the key points includes determining the feature scale of the key points and the main direction of the key points according to the gradient direction of the pixels around the key points; and blocking the input image according to the feature scale and the main direction of each key point, wherein the size of the blocked input image is determined by the feature scale and the main direction.

[0076] For each key point, an image block (which can be 32x32 pixels in size, for example) containing the key point can be cropped by a sampler (such as a STN (spatial transformation network) sampler). The sampler can adaptively adjust the size range of the cropped image block according to the feature scale and the main direction of the key point, so as to ensure that each image block contains key feature information.

[0077] The size of the image block can be flexibly adjusted based on the feature scale and the principal direction of the key point, so that the image block can reflect the feature information of the image content around the key point in the case of containing the key point, so that the semantic feature information of the image content can be extracted from the image block subsequently, and better performance is achieved in the face of complex scenes.

[0078] In some embodiments, generating the feature descriptor of the corresponding key point according to the image block in which the key point is located comprises: performing feature encoding on the image block according to the descriptor network, and generating the feature descriptor of the corresponding key point.

[0079] The cropped image block can be input into the descriptor network, and the descriptor network can perform feature encoding on the image block through convolutional layers and fully connected layers, and output a local feature descriptor of a fixed dimension (for example, 128 dimensions). The disclosure does not limit the role of the generated feature descriptor. For example, in some embodiments, the feature descriptor can be used for feature matching, and then used for three-dimensional reconstruction.

[0080] In some embodiments, the method further comprises: optimizing the generation process of the feature descriptor based on a first loss and a second loss; wherein the first loss is used to represent the positioning error of the key point, and the second loss is used to represent the matching error of the feature descriptor.

[0081] For example, the first loss can be the loss function of the image-level branch of the LF-Net learning network, and the second loss can be the loss function of the patch-level branch of the LF-Net learning network.

[0082] In the generation process of the descriptor, the image-level branch loss (key point positioning error) and the patch-level loss (descriptor matching error) can be jointly optimized, so as to improve the distinguishability of the descriptor, so as to realize accurate positioning of the key point, and avoid the feature descriptors being too similar, which can cause false matching in subsequent feature matching.

[0083] The above embodiments of the disclosure use a lightweight LF-net deep learning network to replace SIFT, and the above image processing process can be processed by means of a GPU (Graphics Processing Unit, graphics processor), so that the processing speed is faster. The lightweight network structure combined with GPU processing can make the feature extraction speed in the above embodiments 3-5 times faster than SIFT. And generating feature descriptors based on the blocked image blocks is beneficial to capture semantic features in images, and is more suitable for complex scenes, such as illumination change scenes, occlusion scenes, and repetitive texture scenes.

[0084] In some embodiments, when the input image comprises a plurality of input images, the method further comprises: based on a self-attention mechanism and a cross-attention mechanism, performing associated matching on the key points of the plurality of input images to form first matching pairs between the key points of the plurality of input images; determining matching confidence of the first matching pairs, and determining the first matching pairs with the matching confidence greater than a first confidence threshold as second matching pairs.

[0085] The input image can comprise a plurality of input images, for example, comprising a first image and a depth image corresponding to the first image.

[0086] When the feature extraction method of the image proposed in the present disclosure is used for three-dimensional reconstruction, after feature extraction, feature matching can be performed between the first image and the depth image based on the extracted feature descriptors, so as to facilitate subsequent three-dimensional reconstruction.

[0087] For example, the key points of the plurality of input images can be associated matched based on a self-attention mechanism and a cross-attention mechanism. The self-attention mechanism is used to establish an association relationship between the key points on the same image, and the cross-attention mechanism is used to establish an association relationship between the key points of multiple images.

[0088] Since there can be complex scenes in the image, if only the key points are matched based on the feature descriptors, a large matching error can be caused. However, based on the self-attention mechanism and the cross-attention mechanism, the key points can be more accurately matched, and the stability and accuracy are higher in complex scenes.

[0089] For example, the image contains repeated texture content. In this case, the feature descriptors of the key points on the first image at the repeated texture content can be matched with the key points at the repeated texture content on the depth image. However, if the self-attention mechanism is used, it can be determined that the feature descriptors of several key points on the first image have similarity, so that it is determined that there is repeated texture content on the image, so that when matching with the depth image, the repeated texture content can be considered, thereby achieving more accurate matching of the key points.

[0090] For example, the image contains illumination changes, and part of the content on the first image appears high light on the depth image due to illumination changes, so that the feature descriptor corresponding to the part of the content on the depth image is greatly different from the feature descriptor corresponding to the part of the content on the first image. If matching is performed using related technologies, the same content may be difficult to effectively pair due to the large difference in feature descriptors caused by different illuminations. However, based on the self-attention mechanism, the embodiment can determine the association relationship between the part of the content and other key points in the first image, so that in the depth image, based on the association relationship between the part of the content and other key points in the depth image, accurate pairing can still be achieved even if there is a certain difference in the feature descriptors.

[0091] It should be noted that after the first matching pair is formed, the first matching pair with a matching confidence greater than the first confidence threshold can be determined as a second matching pair according to the matching confidence. In other words, the matched matching pairs can be further screened by the matching confidence to retain matching pairs with higher confidence. For example, matching pairs with a confidence greater than 0.7 can be retained.

[0092] In some embodiments, feature matching can include an initialization process, an attention propagation process, and a confidence prediction process.

[0093] The key point information (for example, including position, feature scale, main direction, etc.) and the corresponding feature descriptors of the plurality of input images extracted by the above embodiments can be obtained. Based on the key point information and the feature descriptors, a matching pair is determined by a neural network.

[0094] For example, a SuperGlue algorithm can be used to construct a graph neural network (GNN), and the key points of each input image can be used as graph nodes, and the node features can be the feature descriptors corresponding to the key points. The initial edge weight can be calculated based on the cosine similarity of the descriptors (used to preliminarily measure the matching possibility between nodes).

[0095] Attention propagation is performed based on the graph neural network. For example, the GNN can realize information interaction between nodes through a self-attention and a cross-attention mechanism. The self-attention mechanism is used for feature optimization of nodes in the same image, and can calculate the attention weight of each node with other nodes in the same image based on the feature similarity, and aggregate the features of the nodes related to the association relationship to update the own features. The cross-attention mechanism is used for node interaction between two images, and can calculate the matching attention weight between nodes of two images based on the feature similarity, and emphasize the association relationship of high-similarity node pairs, and pair the nodes.

[0096] After multiple rounds of GNN iterations (for example, 3-5 layers), the matching confidence of each node pair (the first matching pair, the potential matching pair of the key points in the two images) can be predicted by the output layer of the network, which is a probability value ranging from 0 to 1, and then high-confidence candidate matching pairs (i.e., the second matching pair) with a matching confidence greater than 0.7 are screened out.

[0097] Compared with the brute-force matching adopted in the related art, the feature matching manner adopted in this embodiment can reduce invalid matching calculation, improve processing efficiency, and reduce time consumption.

[0098] In some embodiments, the method further comprises: performing geometric consistency verification on the second matching pair to screen out a third matching pair; wherein the third matching pair is used for camera pose estimation and / or three-dimensional reconstruction.

[0099] The second matching pair can be further purified based on clustering.

[0100] For example, the second matching pair can be clustered, and second matching pairs that do not meet the geometric constraints are screened out through geometric consistency verification after clustering, thereby screening out the third matching pair. The accuracy and reliability of the third matching pair are higher, and therefore, the efficiency of camera pose estimation and / or three-dimensional reconstruction based on the third matching pair is higher, the time consumption is shorter, and the robustness is significantly improved.

[0101] In some embodiments, the geometric consistency verification on the second matching pair comprises: clustering the second matching pair to determine a first number of clustering clusters; estimating a geometric model for each clustering cluster and determining the proportion of matching pairs in the clustering cluster that meet the geometric constraints; and merging the second matching pairs in the clustering cluster with a proportion greater than a first proportion threshold as the third matching pair.

[0102] The pixel coordinates of the second matching pair (for example, the coordinates in the first image are (x1, y1), and the coordinates in the second image or the depth image are (x2, y2)) can be subjected to DBSCAN clustering (Density-Based Spatial Clustering of Applications with Noise), a spatial neighborhood radius (for example, 5 pixels) can be set, and / or a first number of minimum clustering points (for example, 3) can be set, thereby aggregating matching pairs with high spatial distribution consistency into the same cluster, and the clustering cluster formed after aggregation can reflect the cross-view matching of the same spatial structure.

[0103] After clustering, isolated low-confidence matching pairs can be removed. The association between these matching pairs and other nodes is weak, and the confidence is low, so they are likely to be mis-matching. Removing them helps to improve the accuracy of the matching pair.

[0104] For each cluster, geometric consistency verification can be performed. For example, RANSAC algorithm can be used to estimate a geometric model (e.g., fundamental matrix F). The proportion of inliers (matched pairs) in the cluster that satisfy the geometric constraint can be counted, and the cluster with the proportion of inliers greater than a first proportion threshold (e.g., 80%) can be retained. If the proportion of inliers is greater than the first proportion threshold, it can be considered that the cluster passes the geometric consistency verification, and it can be considered that the key point matched pairs in the cluster have high reliability.

[0105] After retaining the cluster with the proportion of inliers greater than the first proportion threshold, the cluster can be merged, and the inliers of the cluster with the proportion of inliers greater than the first proportion threshold can be merged, so as to form accurate matched pairs (third matched pairs).

[0106] It should be noted that the geometric model can include H / F / E matrix, and the type of the geometric model can be selected through scene semantic analysis (e.g., plane area proportion).

[0107] In some embodiments, the third matched pairs can be used to initialize RANSAC.

[0108] The high-confidence third matched pairs output in the above embodiments can be used to initialize RANSAC, so as to reduce the number of iterations. For example, the number of iterations can be reduced from 1000 to 200.

[0109] In the initialization stage, the camera intrinsic parameters (e.g., focal length) can be synchronized, and the "pseudo parameter" matching can be removed through BA pre-optimization.

[0110] In the embodiments of the present disclosure, the dynamic geometric verification pre-initialization intrinsic parameter optimization is combined, which can avoid accumulation of errors, improve the convergence speed of BA by 50%, and achieve sub-pixel level accuracy of the three-dimensional model.

[0111] In some embodiments, the feature extraction method of the image proposed in the present disclosure can be applied to the field of automatic driving.

[0112] The feature extraction method of the image proposed in the present disclosure can be deployed in a vehicle-mounted GPU, so as to realize real-time feature extraction and matching of dynamic scenes (e.g., urban road scenes), and improve the SLAM accuracy (Simultaneous Localization and Mapping Accuracy) by combining inertial navigation.

[0113] In some embodiments, the feature extraction method of the image proposed in the present disclosure can be applied to the field of medical image three-dimensional reconstruction.

[0114] The organ boundary features can be extracted from CT / MRI sequence images by the method of the present disclosure, and then the precise matching of cross-layer highlighting is realized, thereby assisting the three-dimensional modeling of tumors.

[0115] Corresponding to the embodiments of the feature extraction method of the image of the present disclosure, the present disclosure also provides corresponding embodiments of the feature extraction device of the image.

[0116] Please refer to Figure 3 , Figure 3 is a block diagram of the feature extraction device of the image in an embodiment of the present disclosure. As Figure 3 shown, the feature extraction device of the image comprises:

[0117] The score map determination unit 310 is configured to determine a score map of the confidence of each pixel point of the input image as a key point;

[0118] The key point determination unit 320 is configured to determine the key points of the input image based on the score map;

[0119] The image blocking unit 330 is configured to block the input image according to the key points, wherein each block of the input image contains one key point, and each block of the image contains feature information;

[0120] The descriptor generation unit 340 is configured to generate a feature descriptor of the corresponding key point according to the image block where the key point is located.

[0121] In some embodiments, the blocking of the input image according to the key points comprises: determining the feature scale of the key point, and determining the main direction of the key point according to the gradient direction of the pixels around the key point; blocking the input image according to the feature scale and the main direction of each key point, wherein the size of the blocked input image is determined by the feature scale and the main direction.

[0122] In some embodiments, the device is further configured to optimize the generation process of the feature descriptor based on a first loss and a second loss; wherein the first loss is used to represent the positioning error of the key point, and the second loss is used to represent the matching error of the feature descriptor.

[0123] In some embodiments, in the case where the input image comprises a plurality of input images, the device is further configured to: based on a self-attention mechanism and a cross-attention mechanism, associate and match the key points of the plurality of input images to form a first matching pair between the key points of the plurality of input images; determine the matching confidence of the first matching pair, and determine the first matching pair with the matching confidence greater than a first confidence threshold as a second matching pair.

[0124] In some embodiments, the apparatus is further configured to perform geometric consistency verification on the second matching pairs to screen out third matching pairs, wherein the third matching pairs are used for camera pose estimation and / or three-dimensional reconstruction.

[0125] In some embodiments, the geometric consistency verification on the second matching pairs comprises: clustering the second matching pairs to determine a first number of cluster groups; estimating a geometric model for each cluster group and determining a proportion of matching pairs in the cluster group that meet a geometric constraint; and merging the second matching pairs in a cluster group whose proportion is greater than a first proportion threshold as the third matching pairs.

[0126] In some embodiments, the determining the key points of the input image based on the score map comprises: dividing the score map into a plurality of regions; determining a first pixel point with the highest score in each region; and determining a second pixel point with a score greater than a preset threshold in the first pixel point as the key point.

[0127] The implementation process of the functions and roles of each unit in the above apparatus is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0128] Embodiments of the present disclosure also propose an electronic device, comprising: a processor, a memory; the memory is used to store a computer program; the processor is used to execute the feature extraction method of the image as any of the above embodiments by calling the computer program.

[0129] Embodiments of the present disclosure also propose a computer readable storage medium, which stores a computer program, characterized in that the program is executed by a processor to implement the feature extraction method of the image as any of the above embodiments.

[0130] Embodiments of the present disclosure also propose a computer program product comprising a computer program, which is executed by a processor to implement the method as described in any of the above embodiments.

[0131] Figure 4 is a schematic block diagram of a feature extraction device 400 for images according to an embodiment of the present disclosure. For example, the device 400 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0132] Referring to Figure 4 , the device 400 can include one or more of the following components: a processing component 402, a memory 404, a power supply component 406, a multimedia component 408, an audio component 410, an input / output (I / O) interface 412, a sensor component 414, and a communication component 416.

[0133] The processing component 402 generally controls the overall operations of the device 400, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 402 can include one or more processors 420 to execute instructions to complete the above-mentioned steps of the feature extraction method of images, in whole or in part. In addition, the processing component 402 can include one or more modules to facilitate the interaction between the processing component 402 and other components. For example, the processing component 402 can include a multimedia module to facilitate the interaction between the multimedia component 408 and the processing component 402.

[0134] The memory 404 is configured to store various types of data to support operations of the device 400. Examples of these data include instructions for any application or methods operating on the device 400, contact data, phonebook data, messages, pictures, videos, and so on. The memory 404 can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0135] The power component 406 provides power to the various components of the device 400. The power component 406 can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the device 400.

[0136] The multimedia component 408 includes a screen providing an output interface between the device 400 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes the touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensors can not only sense a boundary of a touching or sliding action, but also detect duration and pressure related to the touching or sliding action. In some embodiments, the multimedia component 408 includes a front camera and / or a back camera. The front camera and / or the back camera can receive external multimedia data when the device 400 is in an operating mode, such as a shooting mode or a video mode. Each of the front and back cameras can be a fixed optical lens system or have a focal length and optical zoom capability.

[0137] The audio component 410 is configured to output and / or input audio signals. For example, the audio component 410 includes a microphone (MIC) that is configured to receive an external audio signal when the device 400 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 404 or transmitted via the communication component 416. In some embodiments, the audio component 410 also includes a speaker for outputting audio signals.

[0138] The I / O interface 412 provides an interface between the processing component 402 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0139] The sensor component 414 includes one or more sensors for providing status assessments of various aspects of the device 400. For example, the sensor component 414 can detect an open / closed position of the device 400, relative positioning of components, such as a display and a keypad of the device 400, a change of position of the device 400 or a component of the device 400, presence or absence of user contact with the device 400, orientation or acceleration / deceleration / g-force and temperature of the device 400. The sensor component 414 can include an orientation sensor, an acceleration sensor, a

[0140] The communication component 416 is configured to facilitate wired or wireless communication between the device 400 and another device. The device 400 can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G LTE, 5G NR, or a combination thereof. In an example embodiment, the communication component 416 receives broadcast signals or broadcast-related information from an external broadcasting management system via a broadcast channel. In an example embodiment, the communication component 416 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) techniques, infrared data association (IrDA) techniques, ultra-wideband (UWB) techniques, Bluetooth (BT) techniques, and other techniques.

[0141] In exemplary embodiments, the apparatus 400 can be implemented using one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic devices, to perform the above-described image feature extraction method.

[0142] In exemplary embodiments, a non-transitory computer readable storage medium including instructions, such as the memory 404 including instructions, is also provided, which can be executed by the processor 420 of the apparatus 400 to complete the above-described image feature extraction method. For example, the non-transitory computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0143] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the present disclosure cover any and all variations of the present disclosure including those variations that can be incorporated into the above detailed description and making use of the general principles of the present disclosure, which are defined in the following claims. The specification and examples given herein are only exemplary and are intended to provide an overview or framework for understanding the nature and character of the disclosure. The true scope of the present disclosure is defined by the following claims.

[0144] It should be understood that the present disclosure is not limited to the precise structures as set forth above and shown in the attached drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the claims appended hereto.

[0145] It should be noted that, in the present document, relational terms such as first and second, and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. The terms "includes", "including", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0146] The method and device provided by the embodiments of the present disclosure are described in detail above, the principles and implementation manners of the present disclosure are described by applying specific examples, the above embodiment description is only used to help understand the method of the present disclosure and the core idea thereof; meanwhile, for those skilled in the art, according to the idea of the present disclosure, the specific implementation manners and application ranges will have changes, and the above description should not be understood as a limitation on the present disclosure.

Claims

1. A feature extraction method of an image, characterized by, The method comprises: determining a score map of the confidence of each pixel point of an input image as a key point; determining a key point of the input image based on the score map; dividing the input image according to the key point, wherein each block of the input image contains one key point, and each block of the image contains feature information; generating a feature descriptor of the corresponding key point according to the image block where the key point is located.

2. The method of claim 1, wherein, The dividing of the input image according to the key point comprises: determining the feature scale of the key point, and determining the main direction of the key point according to the gradient direction of the pixels around the key point; dividing the input image according to the feature scale and the main direction of each key point, wherein the size of the divided input image is determined by the feature scale and the main direction.

3. The method of claim 1, wherein, The method further comprises: optimizing the generation process of the feature descriptor based on a first loss and a second loss; wherein the first loss is used to represent the positioning error of the key point, and the second loss is used to represent the matching error of the feature descriptor.

4. The method of claim 1, wherein, In the case where the input image comprises a plurality of input images, the method further comprises: based on a self-attention mechanism and a cross-attention mechanism, associating and matching the key points of the plurality of input images to form a first matching pair between the key points of the plurality of input images; determining the matching confidence of the first matching pair, and determining the first matching pair with a matching confidence greater than a first confidence threshold as a second matching pair.

5. The method of claim 4, wherein, The method further comprises: performing geometric consistency verification on the second matching pair to filter out a third matching pair; wherein the third matching pair is used for camera pose estimation and / or three-dimensional reconstruction.

6. The method of claim 5, wherein, The geometric consistency verification on the second matching pair comprises: clustering the second matching pair to determine a first number of clustering clusters; estimating a geometric model for each clustering cluster, and determining the proportion of matching pairs in the clustering cluster that meet the geometric constraint; merging the second matching pairs in the clustering cluster with a proportion greater than a first proportion threshold as the third matching pair.

7. The method according to any one of claims 1 to 6, characterized in that, The determination of the key point of the input image based on the score map comprises: dividing the score map into a plurality of regions; determining a first pixel point with the highest score in each region; determining a second pixel point with a score greater than a preset threshold in the first pixel point as the key point.

8. An image feature extraction apparatus characterized by comprising: The device comprises: a score map determination unit configured to determine a score map of the confidence of each pixel point of an input image as a key point; a key point determination unit configured to determine a key point of the input image based on the score map; an image division unit configured to divide the input image according to the key point, wherein each block of the input image contains one key point, and each block of the image contains feature information; a descriptor generation unit configured to generate a feature descriptor of the corresponding key point according to the image block where the key point is located.

9. An electronic device, comprising: comprise: a processor, a memory; the memory is used to store a computer program; the processor is used to execute the feature extraction method of the image as claimed in any one of claims 1-7 by calling the computer program.

10. A computer program product, characterised in that, A computer program comprising computer program elements which, when executed by a processor, implement the method described in any one of claims 1-7. A computer program comprising computer program elements which, when executed by a processor, implement the method described in any one of claims 1-7.

Images