Image feature extraction method, system and medium with repeatability and reliability

Abstract

The application discloses an image feature extraction method and system with repeatability and reliability, and a medium, comprising: extracting feature descriptors and feature detectors in parallel for each RGB image through a full convolution network; training a trained detector network model with local perception and global perception through similarity loss between different view subgraphs of the feature detector and correlation loss according to the feature descriptors and the feature detectors; inputting a to-be-detected image, calculating descriptors and detectors according to the detector network model, extracting key points in the image through the detectors and optimizing the key points using non-maximum suppression, selecting key point positions greater than a preset threshold in the detectors, and obtaining feature vectors in corresponding positions of the descriptors; and the application simultaneously has the detectors with local perception and global perception through similarity and relative loss learning, has reliability and repeatability, and improves feature detection efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and machine learning, and more specifically to image feature extraction methods, systems, and media that are repeatable and reliable. Background Technology

[0002] Image feature point extraction and detection have wide applications in visual positioning, camera pose estimation and 3D reconstruction. The accuracy, repeatability, robustness and specificity of detected feature points facilitate more accurate feature point matching and more detailed 3D model reconstruction.

[0003] Currently, traditional methods for extracting feature points by calculating response values ​​using image operators (such as DOG, LOG, DOH, etc.) have poor robustness. In recent years, feature point extraction methods based on machine learning have become popular.

[0004] Some methods supervise the location of feature points: for example, TILDE uses feature points extracted by SIFT as supervision to train a classifier; however, using SIFT-extracted feature points as supervision cannot guarantee the robustness of the feature points, and can only enhance them. TCDET uses feature points extracted by TILDE as supervision to train a network to estimate the affine transformation of patches and divides the image into a grid, detecting keypoints based on the rules of the affine transformation of the grid. Unlike the above methods, SuperPoint uses a self-supervised approach to predict keypoints. It pre-trains the model by generating simple geometry and obtaining real keypoints, and then uses the pre-trained model to train the detector; however, the simple geometry differs greatly from the real scene.

[0005] Since feature points in datasets are difficult to label, many detector training methods do not supervise the location of feature points. Quad-Network first proposed a method of randomly collecting point pairs and comparing them to train the score map and obtain key points. Because Quad-Network's network structure and loss function are relatively simple, "Learning to DetectFeatures in Texture Images" proposed rank loss and peak loss to improve the repeatability and reliability of features. Based on the above methods that do not supervise the location of feature points, R2D2 improves the network structure and uses peak loss and cosine loss to improve the repeatability of feature points, and uses AP loss to improve the reliability of feature points. The feature points extracted by this method have good performance in image matching, 3D reconstruction and other tasks.

[0006] Although R2D2 takes reliability and repeatability into account, this method uses two detectors to detect keypoint extraction, which has high algorithmic complexity; and there is still room for improvement in the repeatability of keypoints.

[0007] Therefore, how to provide an image feature extraction method with repeatability and reliability is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0008] In view of this, the present invention provides an image feature extraction method, system, and medium with repeatability and reliability to solve some of the technical problems mentioned in the background art.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A repeatable and reliable image feature extraction method includes the following steps:

[0011] S1. Acquire images and perform viewpoint transformation to obtain image pairs. For each RGB image of the image pair, extract feature descriptors and feature detectors in parallel through a fully convolutional network.

[0012] S2. Process the feature detectors to obtain sub-graphs, and train them using similarity loss to obtain a keypoint detector for predicting global perception;

[0013] S3. Based on the feature descriptors and feature detectors, the relative relationship between the feature descriptors and detectors is learned through relative loss training, and the detectors with local awareness are predicted to obtain a trained detector network model with both local and global awareness.

[0014] S4. Input the image to be tested. Based on the detector network model, calculate the descriptor and detector. Use non-maximum suppression to extract key points in the image through the detector and optimize them. Select the key point positions in the detector that are greater than the preset threshold and obtain the feature vector of the corresponding position in the descriptor. These are the extracted key point positions and feature vectors.

[0015] Preferably, the image feature extraction method with repeatability and reliability further includes:

[0016] S5. Scaling the RGB image to different sizes, repeating steps S1-S4 at multiple scales, and selecting key points by score.

[0017] Preferably, in step S1, the fully convolutional network contains 6 dilated convolutional blocks and 3 convolutional layers. A detector is generated from the feature map using one convolutional layer and a softplus function, and a descriptor is generated from the feature map using a square operation and an L2 normalization.

[0018] Preferably, step S2 includes the following:

[0019] S21. Perform a viewpoint transformation operation on the feature detector;

[0020] S22. Divide the detectors from the two perspectives into multiple patches and extract the pixels at the same position in each patch, then stitch and stack them to obtain the sub-images of the detectors from the two perspectives, and compress them into a one-dimensional vector.

[0021] S23. The similarity between the detectors extracted from the two perspectives is calculated by calculating the cosine similarity between the one-dimensional vectors generated after the subgraph compression of the detectors from the two perspectives.

[0022] S24. Use a similarity loss function to train and obtain a keypoint detector for predicting global awareness.

[0023] Preferably, the specific content of step S3 is as follows:

[0024] S31. Based on the extracted feature descriptors, calculate the local specificity of each pixel in the feature descriptor with respect to the surrounding region;

[0025] S32. Calculate the square of the difference between the extracted feature detector and the local specificity, denoted as the correlation loss function;

[0026] S33. Use the correlation loss function for training and extract feature points with local specificity through the detector.

[0027] Preferably, the image feature extraction method with repeatability and reliability further includes training a reliable descriptor using the AP loss function and training a detector with sharp characteristics at key points using the Peaky loss function.

[0028] Preferably, the similarity loss function in step S2 is:

[0029]

[0030] Where V is the vector corresponding to the subgraphs R and T at the same position (u, v) in the image pair, and p is the size of the subgraph.

[0031] Preferably, the correlation loss function in step S3 is:

[0032] L relative =∑(Local(F, i, j) - D i，j ) 2

[0033] in,

[0034]

[0035] Where Local(F,i,j) is the difference between the pixel feature in the i-th row and j-th column of the image and the features of the surrounding pixels, D is the feature detector, F is the feature descriptor, and u and v are the offsets of the pixel in the surrounding region.

[0036] A repeatable and reliable image feature extraction system, based on the aforementioned repeatable and reliable image feature extraction method, includes: a feature extraction module, a key point extraction module, and a training module.

[0037] The feature extraction module is used to extract feature descriptors and feature detectors for each RGB image in parallel using a fully convolutional network.

[0038] The training module is used to process the feature detectors to obtain sub-graphs from different perspectives, and to train the key point detector with global awareness by using the similarity loss between sub-graphs. Based on the feature descriptors and feature detectors, the module trains the relative relationship between the feature descriptors and detectors through the correlation loss, predicts the detectors with local awareness, and obtains a trained detector network model with both local and global awareness.

[0039] The key point extraction module takes the image to be tested as input, calculates descriptors and detectors based on the detector network model, uses non-maximum suppression to extract key points in the image through the detectors and optimizes them, selects key point positions in the detectors that are greater than a preset threshold, and obtains the feature vectors of the corresponding positions in the descriptors, which are the extracted key point positions and feature vectors.

[0040] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image feature extraction method described above, which is repeatable and reliable.

[0041] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses an image feature extraction method, system and medium with repeatability and reliability. It learns a detector with both local and global awareness through similarity and relative loss, which has reliability and repeatability, improves the efficiency and accuracy of feature detection algorithms, and helps to improve the effectiveness in image matching, 3D reconstruction, visual localization and homography estimation. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0043] Figure 1 A schematic diagram illustrating an image feature extraction method with repeatability and reliability provided by the present invention;

[0044] Figure 2 This is a schematic diagram of the sub-graph acquisition of the detector provided by the present invention;

[0045] Figure 3 This is a schematic diagram comparing the detector provided by the present invention and the R2D2 detector;

[0046] Figure 4 This diagram illustrates a comparison of the features provided by the present invention with other methods in image matching tasks. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] This invention discloses an image feature extraction method with repeatability and reliability, such as... Figure 1 This includes the following steps:

[0049] S1. Acquire images and perform viewpoint transformation to obtain image pairs. For each RGB image of the image pair, extract feature descriptors and feature detectors in parallel through a fully convolutional network.

[0050] S2. Process the feature detectors to obtain sub-graphs, and train them using similarity loss to obtain a keypoint detector for predicting global perception;

[0051] S3. Based on the feature descriptors and feature detectors, the relative relationship between the feature descriptors and detectors is learned through relative loss training, and the detectors with local awareness are predicted to obtain a trained detector network model with both local and global awareness.

[0052] S4. Input the image to be tested. Based on the detector network model, calculate the descriptor and detector. Use non-maximum suppression to extract key points in the image through the detector and optimize them. Select the key point positions in the detector that are greater than the preset threshold and obtain the feature vector of the corresponding position in the descriptor. These are the extracted key point positions and feature vectors.

[0053] To further implement the above technical solution, an image feature extraction method with repeatability and reliability further includes:

[0054] S5. Scaling the RGB image to different sizes, repeating steps S1-S4 at multiple scales, and selecting key points by score.

[0055] To further implement the above technical solution, in step S1, the fully convolutional network contains 6 dilated convolutional blocks and 3 convolutional layers. A detector is generated from the feature map using one convolutional layer and a softplus function, and a descriptor is generated from the feature map using a square operation and an L2 normalization.

[0056] To predict a globally aware detector, a low-resolution sub-detector is applied to learn the globally aware detector in low resolution.

[0057] To further implement the above technical solutions, such as Figure 2 The specific content of step S2 includes:

[0058] S21. In order to compare the detection pairs with the same viewpoint, a viewpoint transformation operation is performed on the feature detections;

[0059] S22. Divide the detectors from the two perspectives into multiple patches and extract the pixels at the same position in each patch, then stitch and stack them to obtain the sub-images of the detectors from the two perspectives, and compress them into a one-dimensional vector.

[0060] S23. The similarity between the detectors extracted from the two perspectives is calculated by calculating the cosine similarity between the one-dimensional vectors generated after the subgraph compression of the detectors from the two perspectives.

[0061] S24. Use a similarity loss function to train and obtain a keypoint detector for predicting global awareness.

[0062] To further implement the above technical solution, the specific content of step S3 is as follows:

[0063] S31. Based on the extracted feature descriptors, calculate the local specificity of each pixel in the feature descriptor with respect to the surrounding region;

[0064] S32. Calculate the square of the difference between the extracted feature detector and the local specificity, denoted as the correlation loss function;

[0065] S33. Use the correlation loss function for training and extract feature points with local specificity through the detector.

[0066] To further implement the above technical solution, an image feature extraction method with repeatability and reliability also includes training a reliable descriptor using the AP loss function and training a detector with sharp characteristics at key points using the Peaky loss function; firstly, the true matching pixel pairs are defined as positive samples, while other pairs are defined as negative samples; for a given image pair, a set of points is sampled in the image using a random sampler, and during training, this part of the loss function is trained in parallel with the loss functions in steps S2-S3.

[0067] To further implement the above technical solution, the similarity loss function in step S2 is:

[0068]

[0069] Where V is the vector corresponding to the subgraphs R and T at the same position (u, v) in the image pair, and p is the size of the subgraph.

[0070] To further implement the above technical solution, the correlation loss function in step S3 is:

[0071] L relative =∑(Local(F,i,j)-D i，j ) 2

[0072] in,

[0073]

[0074] Where Local(F,i,j) is the difference between the pixel feature in the i-th row and j-th column of the image and the features of the surrounding pixels, D is the feature detector, F is the feature descriptor, and u and v are the offsets of the pixel in the surrounding region.

[0075] In this embodiment, the calculation principle of the correlation loss function is as follows: First, the detectors and descriptors of the image are obtained through a neural network; on the descriptor, the difference between the descriptor corresponding to each pixel and the surrounding pixels is calculated, i.e., Local(F,i,j) in the formula; the greater the specificity of the pixel on the descriptor, the higher the corresponding detector score. Therefore, the specificity of the detector and the pixel on the descriptor are compared in the loss function, so that key points with sufficient specificity can be detected in feature detection.

[0076] In one embodiment, the results of the detector of the present invention and the R2D2 detector are compared, such as Figure 3 As shown, the method of the present invention can effectively filter out key points that are not conducive to matching, such as the sky and rivers.

[0077] In another embodiment, image pairs are generated using a random homography matrix and random photometric distortion from the Revisited Oxford & Pairs dataset and the Aachen Day-Light dataset. In particular, the Revisited Oxford & Pairs dataset, generated by modifying the annotations and image sizes of the Oxford and Pairs building datasets, contains over 10,000 images, from which 3,125 images were selected for training. The Aachen dataset, taken in the old town of Aachen, Germany, contains 4,479 references to 369 query images. Automatic style transfer is applied to reprocess the images, and optical flow is used for annotation. The two datasets are merged, resulting in 16,010 image pairs for the training dataset.

[0078] In this embodiment, the model's performance is evaluated on three evaluation criteria: a 2D image matching task on the Hpatches dataset and the FM evaluation criterion, and a 3D reconstruction task on the ETH evaluation criterion. Specifically:

[0079] The 2D image matching task was evaluated using the Hpatches dataset, a well-known 2D homography estimation benchmark containing 116 image sequences with ground truth homography matrices. The average matching accuracy on this dataset was evaluated using 52 and 56 sequences with illumination and viewpoint variations, respectively. Figure 4 ;

[0080] The performance of the method of the present invention in real and complex 2D scenes was further evaluated using the FM evaluation metric. This benchmark combined four datasets, namely TMU, KITTI, Tanks and Temples (T&T), and Community PhotoCollection (CPC), all of which were taken in real-world scenarios. For each dataset, 1000 overlapping image pairs were randomly selected for evaluation with the real basis matrix. Compared with R2D2, the present invention improved the mean recall, first-frame intra-point rate, and second-frame intra-point rate by 21%, 5.1%, and 19.1%, respectively.

[0081] The 3D reconstruction performance of this invention on the ETH evaluation metric was evaluated using two small datasets, Fountain and Herzjesu, each consisting of approximately 10 high-resolution images, and three medium-sized datasets, Madrid Metropolis, Gendarmenmarkt, and Tower of London. The results show that the method of this invention achieves the most comprehensive reconstruction in terms of the number of reconstructed images and the average reprojection error; for example, the method of this invention outperforms R2D2, achieving an enhancement of 42.5%.

[0082] These results demonstrate that the loss function proposed in this invention is effective in improving the repeatability and reliability of keypoint detectors.

[0083] A repeatable and reliable image feature extraction system, based on a repeatable and reliable image feature extraction method, includes: a feature extraction module, a key point extraction module, and a training module.

[0084] The feature extraction module is used to extract feature descriptors and feature detectors for each RGB image in parallel using a fully convolutional network.

[0085] The training module is used to process the feature detectors to obtain sub-graphs from different perspectives, and to train the key point detector with global awareness by using the similarity loss between sub-graphs. Based on the feature descriptors and feature detectors, the module trains the relative relationship between the feature descriptors and detectors through the correlation loss, predicts the detectors with local awareness, and obtains a trained detector network model with both local and global awareness.

[0086] The key point extraction module is used to take the image to be tested as input, calculate descriptors and detectors according to the detector network model, extract key points in the image through the detectors using non-maximum suppression, optimize, select key point positions in the detectors that are greater than a preset threshold, and obtain the feature vectors of the corresponding positions in the descriptors, which are the extracted key point positions and feature vectors.

[0087] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a repeatable and reliable image feature extraction method.

[0088] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0089] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image feature extraction method with repeatability and reliability, characterized in that, Includes the following steps: S1. Acquire images and perform viewpoint transformation to obtain image pairs. For each RGB image of the image pair, extract feature descriptors and feature detectors in parallel through a fully convolutional network. S2. Process the feature detectors to obtain sub-graphs from different perspectives, and train them using the similarity loss between sub-graphs to obtain a keypoint detector for predicting global perception. S3. Based on the feature descriptors and feature detectors, the relative relationship between the feature descriptors and detectors is learned through correlation loss training, and the detectors with local awareness are predicted to obtain a trained detector network model with both local and global awareness. S4. Input the image to be tested. According to the detector network model, calculate the descriptor and detector. Use non-maximum suppression to extract key points in the image through the detector and optimize them. Select the key point positions in the detector that are greater than the preset threshold and obtain the feature vector of the corresponding position in the descriptor. These are the extracted key point positions and feature vectors. The specific content of step S2 includes: S21. Perform a viewpoint transformation operation on the feature detector; S22. Divide the detectors from the two perspectives into multiple patches and extract the pixels at the same position in each patch, then stitch and stack them to obtain the sub-images of the detectors from the two perspectives, and compress them into a one-dimensional vector. S23. The similarity between the detectors extracted from the two perspectives is calculated by calculating the cosine similarity between the one-dimensional vectors generated after the subgraph compression of the detectors from the two perspectives. S24. Use a similarity loss function to train and obtain a keypoint detector for predicting global awareness; The similarity loss function is: ； Where V is the vector corresponding to the subgraphs R and T at the same position (u, v) in the image pair, and p is the size of the subgraph; The specific content of step S3 is as follows: S31. Based on the extracted feature descriptors, calculate the local specificity of each pixel in the feature descriptor with respect to the surrounding region; S32. Calculate the square of the difference between the extracted feature detector and the local specificity, denoted as the correlation loss function; S33. Use the correlation loss function for training and extract feature points with local specificity through the detector; The correlation loss function is: ； in, ； Where Local(F,i,j) is the difference between the pixel feature in the i-th row and j-th column of the image and the features of the surrounding pixels, D is the feature detector, F is the feature descriptor, and u and v are the offsets of the pixel in the surrounding region.

2. The image feature extraction method with repeatability and reliability according to claim 1, characterized in that, Also includes: S5. Scaling the RGB image to different sizes, repeating steps S1-S4 at multiple scales, and selecting key points by score.

3. The image feature extraction method with repeatability and reliability according to claim 1, characterized in that, In step S1, the fully convolutional network contains 6 dilated convolutional blocks and 3 convolutional layers. A detector is generated from the feature map using one convolutional layer and a softplus function, and a descriptor is generated from the feature map using a square operation and an L2 normalization.

4. The image feature extraction method with repeatability and reliability according to claim 1, characterized in that, It also includes training reliable descriptors using the AP loss function and training detectors with sharp features at key points using the Peaky loss function.

5. An image feature extraction system with repeatability and reliability, characterized in that, The image feature extraction method with repeatability and reliability according to any one of claims 1-4 includes: a feature extraction module, a key point extraction module, and a training module; The feature extraction module is used to extract feature descriptors and feature detectors for each RGB image in parallel using a fully convolutional network. The training module is used to process the feature detectors to obtain sub-graphs from different perspectives, and to train the key point detector with global awareness by using the similarity loss between sub-graphs. Based on the feature descriptors and feature detectors, the module trains the relative relationship between the feature descriptors and detectors through the correlation loss, predicts the detectors with local awareness, and obtains a trained detector network model with both local and global awareness. The key point extraction module is used to take the image to be tested as input, calculate descriptors and detectors according to the detector network model, extract key points in the image through the detectors using non-maximum suppression, optimize, select key point positions in the detectors that are greater than a preset threshold, and obtain the feature vectors of the corresponding positions in the descriptors, which are the extracted key point positions and feature vectors.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the image feature extraction method with repeatability and reliability as described in any one of claims 1-4.

Images