Image feature extraction method and device, storage medium and electronic equipment

Abstract

The invention discloses an image feature extraction method and device, a storage medium and electronic equipment. The method comprises the following steps: sampling an image to be processed by adopting a plurality of sampling scales to obtain a first image, the first image comprising a plurality of grids; for each grid, determining a filtering threshold value according to the pixel values of the pixel points in each grid; performing feature detection on the pixel points in the corresponding grids based on the filtering threshold values to obtain a plurality of feature points; performing region coverage suppression processing on the plurality of feature points, and removing redundant feature points in the plurality of feature points to obtain target feature points; and performing feature description on the target feature point to obtain a feature vector corresponding to the target feature point. According to the scheme provided by the embodiment of the invention, the real-time performance and robustness of image feature extraction can be considered.

Description

Technical Field

[0001] The present application belongs to the field of feature extraction technology, and in particular relates to an image feature extraction method, device, storage medium and electronic device. Background Art

[0002] Image feature point extraction is an important task in computer vision, which is widely used in image matching, three-dimensional reconstruction, target recognition and other fields.

[0003] However, related technologies typically extract image features based on the grayscale difference between the pixel being detected and its surrounding pixels. When image contrast differs significantly, this often results in a small number of feature points and a concentrated distribution. Furthermore, these algorithms suffer from poor image scale invariance and robustness. Summary of the Invention

[0004] The embodiments of the present application provide a method, device, storage medium, and electronic device for extracting image features, which can take into account both the real-time and robustness of image feature extraction.

[0005] In a first aspect, an embodiment of the present application provides a method for extracting image features, the method comprising: sampling an image to be processed using multiple sampling scales to obtain a first image, the first image comprising multiple grids; for each grid, determining a filtering threshold based on the pixel values of the pixels within each grid; performing feature detection on the pixel points in the corresponding grid based on the filtering threshold to obtain multiple feature points; performing regional coverage suppression processing on the multiple feature points to eliminate redundant feature points from the multiple feature points to obtain target feature points; performing feature description on the target feature points to obtain feature vectors corresponding to the target feature points.

[0006] In the second aspect, an embodiment of the present application provides an image feature extraction device, which includes: a sampling module, which is used to sample the image to be processed using an image pyramid with multiple different scales to obtain a first image, and the first image includes multiple grids; a threshold determination module, which is used to determine the filtering threshold for each grid according to the pixel value of the pixel point in each grid; a feature detection module, which is used to perform feature detection on the pixel points in the corresponding grid based on the filtering threshold to obtain multiple feature points; a feature filtering module, which is used to perform regional coverage suppression processing on the multiple feature points, eliminate redundant feature points in the multiple feature points, and obtain target feature points; a feature description module, which is used to perform feature description on the target feature points to obtain feature vectors corresponding to the target feature points.

[0007] In a third aspect, an embodiment of the present application provides an electronic device comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the image feature extraction method described in the first aspect is implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the method for extracting image features as described in the first aspect is implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the image feature extraction method as described in the first aspect.

[0010] As can be seen from the above content, in the embodiment of the present application, multiple sampling scales are used to sample the image to extract the feature points in the image. While improving the real-time performance of image feature extraction, the scale invariance of image feature extraction is also improved, thereby improving the robustness. In addition, in the embodiment of the present application, the filtering threshold is determined based on the pixel value of the pixel point in the grid, that is, in the embodiment of the present application, for each grid, there is a different filtering threshold corresponding to it, and for different grids, the use of corresponding filtering thresholds can improve the accuracy of feature point detection, thereby improving the accuracy and robustness of image feature extraction. In order to further homogenize the feature points in the image, the influence of redundant feature points on the accurate analysis of image features will be eliminated. In the embodiment of the present application, the redundant feature points are eliminated by regional coverage suppression to reduce the computational complexity of adaptive non-maximum suppression, thereby improving the real-time and robustness of image feature extraction.

[0011] It can be seen that the solution provided by the embodiment of the present application can take into account both the real-time and robustness of image feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0013] Figure 1 This is a flowchart of an image feature extraction method provided by an embodiment of the present application;

[0014] Figure 2 is a schematic diagram of regional coverage suppression provided by an embodiment of the present application;

[0015] Figure 3 This is a schematic diagram of the principle of sub-pixel interpolation provided by an embodiment of the present application;

[0016] Figure 4 This is a schematic diagram of the overall process of an image feature extraction method provided by an embodiment of the present application;

[0017] Figure 5 is a structural diagram of an image feature extraction device provided by another embodiment of the present application;

[0018] Figure 6 This is a structural diagram of an electronic device provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0019] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present application by illustrating the examples of the present application.

[0020] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, the elements defined by the phrase "comprising..." do not exclude the presence of other identical elements in the process, method, article, or device comprising the elements.

[0021] For ease of understanding, before explaining the solution provided in this application, the background of the solution provided in this application is first explained.

[0022] Image feature point extraction is a crucial task in computer vision, widely used in image matching, 3D reconstruction, object recognition, and other fields. Currently, image feature extraction methods can be categorized into traditional methods and deep learning-based methods. Traditional methods include classic scale-invariant feature transformation algorithms, accelerated robust feature algorithms, and directional rotation robustness algorithms. Deep learning methods include methods based on CNNs (Convolutional Neural Networks) and SuperPoint self-supervised learning.

[0023] At present, image feature extraction algorithms are increasingly used in the field of UAV target detection. In the field of UAV target detection, the quadtree method can be used to realize the detection of image feature points, and CNN can also be used to detect and describe image feature points.

[0024] However, the method of using quadtree to detect feature points may result in too many splits, thus affecting the real-time performance of the algorithm. Moreover, in this method, the BEBLID (Boosted Efficient Binary Local Image Descriptor) algorithm is used to describe feature points, but the BEBLID algorithm has a relatively weak ability to identify and distinguish similar but different image features, and the algorithm lacks robustness.

[0025] The method of using CNN to detect and describe image feature points has a relatively complex calculation process and high training cost, and is not suitable for scenarios such as drones that require real-time image processing.

[0026] In related technologies, traditional ORB (Oriented FAST and Rotated BRIEF) feature point extraction is based on the grayscale difference between the pixel to be detected and its surrounding pixels. This method is computationally simple and highly real-time. However, when image contrast varies significantly, the number of feature points is often small and concentrated, and the algorithm also exhibits poor scale invariance and robustness.

[0027] In response to the problems existing in the related art, the embodiments of the present application provide a method, device, storage medium and electronic device for extracting image features, which not only ensure the high real-time performance of image feature extraction, but also ensure the accuracy and robustness of image feature extraction.

[0028] The following first introduces the image feature extraction method provided in the embodiment of the present application. Among them, the image feature extraction method provided in the embodiment of the present application can be applied to but not limited to technical fields such as drone emergency communication and communication line detection, and the image processing system can serve as the execution subject of the method provided in the embodiment of the present application. As an example, the image processing system may include a computing device (for example, a computer) that can perform image processing on the image to be processed, extract and display the feature vectors corresponding to the feature points obtained by feature extraction of the image to be processed.

[0029] In one embodiment, Figure 1 FIG. 1 is a flow chart showing a method for extracting image features provided by an embodiment of the present application. Figure 1 As shown, the method includes the following steps S101 to S105:

[0030] Step S101 : sampling an image to be processed using multiple sampling scales to obtain a first image.

[0031] In step S101, the image to be processed may be an image captured by a drone. As an example, the image processing system may include a server that communicates with the drone, receives the image captured by the drone, extracts image features from the image, and then returns the image features to the drone, enabling the drone to perform the next action based on the image features. As another example, the image processing system may include an image processing unit, which is located in the drone and processes the image captured by the drone in real time, extracts features from the image, and analyzes the features.

[0032] In step S101, the multiple sampling scales can be different sampling scales, that is, in the embodiment of the present application, different sampling scales are used in sequence to sample the image to be processed, so that when the scale of the image changes, the accuracy of feature matching will not be reduced, providing a basis for the accuracy of subsequent image registration.

[0033] In addition, in step S101, the first image includes multiple grids, wherein the multiple grids can be grids of the same size or grids of different sizes. In practical applications, the size can be set according to needs. In the embodiment of the present application, the multiple grids are grids of the same size as an example for explanation.

[0034] Step S102 : for each grid, determining a filtering threshold according to the pixel values of the pixels in each grid.

[0035] In step S102, the filter threshold corresponding to each grid can be determined by the pixel values of the pixels within the corresponding grid. Since the pixel values of the pixels in different grids are different, the filter threshold corresponding to each grid is also different. In the embodiment of the present application, calculating the corresponding filter threshold for each grid can improve the accuracy of grid feature detection, thereby improving the robustness of image feature extraction.

[0036] Step S103 : performing feature detection on the pixel points in the corresponding grid based on the filtering threshold to obtain a plurality of feature points.

[0037] In step S103, for each grid, the corresponding filtering threshold is used to perform feature detection on the corresponding grid. For example, the pixel points in the grid whose corresponding pixel values are greater than the filtering threshold can be used as feature points, thereby improving the accuracy of grid feature detection and further improving the robustness of image feature extraction.

[0038] Step S104 , performing area coverage suppression processing on the multiple feature points, eliminating redundant feature points from the multiple feature points, and obtaining target feature points.

[0039] In step S104, in order to homogenize the feature points in the image, in an embodiment of the present application, a method of accelerated adaptive non-maximum suppression based on area coverage suppression adopts square approximation within the search range to suppress redundant feature points, so as to eliminate redundant feature points, thereby reducing the computational complexity of adaptive non-maximum suppression and improving the robustness of image feature extraction.

[0040] Step S105 , performing feature description on the target feature point to obtain a feature vector corresponding to the target feature point.

[0041] After determining the target feature points through steps S101 to S104, it is necessary to determine the feature vectors corresponding to the target feature points. In the embodiment of the present application, TEBLID (Triplet-based Efficient Binary Local Image Descriptor) is used to implement vector descriptions of target feature points. It can improve the image matching effect through techniques such as ternary loss, difficult negative sample mining, and anchor point exchange. TEBLID is an improvement over BEBLID, which can provide higher matching accuracy and compactness while maintaining high real-time performance.

[0042] Based on the scheme defined by the above steps S101 to S105, it can be known that in the embodiment of the present application, multiple sampling scales are used to sample the image to extract feature points in the image. While improving the real-time performance of image feature extraction, the scale invariance of image feature extraction is also improved, thereby improving robustness. In addition, in the embodiment of the present application, the filtering threshold is determined based on the pixel value of the pixel point in the grid, that is, in the embodiment of the present application, for each grid, it corresponds to a different filtering threshold, and for different grids, the use of corresponding filtering thresholds can improve the accuracy of feature point detection, thereby improving the accuracy and robustness of image feature extraction. In order to further homogenize the feature points in the image, the influence of redundant feature points on the accurate analysis of image features will be eliminated. In the embodiment of the present application, the redundant feature points are eliminated by regional coverage suppression to reduce the computational complexity of adaptive non-maximum suppression, thereby improving the real-time and robustness of image feature extraction.

[0043] It can be seen that the solution provided by the embodiment of the present application can take into account both the real-time and robustness of image feature extraction.

[0044] The following introduces the implementation process of the method provided in the embodiment of the present application.

[0045] In some embodiments, the image processing system may use an image pyramid to sample the image to be processed. Specifically, the image processing system sequentially downsamples the image to be processed using different sampling scales within the image pyramid to obtain multiple image pyramid layers. The multiple image pyramid layers are then fused to obtain a second image. The second image is then divided into multiple grids to obtain the first image.

[0046] In the above embodiment, the image pyramid has multiple image pyramid layers with different sampling scales, the multiple image pyramid layers including multiple first pyramid layers and multiple second pyramid layers, with the second pyramid layers being arranged between two adjacent first pyramid layers. The image processing system downsamples the image to be processed at a first magnification to obtain the first pyramid layer, and downsamples the image to be processed at a second magnification to obtain the second pyramid layer, where the first magnification is greater than the second magnification.

[0047] It should be noted that, due to the poor scale invariance of the original ORB algorithm, when the ORB algorithm is used for feature matching, if there is a scale change in the image, the accuracy of feature matching will be reduced, thereby affecting the accuracy of subsequent image registration. The classic SIFT (Scale-Invariant Feature Transform) algorithm constructs a Gaussian scale space to detect feature points, and the number of detected feature points is large. Although the algorithm has improved the accuracy of feature matching, its scale space structure is complex, the calculation process is long, and the real-time performance is not high. To solve the above problems, in an embodiment of the present application, a method combining FAST feature detection and construction of image scale space is adopted to improve the scale invariance of the algorithm while ensuring the high real-time performance of the ORB algorithm. As an example, in an embodiment of the present application, the image pyramid can be a Gaussian pyramid, wherein the pyramid scale space corresponding to the Gaussian pyramid can be as shown in Table 1:

[0048] Table 1

[0049]

[0050]

[0051] In order to obtain stable key points at different scales, in the embodiment of the present application, the FAST algorithm is used to solve the key points in different scale spaces. The image pyramid layers at different scales are composed of n octaves layers c i (i.e. the first pyramid layer) and n intra-octave layers d i(i.e. the second pyramid layer), i = {0, 1, ..., n-1}, where n is 4. Each octave layer is obtained by continuously downsampling the original image (i.e. the image to be processed) by a factor of two (i.e. the first magnification). Each intra-octave layer d i In c i Layer and c i +1 layer. The first intra-octaves layer d0 is obtained by downsampling the original image c0 by 1.5 times, and the other intra-octaves layers are obtained by continuously downsampling d0 by a factor of two. Therefore, t represents the scale, then t(c i )=c0 / 2 i , t(d i )=d0 / 2 i .

[0052] After downsampling the image to be processed using the above-mentioned multiple sampling scales, image fusion is performed on the multiple image pyramid layers to obtain a second image. Next, image segmentation is performed on the second image to obtain a first image having multiple grids.

[0053] It should be noted that in the traditional ORB algorithm, a manually set global empirical threshold is used when extracting feature points, and the specific situation of the local neighborhood of the pixel point is not taken into account. Therefore, the feature points detected by the ORB algorithm are concentrated in areas where the image changes significantly, and there are a large number of redundant feature points. For uniform image areas, due to the similarity of the grayscale features of the images, the FAST algorithm can only extract fewer feature points, resulting in uniform feature information in the overlapping areas of the two images to be registered, but no feature points are extracted, which leads to the failure of image registration. In order to solve the problem of excessive aggregation of ORB feature points, and in order to detect more abundant feature points, in an embodiment of the present application, the image processing system segments the image to be processed, for example, the image to be processed with a size of X·Y is evenly divided into n·n rectangular grid images, and the grid size is set to x·y, where x=X / n, y=Y / n.

[0054] Furthermore, after completing the sampling and image segmentation of the image to be processed, the image processing system determines the filtering threshold corresponding to each grid to perform feature detection on the corresponding grid.

[0055] In an embodiment of the present application, the filtering threshold includes a first filtering threshold or a second filtering threshold. The image processing system can determine the pixel average value of the corresponding grid based on the number of pixels in each grid and the pixel value of each pixel; and determine the first filtering threshold based on the number of pixels, the pixel value of each pixel and the pixel average value; then, the ratio of the first filtering threshold to the target value is calculated to obtain the second filtering threshold.

[0056] In the above embodiment, the target value is a positive integer, for example, 2; the first filtering threshold is greater than the second filtering threshold, for example, the first filtering threshold is p1, the second filtering threshold is p2, then p2=p1 / 2.

[0057] It should be noted that the traditional FAST feature point extraction is a global threshold set by manual experience, without considering the specific situation of the local neighborhood of the pixel point. Therefore, the feature points detected by the ORB algorithm are concentrated in the area where the image changes significantly, and there are a large number of redundant feature points. For uniform image areas, due to the similarity of the grayscale features of the image, the FAST algorithm can only extract fewer feature points. In response to the above problems, in an embodiment of the present application, the image processing system increases the number of detected feature points by setting different filtering thresholds for each grid. Among them, the filtering thresholds p1 and p2 in each grid can be expressed by formula (1) and formula (2) respectively:

[0058]

[0059] p2=p1 / 2 (2)

[0060] In formula (1), k is the number of pixels in the grid, I(x i ) is the pixel x in the grid i The pixel value (e.g., grayscale value), is the average value of the pixels in the grid.

[0061] During the feature detection process, for each grid, the image processing system determines the pixel points in each grid whose pixel values are greater than the first filtering threshold as feature points, and then counts the number of feature points; and when the number of feature points is less than the preset number threshold, the image processing system filters out the pixel points in each grid whose pixel values are greater than the second filtering threshold to obtain feature points.

[0062] That is, in the embodiment of the present application, formula (1) can be used to solve different pixel thresholds based on the pixel information in different grids, thereby obtaining a larger number of feature points. If no feature points are extracted in a grid, the threshold is lowered to p2, and feature point detection is continued within the grid until all grids in the image are traversed, completing the adaptive extraction of FAST feature points.

[0063] Furthermore, in order to further homogenize the feature points in the image and eliminate redundant feature points among multiple feature points, in an embodiment of the present application, the image processing system uses the square approximation of the search range to suppress irrelevant points to reduce the computational complexity of adaptive non-maximum suppression. At the same time, a new strategy of initializing the search range based on the image dimension is also adopted to further improve the convergence speed of the algorithm.

[0064] Specifically, for each grid, the image processing system determines the subgrid size of each subgrid based on the grid size and the number of feature points within the corresponding grid. Each grid is then divided into multiple subgrids based on the subgrid size, and the response values of the feature points within each subgrid are obtained. Redundant feature points within each subgrid whose response values are below the response threshold are eliminated to obtain the target feature points. Adjacent subgrids are separated by a preset number of pixels.

[0065] As an example, after extracting the feature points, in order to reduce the clustering of feature points in the grid, m feature points with uniform distribution are obtained from each grid. In the embodiment of the present application, the image processing system accelerates the adaptive non-maximum suppression strategy with an edge of 2s. h The area of ​​covers the rectangular grid image (i.e., sub-grid), and the distance between the area centers is at least s h +1, as Figure 2 The area coverage suppression diagram is shown.

[0066] Given s h and the grid width x, we can get the maximum number of squares in a row of the image, and define the row as a set of subgrids at the same height as the first and last subgrids in the image, and a row that is completely aligned with the grid boundary. w points (i.e., subgrid centers), then the distance between points in each row is k w -1. In addition, the left and right edge points are located at a distance s from the left and right boundaries of the rectangular grid. h . Then the grid width x can be expressed by formula (3):

[0067] x=2s h +(s h +1)(k w -1) (3)

[0068] Therefore, according to formula (4), the number of points in each row is:

[0069]

[0070] Similarly, the maximum number of square centers k that can be fitted within the grid height y is l It can be shown by formula (5):

[0071] y=2s h +(s h +1)(k l -1) (5)

[0072] The number of points after screening is m = k w and k l By integrating formulas (3)-(5), we can obtain formula (6):

[0073] (m-1)s h 2 +s h (x+2m+y)+m+x+xy=0 (6)

[0074] Solve formula (6) and get s h As shown in formula (7):

[0075]

[0076] In formula (7), the discriminant Δ of the quadratic equation can be expressed by formula (8):

[0077] Δ=4x+4m+4ym+x 2 -2xy+4xym (8)

[0078] The following strategies can be used to quickly remove redundant feature points:

[0079] If m / k w Unable to obtain integer values, the smallest unit based on the image is 1 pixel, and the approximate value k can be obtained by rounding l The lower bound s of the binary search is determined by observing the worst possible point distribution l If n feature points are all clustered in a single grid square and there is no space between feature point sets, a smaller side length of 2s can be used. l The area is subjected to secondary coverage suppression processing to retrieve at least m feature points, that is, m(2s l ) 2 =n. Therefore, s l It can be expressed by formula (9):

[0080]

[0081] A feature point screening method based on area coverage and accelerated non-maximum suppression is used to obtain feature points with uniform distribution and superior performance, so as to improve the matching accuracy of the ORB algorithm.

[0082] Furthermore, in order to improve the accuracy of image registration, in an embodiment of the present application, after removing redundant feature points, the image processing system also uses a sub-pixel interpolation method to accurately locate the coordinates and scale information of the feature points.

[0083] In some embodiments, the image processing system obtains the feature point position of the target feature point in the first image, the scale space where the target feature point is located, and the image layer corresponding to the target feature point; then, a two-dimensional quadratic function is constructed based on the feature point position and the image layer, and the two-dimensional quadratic function is solved to determine the extreme point position corresponding to the target feature point; one-dimensional interpolation is performed at the extreme point position in the scale direction corresponding to the scale space to obtain the interpolated target feature point.

[0084] In one example, Figure 3 Taking the schematic diagram of the principle of sub-pixel interpolation as an example, the feature points are detected layer by layer in the scale space of the image pyramid, and the image layer where the extreme point corresponding to the feature point is located is preliminarily determined to be the c of octaves. i layer, and determine the coordinate position of the extreme point neighborhood. Then, with the extreme point as the center, in the c of the current octaves i layer and its upper and lower adjacent image layers (such as Figure 3 d in i Layer and d i-1 In the image layer), the FAST score value (t is the range) of the extreme point and its surrounding neighborhood (for example, a 3×3 area) is selected. The score value is fitted by a two-dimensional quadratic function to calculate the sub-pixel precise coordinates of the extreme point in the image plane (i.e., x and y directions) to eliminate the positioning error caused by discrete sampling. Then, in the scale direction of the image pyramid, based on the score values of the image layer where the extreme point is located and its adjacent image layers, one-dimensional quadratic or cubic interpolation is performed to determine the optimal scale corresponding to the extreme point (i.e., the exact number of layers in the image pyramid). Finally, the sub-pixel coordinates in the spatial domain and the optimal scale in the scale domain are combined to obtain the final precise position (x, y, σ) of the feature point, where σ is the scale parameter.

[0085] It should be noted that by refining the extreme points in the image plane (x, y) and scale direction (σ) respectively through quadratic interpolation, the sub-pixel accuracy and scale accuracy of feature point positioning can be improved.

[0086] After determining the target feature points, the image processing system performs feature description on the target feature points to determine feature vectors corresponding to the target feature points.

[0087] Specifically, for each feature point, the image processing system obtains the grid group corresponding to each feature point; calculates the pixel mean corresponding to each target grid; and compares the pixel means of two adjacent target grids to obtain a comparison result; then, based on the comparison result, determines the feature vector of the target feature point. A grid group includes multiple target grids corresponding to a feature point.

[0088] It should be noted that TEBLID is an efficient binary local image descriptor algorithm that aims to improve the image matching effect through technologies such as triplet loss, difficult negative sample mining, and anchor point exchange. It is an improvement on BEBLID and can provide higher matching accuracy and compactness while maintaining high real-time performance.

[0089] For each detected feature point, TEBLID calculates the local image descriptor (i.e., feature vector) around it. In the embodiment of the present application, the feature vector of the feature point is calculated based on the grayscale values of different grids.

[0090] Specifically, the image processing system first calculates the integral image. int (x, y) is the cumulative sum image of the original image (i.e., the image to be processed) I(x, y), which is used to quickly calculate the pixel sum of any grid. The calculation formula of the integral image can be expressed as formula (10):

[0091]

[0092] Next, the image processing system calculates the box average difference. The descriptor consists of multiple box average differences. Each box represents the average grayscale value of a grid. Let R(p,s) represent a rectangular area with p as the center and side length s. Its average grayscale value can be expressed by formula (11):

[0093]

[0094] Using the integral image, the sum of pixels in a rectangular area can be quickly calculated to obtain the average grayscale value.

[0095] Then, for each feature point, the image processing system calculates the average difference of multiple boxes to form a binary descriptor, i.e., a feature vector. In the embodiment of the present application, the feature vector is a set of vectors composed of binary bits, and each feature bit in the feature vector represents the comparison result of the average grayscale values of two boxes. For example, the feature vector is set to include K feature bits, that is, K pairs of box areas are randomly selected around the feature point, the grayscale mean of each pair of boxes is calculated and compared, and a binary bit sequence is generated, which is the feature vector. The value corresponding to the kth feature bit can be determined by formula (12):

[0096]

[0097] In formula (12), and are the center points of the kth pair of boxes, and are the side lengths of the kth pair of boxes respectively.

[0098] In order to improve the discriminative ability of descriptors, TEBLID uses a triplet loss function for training. The goal of the triplet loss function is to make similar feature points have similar descriptors, and different feature points have different descriptors. In the embodiment of the present application, the loss function can be expressed by formula (13):

[0099]

[0100] In formula (13), d is the distance between descriptors (i.e., feature vectors); f(x) is the descriptor of the feature point; x i is a feature point; is a positive sample, belonging to the same category as the anchor point; is a negative sample that belongs to a different category from the anchor point; α is a hyperparameter used to control the distance difference between positive and negative samples.

[0101] The descriptor parameters are updated through the back-propagation algorithm to minimize the loss function value. The specific steps are as follows:

[0102] (1) Forward propagation: Calculate the loss value of the current descriptor.

[0103] (2) Back propagation: Calculate the gradient of the loss function with respect to the descriptor parameters.

[0104] (3) Parameter update: Update the descriptor parameters according to the gradient to reduce the loss value.

[0105] TEBLID significantly improves the matching accuracy and computational efficiency of image descriptors by introducing the triplet loss function and difficult negative sample mining technology. While maintaining high efficiency, it provides higher discrimination ability and compactness, and is a very efficient image descriptor algorithm.

[0106] In some embodiments, Figure 4 The overall flow chart of the method provided in the embodiment of the present application is shown in FIG. Figure 4 As shown, the method includes the following steps S401 to S408:

[0107] Step S401: construct an image pyramid and perform downsampling processing on the image based on the image pyramid.

[0108] Step S402 , gridding the downsampled image to obtain a plurality of grids.

[0109] Step S403: Perform local dynamic threshold FAST feature detection on each grid to obtain a detection result.

[0110] Step S404: If a feature point is detected, execute step S406; otherwise, execute step S405.

[0111] Step S405: lower the threshold and perform FAST feature detection again.

[0112] Step S406: performing area coverage suppression processing on the feature points to remove redundant feature points.

[0113] Step S407 , performing sub-pixel interpolation processing on the target feature points obtained after screening to obtain interpolated target feature points.

[0114] Step S408 : performing TEBLID feature description on the interpolated target feature points to obtain feature vectors of the interpolated target feature points.

[0115] This concludes the introduction to the methods provided in the embodiments of the present application.

[0116] With the rapid development of drone technology, image feature extraction technology has been widely used in the field of drone remote sensing. However, the convolutional neural network in the traditional deep learning scheme can effectively extract local features such as edges and textures in the image. The deep learning model it constructs is highly robust to noise and deformation and can maintain high performance in complex environments. However, the calculation process is relatively complex and the learning cost is high. For example, a high-performance GPU (Graphics Processing Unit) is required, and the real-time performance is relatively insufficient. The traditional ORB algorithm has strong real-time performance but insufficient robustness. Researchers have made improvements to the algorithm, but all of them are based on sacrificing some real-time performance or robustness.

[0117] Considering the complex environment and real-time processing scenarios encountered by low-altitude drones, this embodiment first constructs a Gaussian pyramid model to improve the algorithm's scale invariance. A grid-based approach, combined with the FAST method and regional acceleration suppression, is then used to rapidly detect a large number of evenly distributed feature points with complete feature information. Finally, the TEBLID algorithm is introduced for feature description, significantly reducing feature description time while improving the algorithm's overall real-time performance.

[0118] In the process of feature point detection, a pyramid model is constructed to solve the problem of poor scale invariance of the traditional ORB algorithm. After the image segmentation strategy is adopted and a large number of feature points are detected using adaptive thresholds, the regional accelerated coverage suppression method is introduced to remove redundant feature points to solve the defects of the traditional ORB algorithm with a small number of feature points and concentrated distribution; in the feature description process, the TEBLID descriptor with stronger robustness and higher computational efficiency is used to describe the feature points, which further improves the real-time performance of the algorithm on the basis of ensuring the robustness of the algorithm.

[0119] The present application also provides an image feature extraction device, such as Figure 5As shown, the device 500 includes: a sampling module 501, a threshold determination module 502, a feature detection module 503, a feature filtering module 504 and a feature description module 505.

[0120] A sampling module 501 is configured to sample an image to be processed using an image pyramid having multiple scales to obtain a first image, where the first image includes multiple grids;

[0121] A threshold determination module 502 is configured to determine a filtering threshold for each grid according to the pixel values of the pixels in each grid;

[0122] A feature detection module 503 is configured to perform feature detection on pixel points in a corresponding grid based on a filtering threshold to obtain a plurality of feature points;

[0123] The feature filtering module 504 is used to perform regional coverage suppression processing on multiple feature points, eliminate redundant feature points from the multiple feature points, and obtain target feature points;

[0124] The feature description module 505 is used to perform feature description on the target feature point to obtain a feature vector corresponding to the target feature point.

[0125] In some embodiments, the sampling module includes: a downsampling module, an image fusion module, and a grid division module. The downsampling module is configured to sequentially downsample the image to be processed using different sampling scales in the image pyramid to obtain multiple image pyramid layers; the image fusion module is configured to fuse the multiple image pyramid layers to obtain a second image; and the grid division module is configured to divide the second image into multiple grids to obtain the first image.

[0126] In some embodiments, an image pyramid has multiple image pyramid layers with different sampling scales, where the multiple image pyramid layers include multiple first pyramid layers and multiple second pyramid layers, with the second pyramid layer being disposed between two adjacent first pyramid layers. The downsampling module is specifically configured to perform downsampling processing on the first pyramid layer at a first magnification to obtain the first pyramid layer, and to perform downsampling processing on the second pyramid layer at a second magnification to obtain the second pyramid layer, where the first magnification is greater than the second magnification.

[0127] In some embodiments, the filtering threshold includes a first filtering threshold or a second filtering threshold, and the threshold determination module is specifically used to determine the pixel average value of the corresponding grid based on the number of pixels in each grid and the pixel value of each pixel; determine the first filtering threshold based on the number of pixels, the pixel value of each pixel and the pixel average value; calculate the ratio of the first filtering threshold to the target value to obtain the second filtering threshold, wherein the target value is a positive integer and the first filtering threshold is greater than the second filtering threshold.

[0128] In some embodiments, the feature detection module is specifically used to determine, for each grid, pixel points in each grid whose pixel values are greater than a first filtering threshold as feature points; count the number of feature points; and when the number of feature points is less than a preset number threshold, filter out pixel points in each grid whose pixel values are greater than a second filtering threshold to obtain feature points.

[0129] In some embodiments, the feature filtering module is specifically used to determine the subgrid size of the subgrid for each grid based on the grid size of each grid and the number of feature points in the corresponding grid; divide each grid into multiple subgrids based on the subgrid size, wherein a preset pixel interval is between two adjacent subgrids; obtain the response value of the feature point in each subgrid; eliminate redundant feature points in each subgrid whose response value is lower than the response threshold to obtain the target feature point.

[0130] In some embodiments, the feature description module is specifically used to obtain the grid group corresponding to each feature point for each feature point, wherein the grid group includes multiple target grids corresponding to the feature point; calculate the pixel mean corresponding to each target grid; compare the pixel means of two adjacent target grids to obtain a comparison result; and determine the feature vector of the target feature point based on the comparison result.

[0131] In some embodiments, the image feature extraction device also includes: an interpolation module, which is used to perform regional coverage suppression processing on multiple feature points, eliminate redundant feature points among the multiple feature points, and obtain the target feature point, and then obtain the feature point position of the target feature point in the first image, the scale space where the target feature point is located, and the image layer corresponding to the target feature point; construct a two-dimensional quadratic function based on the feature point position and the image layer; solve the two-dimensional quadratic function to determine the extreme point position corresponding to the target feature point; perform one-dimensional interpolation at the extreme point position in the scale direction corresponding to the scale space to obtain the interpolated target feature point.

[0132] The image feature extraction device provided in the embodiment of the present application can implement each process implemented in the aforementioned method embodiment. To avoid repetition, it will not be described here.

[0133] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0134] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0135] The electronic device may include a processor 601 and a memory 602 storing computer program instructions.

[0136] Specifically, the processor 601 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0137] The memory 602 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 602 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 602 may include removable or non-removable (or fixed) media. Where appropriate, the memory 602 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 602 is a non-volatile solid-state memory.

[0138] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0139] The processor 601 reads and executes computer program instructions stored in the memory 602 to implement any one of the image feature extraction methods in the above embodiments.

[0140] In one example, the electronic device may further include a communication interface 603 and a bus 610. Figure 6 As shown, the processor 601, the memory 602, and the communication interface 603 are connected via a bus 610 and communicate with each other.

[0141] The communication interface 603 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0142] Bus 610 comprises hardware, software or both, couples the parts of electronic equipment to each other.For example, and not limitation, bus can comprise accelerated graphics port (AGP) or other graphics bus, enhanced industry standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations.In suitable cases, bus 610 can comprise one or more buses.Although the present application embodiment describes and shows specific bus, the application considers any suitable bus or interconnection.

[0143] In addition, in conjunction with the image feature extraction methods in the above embodiments, embodiments of the present application may provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any of the image feature extraction methods in the above embodiments is implemented.

[0144] In addition, in conjunction with the image feature extraction method in the above embodiments, the present application can provide a computer program product for implementation. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes and implements any of the image feature extraction methods in the above embodiments.

[0145] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0146] The functional modules shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in unit, a function card or the like. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0147] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0148] Various aspects of the present disclosure have been described above with reference to flowcharts and / or block diagrams of methods, devices, storage media, and electronic devices for extracting image features according to embodiments of the present disclosure. It should be understood that each block in the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine such that execution of these instructions by the processor of the computer or other programmable data processing device enables the implementation of the functions / actions specified in one or more blocks in the flowcharts and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It should also be understood that each block in the block diagrams and / or flowcharts, as well as combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware that performs the specified functions or actions, or by a combination of dedicated hardware and computer instructions.

[0149] The above description is only a specific embodiment of the present application. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the scope of protection of the present application.

Claims

1. A method for extracting image features, characterized in that: include: Sampling the image to be processed using multiple sampling scales to obtain a first image, where the first image includes multiple grids; For each grid, determining a filtering threshold according to pixel values of pixels within each grid; Performing feature detection on pixel points in the corresponding grid based on the filtering threshold to obtain multiple feature points; Performing area coverage suppression processing on the plurality of feature points, eliminating redundant feature points from the plurality of feature points, and obtaining target feature points; Perform feature description on the target feature point to obtain a feature vector corresponding to the target feature point.

2. The method according to claim 1, characterized in that The step of sampling the image to be processed using multiple sampling scales to obtain a first image includes: In the image pyramid, downsampling the image to be processed is performed in sequence using different sampling scales to obtain multiple image pyramid layers; performing image fusion on the multiple image pyramid layers to obtain a second image; The second image is divided into a plurality of grids to obtain the first image.

3. The method according to claim 2, characterized in that The image pyramid has a plurality of image pyramid layers with different sampling scales, wherein the plurality of image pyramid layers include a plurality of first pyramid layers and a plurality of second pyramid layers, wherein the second pyramid layers are arranged between two adjacent first pyramid layers; and the method of sequentially downsampling the image to be processed using different sampling scales to obtain the plurality of image pyramid layers includes: For the first pyramid layer, performing downsampling processing on the image to be processed at a first magnification to obtain the first pyramid layer; For the second pyramid level, downsampling processing is performed on the first pyramid level at a second magnification to obtain the second pyramid level, wherein the first magnification is greater than the second magnification.

4. The method according to claim 1, wherein The filtering threshold includes a first filtering threshold or a second filtering threshold, and determining the filtering threshold for each grid according to the pixel values of the pixels in each grid includes: Determine the pixel average value of the corresponding grid according to the number of pixels in each grid and the pixel value of each pixel; Determining the first filtering threshold according to the number of pixels, the pixel value of each pixel, and the pixel average; The ratio of the first filtering threshold to a target value is calculated to obtain the second filtering threshold, wherein the target value is a positive integer and the first filtering threshold is greater than the second filtering threshold.

5. The method according to claim 4, characterized in that The feature detection is performed on the pixel points in the corresponding grid based on the filtering threshold to obtain multiple feature points, including: For each of the grids, determining pixel points within the grid whose pixel values are greater than the first filtering threshold as the feature points; Counting the number of feature points of the feature points; When the number of the feature points is less than a preset threshold, pixel points having pixel values greater than the second filtering threshold are screened out from each grid to obtain the feature points.

6. The method according to claim 1, characterized in that The performing area coverage suppression processing on the plurality of feature points, eliminating redundant feature points from the plurality of feature points, and obtaining target feature points includes: For each of the grids, determining a subgrid size of the subgrid according to the grid size of each grid and the number of feature points in the corresponding grid; Dividing each grid into a plurality of subgrids based on the subgrid size, wherein a preset pixel is spaced between two adjacent subgrids; Obtaining the response value of each feature point in the sub-grid; The redundant feature points in each subgrid whose response value is lower than the response threshold are eliminated to obtain the target feature points.

7. The method according to claim 1, characterized in that The step of performing feature description on the target feature point to obtain a feature vector corresponding to the target feature point includes: For each of the feature points, obtaining a grid group corresponding to each of the feature points, wherein the grid group includes a plurality of target grids corresponding to the feature point; Calculate the pixel mean corresponding to each target grid; Compare the pixel means of two adjacent target grids to obtain the comparison result; Determine a feature vector of the target feature point according to the comparison result.

8. The method according to any one of claims 2 to 7, characterized in that After performing area coverage suppression processing on the plurality of feature points and eliminating redundant feature points from the plurality of feature points to obtain target feature points, the method further includes: Acquire a feature point position of the target feature point in the first image, a scale space where the target feature point is located, and an image layer corresponding to the target feature point; Constructing a two-dimensional quadratic function according to the feature point positions and the image layer; Solving the two-dimensional quadratic function to determine the extreme point position corresponding to the target feature point; One-dimensional interpolation is performed at the extreme point position in the scale direction corresponding to the scale space to obtain the interpolated target feature point.

9. An image feature extraction device, characterized in that: include: a sampling module, configured to sample the image to be processed using an image pyramid having multiple scales to obtain a first image, wherein the first image includes multiple grids; A threshold determination module, configured to determine a filtering threshold for each grid according to pixel values of pixels within each grid; A feature detection module, configured to perform feature detection on pixel points in the corresponding grid based on the filtering threshold to obtain a plurality of feature points; A feature filtering module is used to perform area coverage suppression processing on the multiple feature points, eliminate redundant feature points in the multiple feature points, and obtain target feature points; The feature description module is used to perform feature description on the target feature point to obtain a feature vector corresponding to the target feature point.

10. An electronic device, characterized in that: The electronic device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the image feature extraction method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the method for extracting image features according to any one of claims 1 to 8 is implemented.

Images