Image feature matching method based on GPU, electronic equipment and storage medium

Abstract

The invention provides an image feature matching method based on a GPU, electronic equipment and a storage medium, and relates to the technical field of image processing. A mask image (in which feature points and response function values are marked) is loaded in a calculation shader of a GPU, and a four-dimensional random vector is generated. And determining first N feature point pixels with the maximum response function value in the mask image in a calculation shader of the GPU. Loading the color image in a calculation shader of the GPU, and determining feature vectors of the first N feature points with the maximum response function values in the color image according to the four-dimensional random constant; and calculating a feature distance based on the pairwise feature vectors, and carrying out feature point matching according to the feature distance. According to the method, the image resources are directly accessed in the graph rendering pipeline for feature matching, the operation of copying data through a memory between the GPU and the CPU is avoided, the power consumption is reduced, and meanwhile, the high-concurrency multi-thread processing capacity provided by the GPU is utilized, so that the calculation efficiency of feature point matching is greatly improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a GPU-based image feature matching method, electronic device, and storage medium. Background Art

[0002] At present, in the fields of image processing and machine vision, as well as graphics rendering-related applications such as autonomous driving, VR / AR, etc., computing efficiency and computing accuracy have always been the main goals of the industry. Among them, feature point-based image recognition, matching, and stereo vision have attracted much attention due to their wide range of application scenarios.

[0003] Since the feature point matching algorithm currently used requires a large number of pixel calculations, and the higher the image resolution, the more pixels need to be calculated, so the computing resources and computing time consumed will be greater. Therefore, how to achieve feature point matching of rendered images efficiently and with low power consumption is an urgent problem to be solved. Summary of the invention

[0004] The present application provides a GPU-based image feature matching method, electronic device and storage medium, which solves the problem of large computing resources and long computing time required for image feature point matching.

[0005] In a first aspect, the present application provides a GPU-based image feature matching method, the method comprising:

[0006] Acquire feature vectors corresponding to feature points of a color image; wherein one feature point corresponds to one feature vector; when the color image is a first color image, N feature points of the first color image correspond to N feature vectors; when the color image is a second color image, N feature points of the second color image correspond to N feature vectors;

[0007] Determine N feature distances between each feature point of the first color image and the N feature points of the second color image, respectively, according to the N feature vectors corresponding to the N feature points of the first color image and the N feature vectors corresponding to the N feature points of the second color image;

[0008] If the minimum feature distance among the N feature distances is less than a preset distance threshold, the feature points to which the two feature vectors of the minimum feature distance belong are determined as a matching feature point pair;

[0009] The step of obtaining a feature vector corresponding to a feature point of a color image includes:

[0010] A mask image is loaded in a first computing shader of the GPU, and M four-dimensional random constants are generated according to image coordinate data of feature point pixels in the mask image; the mask image is an image obtained by performing feature point recognition on the color image, and the mask image marks a plurality of feature point pixels and corresponding response function values;

[0011] Loading the mask image in the second computing shader of the GPU, sorting a plurality of feature point pixels marked in the mask image according to the magnitude of the response function value, and determining the first N feature point pixels with the largest response function values ​​in the mask image;

[0012] The color image is loaded into the third compute shader of the GPU, and the brightness difference of adjacent pixels of the feature point pixels in the color image is determined according to the M four-dimensional random constants, and the feature vectors of the first N feature points with the largest response function values ​​are determined according to the brightness difference of adjacent pixels of the feature point pixels in the color image, as the N feature vectors corresponding to the N feature points of the color image.

[0013] Through the GPU-based image feature matching method provided by the embodiment of the present application, a mask image (feature points and response function values ​​are marked in the mask image) is loaded in the compute shader of the GPU to generate a four-dimensional random vector. The mask image is loaded in the compute shader of the GPU and the feature point pixels with the largest response function values ​​in the mask image are determined. The color image is loaded in the compute shader of the GPU and the feature vectors of the feature points with the largest response function values ​​in the color image are determined based on the four-dimensional random constants. The feature distance is calculated based on the pairwise feature vectors and feature point matching is performed based on the feature distance. The present application can avoid the operation of copying data through memory between the GPU and the CPU by directly accessing the image resources inside the graphics rendering pipeline for feature matching, greatly reducing the heating of mobile devices caused by memory bandwidth overhead, and at the same time utilizing the high concurrency multi-threaded processing capability provided by the GPU, which can greatly accelerate the computational efficiency of feature point matching and achieve the purpose of high frame rate application.

[0014] It should be noted that feature point recognition is first performed on the rendered image (or color image) generated in the rendering pipeline, and then image feature matching is performed based on the feature points of each of the two frames of image. After feature point recognition is performed on the color image, a mask image can be obtained. The mask image marks the image feature point pixels and the response function values ​​corresponding to the image feature point pixels. Among them, the first color image corresponds to the first mask image, and the second color image corresponds to the second mask image.

[0015] In some possible implementations, the above-mentioned generating M four-dimensional random constants based on the image coordinate data of the feature point pixels in the mask image includes: multiplying the position index value of the feature point pixels in the mask image by a specified random number scaling coefficient to obtain a random number seed; in the first compute shader of the GPU, for each thread in the local workgroup, using trigonometric functions and exponential functions to operate on the random number seed to generate a four-dimensional random constant, and adjusting the four-dimensional random constant according to an offset parameter to obtain an adjusted four-dimensional random constant; when the local workgroup includes 16×16 threads, 256 adjusted four-dimensional random constants are obtained, and M is equal to 256.

[0016] For example, in the step of calculating the random constant, the first mask image and the second mask image are loaded into the calculation shader of the GPU, and the size of the local workgroup of the specified calculation pipeline is 16×16, and the size of the global workgroup is (1,1,1). In other words, the global workgroup contains only one local workgroup, and 16×16 threads are used as a local workgroup. 16×16 threads are opened on the GPU, that is, 256 threads.

[0017] In an embodiment of the present application, a random number seed is generated in a compute shader. For example, by the following equation 11, the location index (gl_LocalInvocationIndex) value is multiplied by the scaling factor (scale) of the random number specified by the parameter to obtain the random number seed. Each thread in the work group uses a combination of trigonometric functions and exponential functions to operate on the random number seed r to generate a 4-dimensional random vector. Each 4-dimensional random vector is then adjusted according to an offset parameter (offset). Adjustments are made to all 256 4-dimensional random vectors to obtain 256 adjusted 4-dimensional random vectors, which constitute a random number queue.

[0018] In some possible implementations, each of the M four-dimensional random constants includes a first random offset and a second random offset. In this case, determining the brightness difference of adjacent pixels of the feature point pixel in the color image according to the M four-dimensional random constants includes: in the third computing shader of the GPU, for each thread of the local workgroup, determining the first pixel color value after the first random offset is offset relative to the first feature point pixel and the second pixel color value after the second random offset is offset; and calculating the first pixel brightness value according to the first pixel color value, and calculating the second pixel brightness value according to the second pixel color value; and calculating the brightness difference between the second pixel brightness value and the first pixel brightness value.

[0019] In some possible implementations, the feature vectors of the first N feature points with the largest response function values ​​are determined based on the brightness difference of adjacent pixels of the feature point pixels in the color image, including: if the brightness difference is greater than 0, the output result is 1; if the brightness difference is less than or equal to 0, the output result is 0; and a sequence of 0 or 1 is obtained based on the output result as the feature vector corresponding to the first feature point pixel.

[0020] In some possible implementations, when the local working group includes 16×16 threads, the feature vector corresponding to the first feature point pixel is a set of 256-bit sequences of 0 or 1.

[0021] In an embodiment of the present application, in the process of matching image feature points generated by graphics rendering, by directly accessing image resources inside the graphics rendering pipeline, data copy operations through memory between the GPU and the CPU can be avoided, thereby greatly reducing the heating of the mobile device caused by memory bandwidth overhead. At the same time, by utilizing the high-concurrency multi-threaded processing capabilities provided by the GPU and thread atomic operations to access the shared cache between cores, the computational efficiency of feature point matching can be greatly accelerated to achieve the purpose of high frame rate applications.

[0022] In some possible implementations, the method further includes: storing a feature vector of a feature point pixel of the first color image in segments according to S integer variables; storing a feature vector of a feature point pixel of the second color image in segments according to S integer variables; performing an XOR operation on the i-th integer variable of the segmented storage of the first color image and the i-th integer variable of the segmented storage of the second color image, where i is 1, ..., S; using the bitCount instruction of the GPU to count the number of 1s in the XOR operation result; and using the counted number of 1s as the feature distance between a feature point pixel of the first color image and a feature point pixel of the second color image.

[0023] For example, two feature point sequences respectively include N feature points, each feature point corresponds to a feature vector, and the feature vector corresponding to each feature point is a 256-bit sequence of 0s or 1s.

[0024] In some possible implementations, the int data type occupies 4 bytes in memory, i.e., 32 bits, so 8 int variables can be used to represent 256 bits of data. Specifically to the present application, since the feature vector is a set of 256-bit sequences of 0 or 1, the present application uses 8 int variables to save the value of the feature vector in segments. Each int variable stores a 32-bit sequence of 0 or 1. In other words, S can be 8. The feature vectors of the two feature point sequences are saved in segments using 8 int variables, and 8 XOR operations are performed in pairs, and 8 XOR results are obtained accordingly. Each XOR result is a 32-bit sequence of 0 or 1. The bitCount() instruction of the GPU can be used to count the number of 1s in each XOR result, and the number of 1s in the 8 XOR results is accumulated and summed, and the accumulated sum is used as the feature distance of the two feature vectors.

[0025] In an embodiment of the present application, a first feature point sequence is generated for one frame of image and a second feature point sequence is generated for another frame of image, and then a feature vector of each feature point in the first feature point sequence and a feature vector of each feature point in the second feature point sequence are calculated. Then, based on the feature vector of each feature point in the first feature point sequence and the feature vectors of all feature points in the second feature point sequence, the feature distances between each feature point are calculated. The minimum feature distance is compared with a preset distance threshold, and whether the feature points match is determined based on the comparison result to improve the accuracy of feature point matching.

[0026] In some possible implementations, the value of M is equal to the local workgroup size of the computing pipeline in the third computing shader of the GPU. For example, M is 256, and the local workgroup size of the computing pipeline in the third computing shader of the GPU is 16×16.

[0027] In some possible implementations, the size of the local workgroup of the computing pipeline in the third computing shader of the GPU is equal to the size of the local workgroup of the computing pipeline in the first computing shader of the GPU; and the size of the global workgroup of the computing pipeline in the third computing shader of the GPU is equal to the size of the global workgroup of the computing pipeline in the first computing shader of the GPU. For example, the size of the local workgroup of the computing pipeline is 16×16, and the size of the global workgroup of the computing pipeline is (1, 1, 1).

[0028] In some possible implementations, the product of the size of the local workgroup of the computing pipeline in the second computing shader of the GPU and the size of the global workgroup is equal to the size of the mask image. The size of the local workgroup of the computing pipeline in the second computing shader of the GPU is equal to the size of the local workgroup of the computing pipeline in the third computing shader of the GPU. For example, the size of the local workgroup of the computing pipeline is 16×16.

[0029] In some possible implementations, in the first compute shader of the GPU, the size of the local working group of the compute pipeline is 16×16, and the size of the global working group is (1, 1, 1).

[0030] In some possible implementations, in the second compute shader of the GPU, the size of the local working group of the compute pipeline is 16×16, the size of the global working group is (W / 16, H / 16, 1), and the size of the mask image is W×H pixels.

[0031] In some possible implementations, in the third compute shader of the GPU, the size of the local working group of the compute pipeline is 16×16, and the size of the global working group is (1, 1, 1).

[0032] In some possible implementations, after generating M four-dimensional random constants according to the image coordinate data of the pixels in the mask image, the method further includes: storing the M four-dimensional random constants in a first shader storage buffer (SSBO 1). In an embodiment of the present application, a readable and writable SSBO of size 256 and supporting ivec4 vectors is created. By using a shader storage buffer, it is helpful to utilize the high-concurrency multi-threaded processing capability provided by the GPU, and the thread atomic operation to access the shared cache between cores, which can greatly accelerate the computational efficiency of feature point matching and achieve the purpose of high frame rate application.

[0033] In some possible implementations, in the second computing shader of the GPU, the global workgroup includes W / 16×H / 16 local workgroups. In this case, the above-mentioned sorting of the multiple feature point pixels marked in the mask image according to the size of the response function value to determine the first N feature point pixels with the largest response function values ​​in the mask image includes: for W / 16×H / 16 local workgroups, using the atomic operation atomicMax of the GPU to determine the first N feature points with the largest response function values ​​in each local workgroup; the first N feature points with the largest response function values ​​are arranged from large to small according to the response function values; for W / 16×H / 16 local workgroups, determining the first N feature points with the largest response function values ​​in the global workgroup from N×W / 16×H / 16 feature points as the first N feature points with the largest response function values ​​in the mask image.

[0034] In some possible implementations, the above-mentioned determining, for W / 16×H / 16 local work groups, the first N feature points with the largest response function values ​​in the global work group from the N×W / 16×H / 16 feature points includes: for W / 16×H / 16 local work groups, storing the first N feature points with the largest response function values ​​in each local work group in sequence in the second shader storage buffer; determining the first N feature points with the largest response function values ​​in the global work group from the N×W / 16×H / 16 feature points stored in the second shader storage buffer.

[0035] In one implementation, the entire image is divided into multiple local pixel regions, and the N feature points with the largest response values ​​in each local pixel region are sorted and stored in multiple segments of the second shader storage buffer. All feature points in all local pixel regions stored in multiple segments of the first shader storage buffer are sorted, and the first N feature points with the largest response values ​​in the entire image are selected.

[0036] In another implementation, the entire image is divided into multiple local pixel regions, and the N feature points with the largest response values ​​in each local pixel region are sorted and stored in multiple segments of the first shader storage buffer, and then the feature points in each local pixel region are sorted from large to small according to the response value. A feature point with the local maximum response value in each local pixel region is selected, and then all the selected feature points with the local maximum response value are sorted, thereby selecting the first N feature points with the local maximum response value in the entire image.

[0037] In some possible implementations, for W / 16×H / 16 local work groups, the first N feature points with the largest response function values ​​in each local work group are stored in the second shader storage buffer in sequence, including: for W / 16×H / 16 local work groups, the first N feature points with the largest response function values ​​in each local work group are stored in a shared cache area between work groups according to the position index; and the feature points in the shared cache area are copied to the corresponding positions of the second shader storage buffer according to the position index.

[0038] In some possible implementations, the matched feature point pair includes a first feature point and a second feature point. In this case, after determining the feature points to which the two feature vectors of the minimum feature distance belong as the matched feature point pair, the method further includes: extracting the coordinate parameters of the first feature point from the mask image corresponding to the first color image; and extracting the coordinate parameters of the second feature point from the mask image corresponding to the second color image.

[0039] When the matching operation of the feature points in the two frames of images is completed, a pair of feature points representing the same object in different images can be output. For example, assuming that the feature point (x1, y1) in the first color image and the feature point (x2, y2) in the second color image are two feature points that are successfully matched, then the feature point (x1, y1) and the feature point (x1, y1) constitute a feature point pair. Accordingly, the feature point pair can be recorded as [(x1, y1), (x2, y2)].

[0040] In actual implementation, after matching the feature points of two frames of images, the matching feature point pairs between the two frames of images can be determined, and then the position information of the matching feature point pairs can be used for motion estimation, image stitching, background segmentation, etc.

[0041] This application proposes a technology for performing image feature matching in real time and with low power consumption on a mobile device through a GPU. By introducing the graphics rendering unit and graphics rendering pipeline of the GPU, especially introducing the GPU's unique functions such as compute shaders and inter-core shared caches and atomic operations of thread work groups, feature points of different images in the graphics rendering pipeline are matched, thereby greatly improving computing efficiency to reduce the time overhead of pixel-by-pixel calculations, while reducing the overhead of data copying and context switching between feature matching and the graphics rendering pipeline.

[0042] In a second aspect, the present application provides a GPU-based image feature matching device, the device comprising a unit for executing the method in the first aspect. The device may correspond to executing the method described in the first aspect, and the relevant description of the unit in the device can refer to the description of the first aspect, which will not be repeated here for the sake of brevity.

[0043] The method described in the first aspect above can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a processing module or unit, a display module or unit, etc.

[0044] In a third aspect, the present application provides an electronic device, comprising a processor, a computer program or instructions stored in the processor and a memory, wherein the processor is configured to execute the computer program or instructions so that the method in the first aspect is executed.

[0045] In a fourth aspect, the present application provides a computer-readable storage medium on which a computer program (also referred to as instructions or codes) for implementing the method in the first aspect is stored. For example, when the computer program is executed by a computer, the computer can perform the method in the first aspect.

[0046] In a fifth aspect, the present application provides a chip, including a processor. The processor is used to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation thereof. Optionally, the chip also includes a memory, and the memory is connected to the processor via a circuit or wire.

[0047] In a sixth aspect, the present application provides a chip system, including a processor. The processor is used to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation thereof. Optionally, the chip system also includes a memory, and the memory is connected to the processor via a circuit or wire.

[0048] In a seventh aspect, the present application provides a computer program product, which includes a computer program (also referred to as instructions or codes). When the computer program is executed by an electronic device, the electronic device implements the method in the first aspect.

[0049] It can be understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of the process of image feature recognition and matching in an embodiment of the present application;

[0051] Figure 2 A schematic diagram of coordinate system conversion in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of a global work group and a local work group in an embodiment of the present application;

[0053] Figure 4A A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0054] Figure 4B A schematic diagram of the structure of the CPU and GPU provided in the embodiment of the present application;

[0055] Figure 5 A schematic diagram of a software architecture of an electronic device provided in an embodiment of the present application;

[0056] Figure 6 A schematic diagram of an image pyramid provided in an embodiment of the present application;

[0057] Figure 7 A flowchart of image feature recognition based on GPU provided in an embodiment of the present application;

[0058] Figure 8 A schematic diagram of calculating corner points using an image pyramid provided in an embodiment of the present application;

[0059] Fig.9A A schematic diagram of the positional relationship between a certain pixel and adjacent pixels provided in an embodiment of the present application;

[0060] Fig. 9B A schematic diagram of the positional relationship between corner points and adjacent pixels provided in an embodiment of the present application;

[0061] Fig.10 A schematic diagram of obtaining multiple local maximum response values ​​for a mask image provided by an embodiment of the present application;

[0062] Fig.11 A schematic diagram of a mask image after feature point recognition provided by an embodiment of the present application;

[0063] Fig. 12A Schematic diagram of the process of matching feature points of two frames of images provided in the embodiment of the present application Figure 1 ;

[0064] Fig. 12B Schematic diagram of the process of matching feature points of two frames of images provided in the embodiment of the present application Figure 2 ;

[0065] Fig.13 A flowchart of a GPU-based image feature matching method provided in an embodiment of the present application;

[0066] Fig.14A A schematic diagram of specifying a global workgroup and a local workgroup of a computing pipeline when computing random variables through a computing shader in an embodiment of the present application;

[0067] Fig. 14B A schematic diagram of storing random variables when random variables are calculated by a computing shader in an embodiment of the present application;

[0068] Fig.15 A schematic diagram of specifying a global work group and a local work group of a computing pipeline when sorting feature points through a computing shader in an embodiment of the present application;

[0069] Fig.16 A schematic diagram of extracting feature points in units of local working groups when sorting feature points by a compute shader in an embodiment of the present application;

[0070] Fig.17 A schematic diagram of segmented storage of feature points of each local working group when feature points are sorted by a compute shader in an embodiment of the present application;

[0071] Fig.18A It is a schematic diagram of segmentally sorting the feature points of each local working group when sorting the feature points by a compute shader in an embodiment of the present application;

[0072] Fig.18B It is a schematic diagram of segmentally sorting the feature points of each local working group when sorting the feature points by a compute shader in an embodiment of the present application;

[0073] Fig.19 It is a schematic diagram of retaining N feature points with the largest response values ​​in the mask image after sorting the feature points by a computational shader in an embodiment of the present application;

[0074] Fig. 20 A schematic diagram of respectively calculating feature vectors for N feature points with the largest response values ​​in a mask image in an embodiment of the present application;

[0075] Fig.21 A schematic diagram of calculating a feature vector according to a brightness difference of a feature point offset in an embodiment of the present application;

[0076] Fig.22A Schematic diagram of calculating the characteristic distance between characteristic vectors in the embodiment of the present application Figure 1 ;

[0077] Fig. 22B Schematic diagram of calculating the characteristic distance between characteristic vectors in the embodiment of the present application Figure 2 ;

[0078] Fig.23 This is a schematic diagram of matching feature points according to feature distance in an embodiment of the present application;

[0079] Fig.24 A schematic diagram of an application scenario of the GPU-based image feature matching method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0081] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0082] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of objects. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two, for example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0083] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0084] At present, in the fields of image processing and machine vision, as well as graphics rendering-related applications such as autonomous driving, VR / AR, etc., computing efficiency and computing accuracy have always been the main goals of the industry. Among them, feature point-based image recognition, matching, and stereo vision have attracted much attention due to their wide range of application scenarios.

[0085] The following describes the current process of image feature recognition and matching. Figure 1 As shown, the process of image feature recognition and matching may include the following steps:

[0086] (1) Feature recognition: also known as feature detection, which is to find feature points with significant properties in the image, such as corner points, edge points or spots. These feature points are usually located in the significant structures of the image and have good stability and repeatability.

[0087] (2) Feature description: For each detected feature point, calculate its descriptor. A descriptor is a vector or matrix used to represent the local image information around the feature point. Figure 1 As shown, in this step, the feature vectors of each feature point in the two frames of images are calculated respectively.

[0088] (3) Feature matching: By comparing the descriptors of feature points in different images, the most similar feature point pairs are found. Various distance metrics or similarity metrics such as Euclidean distance, Hamming distance, or correlation metrics can be used for feature matching. Usually a threshold is set to filter out matching pairs that meet the requirements. Figure 1 As shown, in this step, image feature matching is performed based on the feature vectors between the feature points of the two frames of images.

[0089] Based on the identified image feature points and matching the feature points between different images, applications such as motion estimation, image stitching, background segmentation, etc. can be realized. These matching algorithms require a large number of pixel calculations. The higher the image resolution, the more pixels need to be calculated, so the more computing resources and computing time will be consumed.

[0090] It should be noted that the image feature point matching solution is mainly calculated by multi-threading on the CPU side, but the degree of parallelism is low and the calculation time is relatively long. In addition, since the image data in the rendering pipeline needs to be copied or mapped in the graphics rendering application, the CPU accesses the image data in the rendering pipeline and performs corresponding processing, resulting in a long processing time and high memory power consumption, making it difficult for mobile applications to run at a high frame rate with limited computing power.

[0091] With the emergence of more customized computing units such as GPU and DSP in mobile devices, it is expected that feature matching can be achieved efficiently and with low power consumption through such computing units on mobile devices. However, even though hardware acceleration of such computing units has been utilized, feature matching and machine vision still face efficiency and power consumption issues on mobile devices, especially when feature matching and machine vision are applied to graphics rendering fields such as VR / AR applications. How to efficiently interact with the graphics rendering pipeline has become the main performance bottleneck.

[0092] Based on this, the embodiments of the present application provide a GPU-based image feature matching method and electronic device. In the process of matching image feature points generated by graphics rendering, the image resources are directly accessed inside the graphics rendering pipeline, and the operation of copying from the GPU memory space through the memory (such as DDR) to the CPU is cancelled, thereby greatly reducing the heat generation of the mobile device caused by the DDR bandwidth overhead. At the same time, the high-concurrency multi-threaded processing capability provided by the GPU and the thread atomic operation access to the shared cache between cores can be used to greatly accelerate the computational efficiency of feature point matching and achieve the purpose of high frame rate application.

[0093] Compared with the above-mentioned related technical solutions, the improvement of the present application solution is that the present application solution directly uses GPU-specific data to perform feature point matching operations in the rendering pipeline in graphics rendering applications, greatly improving / matching efficiency while ensuring matching accuracy and reducing computing power consumption.

[0094] That is to say, the embodiments of the present application utilize some characteristic functions of the GPU and the logic of directly processing feature point matching in the graphics rendering pipeline, which can realize the feature point matching operation of the rendered image efficiently and with low power consumption, and can realize high frame rate image feature point matching function even on a platform with limited computing power on the mobile terminal.

[0095] To facilitate understanding of the embodiments of the present application, some terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0096] Image coordinate system: a two-dimensional coordinate system, such as Figure 2 As shown in the figure, the origin (0,0) of the image coordinate system is located at the center, with pixels as the unit. Based on the image coordinate system, any pixel on the image can be located.

[0097] World coordinate system: It is a three-dimensional coordinate system, and its origin can be determined according to the situation, such as Figure 2 As shown, the world coordinate system can represent objects in space, and the unit is a length unit, such as millimeter (mm).

[0098] Specific to this application plan, such as Figure 2 As shown in the figure, during the image processing, coordinate mapping can be performed between the image coordinate system and the world coordinate system. For example, the image coordinates of the pixel point are mapped to the vertex coordinates of the object, and during the rendering process, the entire texture is fitted to the surface of the three-dimensional object model, so as to achieve a more detailed texture performance effect.

[0099] Homogeneous coordinates: Homogeneous coordinates are a very useful basic concept in computer graphics. By adding an extra dimension W to the Cartesian coordinate system, it can be used to perform matrix transformations on geometric bodies such as scaling, rotation, translation, and perspective projection. For a point (X, Y) in the Cartesian coordinate system, it becomes (x, y, w) in the homogeneous coordinate system. To convert homogeneous coordinates (x, y, w) to Cartesian coordinates, just divide x and y by w. Therefore, X and Y in the Cartesian coordinate system can be re-expressed as X = x / w, Y = y / w.

[0100] Compute shader: A compute shader that can flexibly use GPU high-speed computing to process non-graphic tasks. That is, a shader program that runs on the GPU outside the ordinary rendering pipeline. It is used for large-scale parallel general-purpose graphics processor algorithms and can improve the rendering speed of some games.

[0101] Global workgroup and local workgroup: The invocation of the compute shader uses the global workgroup, which contains local workgroups (also called local workgroups), and each local workgroup contains execution units. For example, Figure 3 As shown, the size of the global working group is (4, 4, 1) and the size of the local working group is 4×4.

[0102] Compute pipeline: It is the pipeline that controls the compute shader stage in the GPU. It is a single-process pipeline with built-in variables that can determine the location (index) of local work groups and execution units.

[0103] gl_LocalInvocationID: Indicates the location of the execution unit in the local working group.

[0104] gl_GlobalInvocationID: Indicates the position of the current execution unit in the global working group, which is a three-dimensional variable.

[0105] gl_LocalInvocationIndex is a one-dimensional index converted from the three-dimensional variable of the execution unit, which is a flattened form of gl_LocalInvocationID.

[0106] Rendering pipeline: It is the process of converting a 3D scene model into screen pixel space output. The graphics rendering pipeline accepts a set of 3D coordinates and converts the 3D coordinates into a colored 2D image on the screen.

[0107] It should be noted that the shader code can be used to instruct the GPU how to draw a pattern. The shader code determines "how to draw a specific effect", and the rendering pipeline controls "what to draw" and "which shaders to use for drawing".

[0108] SSBO: Shader storage buffer object (SSBO), SSBO is used to store byte data and upload it to graphics memory so that the shader program can process the data. For example, SSBO is used to store vertex data, element data (vertex index), uniform block data, etc. In actual implementation, SSBO can be used to pass processed data between two compute shaders.

[0109] The present application scheme adopts two SSBOs, namely SSBO1 and SSBO2. For example, in the step of calculating random numbers, the calculated random number queue is stored in SSBO1. In the step of sorting feature points, the N feature point data with the largest response value are copied to SSBO2.

[0110] Multi-threaded shared cache: The shared cache allows two or more processes or threads to access it. Specifically for the present application, in the feature point sorting step, the first N feature points with larger response values ​​are screened out in the local pixel area, and the N feature point data are sorted in descending order of response value, and the sorted feature point data are stored in the shared cache.

[0111] This application proposes an image processing acceleration solution at the software level. By utilizing some of the features of the GPU and the logic of processing feature point matching directly in the graphics rendering pipeline, the feature point matching operation of the rendered image can be realized efficiently and with low power consumption. On a platform with limited computing power on the mobile side, a high frame rate image feature point matching function can be realized. Through this application solution, the power consumption of mobile devices in feature matching and machine vision algorithms can be effectively reduced, and the application can be extended to graphics rendering application scenarios such as VR / AR, which improves image processing efficiency and calculation accuracy, and reduces computing power consumption.

[0112] The GPU-based image feature matching method provided in the embodiment of the present application can be applied to electronic devices with display functions. Electronic devices may include mobile phones, smart TVs, wearable devices, tablet computers, computers with wireless transceiver functions, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc. The embodiments of the present application do not limit the specific technology and specific device form used by the terminal device.

[0113] First, an exemplary electronic device 100 provided in an embodiment of the present application is introduced. In some embodiments, the electronic device 100 may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra mobile personal computer (UMPC), a netbook, and a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device, and / or a smart city device. The embodiment of the present application does not impose any special restrictions on the specific type of the electronic device 100.

[0114] Figure 4AThe hardware structure diagram of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 4A As shown, the electronic device 100 may include a processor 110 and a display screen 120 .

[0115] The processor 110 may include one or more processing units, for example, the processor 110 may include a central processing unit (CPU) 111 and a graphics processing unit (GPU) 112. The different processing units may be independent devices or integrated into one or more processors.

[0116] See also Figure 4B , the CPU includes a controller, one or more arithmetic and logic units (ALU) and a memory unit. Among them, the controller can be the nerve center and command center of the electronic device 100. The controller can generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. The ALU is used to perform arithmetic and logical operations. The memory unit is used to store instructions and data. In some embodiments, the memory unit can be a cache memory. The storage unit can save instructions or data that the processor 110 has just used or circulated. If the processor 110 needs to use the instruction or data again, it can be directly called from the storage unit. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0117] The GPU 112 is a microprocessor for processing images, and is connected to the CPU 111 and to the display screen 120. Figure 4B , a video memory unit is correspondingly provided for the GPU 112. The GPU 112 can receive the image data sent by the CPU, store the image data in the video memory unit, perform graphics rendering and other processing on the image data, and then send the processed image to the video memory unit, and then the display screen 120 can read the processed image from the video memory unit and display the processed image.

[0118] The display screen 120 is used to display images rendered by the GPU. The display screen 120 includes a display panel. Exemplarily, the display panel may be an organic light-emitting diode (OLED).

[0119] The electronic device 100 implements an image display function through the CPU 111 , the GPU 112 , the display screen 120 , and the like.

[0120] Taking the game scene as an example, the rendering of the image in the game needs to be completed by the CPU and GPU together. The CPU is responsible for handling some global tasks, such as determining the objects and lights that need to be provided to the GPU in a drawing instruction (Drawcall), setting up the GPU's rendering pipeline, and then notifying the GPU to render. The GPU is responsible for the detailed tasks of image drawing.

[0121] In actual implementation, the CPU initiates the rendering process. The game application uses the CPU to determine which objects need to be rendered. After finding the objects that need to be rendered, it calls the corresponding graphics API of the system one by one. Each object will call a Drawcall. The Drawcall is the process of the CPU notifying the CPU to start rendering. When the CPU receives the Drawcall, the CPU will draw the image of the specified object, and then the CPU will display the drawn two-dimensional image on the screen.

[0122] Generally speaking, CPU and GPU have different advantages in image processing.

[0123] On the one hand, the advantages of CPUs are computing power and parallel processing capabilities, which can quickly process large amounts of data and perform complex calculations. CPUs can also perform some operations, such as matrix operations and vectorization, which are very important in computer vision. In addition, CPUs can also perform some time-consuming tasks, such as sorting and searching.

[0124] On the other hand, the advantage of GPU lies in its graphics processing capabilities, which can quickly perform image processing and calculations. GPU can also perform parallel computing, which is particularly useful for large-scale image processing tasks. GPU can also perform texture merging and shaders, which can increase the accuracy of image matching.

[0125] Specifically for the present application, in order to solve the problems of efficiency and power consumption currently faced by feature matching and machine vision in mobile devices, the embodiments of the present application, in the process of matching image feature points generated by graphics rendering, can interact with the graphics rendering pipeline more efficiently by directly accessing image resources within the graphics rendering pipeline, and can reduce the power consumption of mobile devices in feature matching and machine vision algorithms. For example, when applied to graphics rendering application scenarios such as VR / AR, it can greatly improve the matching efficiency and reduce computing power consumption while ensuring matching accuracy.

[0126] The above is a specific description of the embodiment of the present application using the electronic device 100 as an example. It should be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. The electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0127] In addition, an operating system is running on the above components, such as the iOS operating system developed by Apple, the Android open source operating system developed by Google, and the Windows operating system developed by Microsoft. Application programs can be installed and run on the operating system.

[0128] The operating system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0129] Figure 5 1 is a software structure diagram of the electronic device 100 of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer (applications), the application framework layer (application framework), the Android runtime (Android runtime) module and system library, and the kernel layer (kernel).

[0130] The application layer may include a series of application packages. For example, the application layer may include video applications, game applications, autonomous driving applications, AR / VR applications, and other applications, which are not limited in any way by the present application embodiment.

[0131] The application framework layer provides an application programming interface (API) and a programming framework for the application programs of the application layer. The application framework layer includes some predefined functions. Exemplarily, the application framework layer may include a window manager, a content provider, a view system, a resource manager, etc., and the embodiments of the present application do not impose any restrictions on this.

[0132] The Android runtime module is responsible for the scheduling and management of the Android system. The Android runtime module includes a virtual machine and a core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection. The core library contains the function functions that the Java language needs to call.

[0133] The system library can include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as openGL ES), 2D graphics engine (such as SGL), etc. The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

[0134] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver and sensor driver.

[0135] For ease of explanation, Figure 5 The hardware layer that interacts with the above software structure is also embodied in. For example, the hardware layer may include a CPU, a GPU, and a display screen.

[0136] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

[0137] The execution subject of the GPU-based image feature matching method provided in the embodiment of the present application can be the above-mentioned electronic device, or it can be a functional module and / or functional entity in the electronic device that can implement the GPU-based image feature matching method, and the present application scheme can be implemented in hardware and / or software, which can be determined according to actual use requirements, and the embodiment of the present application is not limited. Below, taking an electronic device as an example, the GPU-based image feature matching method provided in the embodiment of the present application is exemplarily described in combination with the accompanying drawings.

[0138] Before introducing the GPU-based image feature matching method provided in the embodiment of the present application, the GPU-based image feature matching method provided in the embodiment of the present application is first introduced in conjunction with a specific embodiment.

[0139] In order to better understand the embodiment of the present application, the process of image feature point recognition provided by the embodiment of the present application is briefly described below. In the embodiment of the present application, the process of image feature point recognition implemented by the electronic device through the GPU may include:

[0140] Step 101: Create an image pyramid. Here, an image pyramid is created for the image to be processed in the rendering pipeline. Figure 6 The pyramid contains N levels of image data. The image data of one pixel in each pyramid layer is the average of the image data of four pixels at the corresponding position in the next pyramid layer. There is only one pixel in the top pyramid layer, and its pixel value is the average of all pixels in the entire image.

[0141] Step 102: Calculate the corner points of the pyramid layer. Here, the GPU's compute shader is used to bind the corresponding level of the image pyramid according to the scaling factor parameter. For example, the level of the 16×16 pixel block in the image pyramid is bound, and the corner points in the 16×16 pixel block are calculated using the image and depth information.

[0142] Step 103: Calculate the response function value (or simply the response value) of the corner point. The response function value is calculated for the pixel at the corner point. If the response function value is greater than or equal to a preset response threshold, the corner point is determined to be a feature point.

[0143] Step 104: Maximum suppression of feature points. The response function values ​​of all feature points in the 16×16 pixel block area are maximized by using the shared cache variables of the thread work group of the compute shader to obtain the local maximum value, and local clustered similar feature points are eliminated.

[0144] Step 105: Output feature point information, wherein the function value and pixel coordinates of the local maximum are recorded, and the response function value is written to the corresponding coordinate position in the mask image.

[0145] At this point, the recognition of image feature points is completed. In the graphics rendering application, the present application solution directly uses GPU-specific data in the rendering pipeline to perform feature point recognition operations, greatly improving recognition efficiency and reducing computing power consumption while ensuring recognition accuracy.

[0146] Combine the following Figure 7 The image feature point recognition process of the embodiment of the present application is described in detail.

[0147] Step 101: Create an image pyramid

[0148] An image pyramid is a collection of images consisting of multiple sub-images of different resolutions of an image. This group of images is generated by continuously downsampling a single image, and the smallest image may have only one pixel.

[0149] In an embodiment of the present application, the color image (or rendered image) generated by the rendering pipeline is bound as the current operation image through a texture binding instruction (glBindTexture function), and then a texture mapping instruction (glGenerateMipmap function) is called to generate an image pyramid.

[0150] The glBindTexture function is used to bind a texture, and its function is to replace the currently bound texture object with the texture object specified in the parameter so that subsequent operations can act on the texture object. For example, after the color image texture 101 is bound, the color image texture 101 can be further subjected to image processing operations.

[0151] In addition, an embodiment of the present application creates a depth image (also referred to as a texture image) DepthMipMap in R16FG16F format and the same size as the above-mentioned color image, binds the depth image DepthMipMap as an operation image through a texture binding instruction (glBindTexture function), and calls a texture mapping instruction (glGenerateMipmap function) to generate a pyramid.

[0152] It should be noted that the related art uses color images for image recognition on the CPU and GPU. The improvement of the present application scheme over the related art is that, in addition to binding the color image as the operation image and generating the corresponding image pyramid on the GPU side, the depth image can also be bound as the operation image and generate the corresponding image pyramid. In this way, the corner points in the image can be determined based on the image data of both the color image and the depth image, and the response function values ​​of the corner points can be calculated, and feature points can be identified based on the response function values ​​of the corner points.

[0153] In the embodiment of the present application, the 0th layer of the pyramid is the original image, and the data of the sampled corresponding depth image is saved in the R channel of the depth image DepthMipMap. In the embodiment of the present application, the original image refers to a depth image of the same size as the above color image.

[0154] In the embodiment of the present application, a for loop is used to calculate the depth value corresponding to the color image for each i layers in the image pyramid, starting from the first layer of the pyramid, so that the depth value corresponding to each layer in the image pyramid can be obtained.

[0155] It should be noted that the process of calculating the image data of a pixel in the i-th layer based on the image data of four pixels in the i-1th layer may include: first converting the four pixel depth values ​​of the i-1th level of the image pyramid to the camera coordinate system, then averaging the camera space coordinates of the four pixels in the camera coordinate system to obtain the camera space coordinates of the i-th level of the image pyramid, and then converting the camera space coordinates of the i-th level of the image pyramid to obtain a pixel depth value corresponding to the i-th level of the image pyramid.

[0156] First, the depth value of the pixel point (x, y) at the i-1th layer in the image pyramid is converted to the three-dimensional camera coordinate system through the following equation 1.

[0157] View_mip_i-1(x,y)=(x,y,Depth_mip_i-1(x,y),1.0)*Inverse(projectionMatrix) (Equation 1)

[0158] Where Depth_mip_i(x,y) represents the depth value corresponding to the i-1th layer in the image pyramid. projectionMatrix represents the projection matrix, and Inverse() represents the inverse matrix.

[0159] View_mip_i-1(x,y) includes View_mip_i-1(x,y).x, View_mip_i-1(x,y) and View_mip_i-1(x,y).z. Then, the camera space coordinates of the pixel point (x,y) at the i-1th layer of the image pyramid are calculated by performing homogeneous coordinate transformation using Equation 2:

[0160] View_mip_i-1(x,y)=(View_mip_i-1(x,y).x / View_mip_i-1(x,y).w,View_mip_i-1(x, y).y / View_mip_i-1(x,y).w,View_mip_i-1(x,y).z / View_mip_i-1(x,y).w,1.0) (Equation 2)

[0161] Among them, View_mip_i-1(x,y).w represents the variable introduced by the homogeneous coordinate transformation.

[0162] Among them, View_mip_i-1(x,y) represents the camera space coordinates of the pixel point (x,y) at the i-1th level of the image pyramid in the three-dimensional camera coordinate system.

[0163] It should be noted that the depth value of the pixel point (x, y) is a one-dimensional coordinate. The depth value of the pixel point (x, y) is converted to a three-dimensional camera coordinate system, and through a homogeneous coordinate transformation, the camera space coordinate of the pixel point (x, y) is obtained, and the camera space coordinate is a three-dimensional coordinate.

[0164] According to the above equations 1 and 2, the depth value of the pixel point (x, y) at the i-1th layer in the image pyramid is converted to the three-dimensional camera coordinate system to obtain the camera space coordinate View_mip_i(x, y) corresponding to the pixel point (x, y) at the i-1th layer of the image pyramid.

[0165] View_mip_i-1(x,y) includes the following three components:

[0166] (1)x component: X1=View_mip_i-1(x,y).x / View_mip_i-1(x,y).w;

[0167] (2)y component: Y1=View_mip_i-1(x,y).y / View_mip_i-1(x,y).w;

[0168] (3) z component: Z1=View_mip_i-1(x,y).z / View_mip_i-1(x,y).w.

[0169] Similarly, according to the above equations 1 and 2, the depth value of the pixel point (x+1, y) at the i-1th layer of the image pyramid can be converted to the 3D camera coordinate system to obtain the camera space coordinate View_mip_i-1(x+1, y) of the pixel point (x+1, y) at the i-1th layer in the 3D camera coordinate system. View_mip_i-1(x+1, y) includes the following three components:

[0170] (1)x component: X2=View_mip_i-1(x+1,y).x / View_mip_i-1(x+1,y).w;

[0171] (2)y component: Y2=View_mip_i-1(x+1,y).y / View_mip_i-1(x+1,y).w;

[0172] (3) z component: Z2=View_mip_i-1(x+1,y).z / View_mip_i-1(x+1,y).w.

[0173] Similarly, according to the above equations 1 and 2, the depth value of the pixel point (x, y+1) at the i-1th layer of the image pyramid can be converted to the 3D camera coordinate system to obtain the camera space coordinate View_mip_i-1(x, y+1) of the pixel point (x, y+1) at the i-1th layer in the 3D camera coordinate system. View_mip_i-1(x, y+1) includes the following three components:

[0174] (1)x component: X3=View_mip_i-1(x,y+1).x / View_mip_i-1(x,y+1).w;

[0175] (2)y component: Y3=View_mip_i-1(x,y+1).y / View_mip_i-1(x,y+1).w;

[0176] (3) z component: Z3=View_mip_i-1(x,y+1).z / View_mip_i-1(x,y+1).w.

[0177] Similarly, according to the above equations 1 and 2, the depth value of the pixel point (x+1, y+1) of the i-1th layer of the image pyramid can be converted to the 3D camera coordinate system to obtain the camera space coordinate View_mip_i-1(x+1, y+1) of the pixel point (x+1, y+1) of the i-1th layer in the 3D camera coordinate system. View_mip_i-1(x+1, y+1) includes the following three components:

[0178] (1)x component: X4=View_mip_i-1(x+1,y+1).x / View_mip_i-1(x+1,y+1).w;

[0179] (2)y component: Y4=View_mip_i-1(x+1,y+1).y / View_mip_i-1(x+1,y+1).w;

[0180] (3) z component: Z4=View_mip_i-1(x+1,y+1).z / View_mip_i-1(x+1,y+1).w.

[0181] Thus, the camera space coordinates of the four pixel points (x, y), (x+1, y), (x, y+1) and (x+1, y+1) of the i-1th level pyramid are calculated by the above equations 1 and 2.

[0182] The camera space coordinates of the pixel point (x, y) at the i-th level can be obtained by calculating the average value of the camera space coordinates of the four pixel points (x, y), (x+1, y), (x, y+1) and (x+1, y+1) at the i-1th level of the image pyramid using the following equation 3.

[0183] View_mip_i(x,y)=(View_mip_i-1(x,y)+View_mip_i-1(x+1,y)+View_mip_i-1(x,y+1)+View_mip_i-1(x+1,y+1)) / 4.0 (Equation 3)

[0184] View_mip_i-1(x,y), View_mip_i-1(x+1,y), View_mip_i-1(x,y+1), and View_mip_i-1(x+1,y+1) respectively represent the camera space coordinates of the four pixels (x,y), (x+1,y), (x,y+1), and (x+1,y+1) at the i-1 level of the image pyramid in the 3D camera coordinate system. It can be understood that the camera space coordinates of these four pixels are summed and then divided by 4 to obtain the camera space coordinates of these four pixels.

[0185] View_mip_i(x,y) represents the camera space coordinates of the pixel at level i obtained by averaging the camera space coordinates of the four pixels at level i-1 in the 3D camera coordinate system. View_mip_i(x,y) includes the following three components:

[0186] (1)x component: View_mip_i(x,y).x=(X1+X2+X3+X4) / 4;

[0187] (2)y component: View_mip_i(x,y).y=(Y1+Y2+Y3+Y4) / 4;

[0188] (3) z component: View_mip_i(x,y).z=(Z1+Z2+Z3+Z4) / 4.

[0189] Then, the pixel camera space coordinate View_mip_i(x, y) obtained in the three-dimensional camera coordinate system is multiplied by the projection matrix through the following equation 4 to obtain the NDC coordinate as an intermediate result:

[0190] NDC_mip_i(x,y)=View_mip_i(x,y)*projectionMatrix (Equation 4)

[0191] Wherein, NDC_mip_i(x,y) represents the NDC coordinate of the pixel at the i-th level of the image pyramid in the three-dimensional camera coordinate system. NDC coordinates are four-dimensional coordinates, including x, y, z, and w components. NDC_mip_i(x,y) includes NDC_mip_i(x,y).x, NDC_mip_i(x,y).y, NDC_mip_i(x,y).z, and NDC_mip_i(x,y).w. Wherein, NDC_mip_i(x,y).w represents the variable introduced by the homogeneous coordinate transformation.

[0192] Then, through the following equation 5, through homogeneous coordinate transformation, the ratio between NDC_mip_i(x, y).z and NDC_mip_i(x, y).w is calculated to obtain the depth value corresponding to the i-th level of the image pyramid:

[0193] Depth_mip_i(x,y)=NDC_mip_i(x,y).z / NDC_mip_i(x,y).w (Equation 5)

[0194] Among them, Depth_mip_i(x,y) represents the depth value corresponding to the i-th level of the image pyramid.

[0195] In an embodiment of the present application, through the above equations 1 to 5, the four pixel depth values ​​of the i-1th level of the image pyramid are first converted to the camera coordinate system, and then the camera space coordinates of the four pixels are averaged in the camera coordinate system to obtain the camera space coordinates of the i-th level of the image pyramid, and then the camera space coordinates of the i-th level of the image pyramid are converted to obtain the pixel depth value corresponding to the i-th level of the image pyramid.

[0196] In the above equations 1 to 5, i is 1, 2, ..., K in sequence, and the depth value corresponding to each layer of the color image in the K-layer image pyramid can be calculated.

[0197] The following example illustrates the process of calculating the depth value corresponding to the color image for each layer of the image pyramid.

[0198] For example, Figure 8 As shown, the image pyramid includes 8 layers. Through a for loop, starting from the first layer of the pyramid, the depth value corresponding to the color image is calculated for each layer of the image pyramid.

[0199] (1) Calculate the depth value corresponding to the first layer of the image pyramid (i is 1)

[0200] First, the color image is converted from the 2D image coordinate system to the 3D camera coordinate system using the following equation:

[0201] View_mip_0(x,y) = (x,y,Depth_mip_0(x,y),1.0) * Inverse(projectionMatrix).

[0202] View_mip_0(x,y) includes View_mip_0(x,y).x, View_mip_0(x,y).y, and View_mip_0(x,y).z.

[0203] Then, through homogeneous coordinate transformation, calculate the camera space coordinates of the four pixel points in the 0th layer in the three-dimensional camera coordinate system:

[0204] View_mip_0(x,y) =

[0205] (View_mip_0(x,y).x / View_mip_0(x,y).w,

[0206] View_mip_0(x,y).y / View_mip_0(x,y).w,

[0207] View_mip_0(x,y).z / View_mip_0(x,y).w, 1.0)

[0209] View_mip_0(x + 1,y) =

[0210] (View_mip_0(x + 1,y).x / View_mip_0(x + 1,y).w,

[0211] View_mip_0(x + 1,y).y / View_mip_0(x + 1,y).w,

[0212] View_mip_0(x + 1,y).z / View_mip_0(x + 1,y).w, 1.0)

[0214] View_mip_0(x,y + 1) =

[0215] (View_mip_0(x,y + 1).x / View_mip_0(x,y + 1).w,

[0216] View_mip_0(x,y + 1).y / View_mip_0(x,y + 1).w,

[0217] View_mip_0(x,y + 1).z / View_mip_0(x,y + 1).w, 1.0)

[0219] View_mip_0(x+1,y+1)=

[0220] (View_mip_0(x+1,y+1).x / View_mip_0(x+1,y+1).w,

[0221] View_mip_0(x+1,y+1).y / View_mip_0(x+1,y+1).w,

[0222] View_mip_0(x+1,y+1).z / View_mip_0(x+1,y+1).w, 1.0)

[0224] View_mip_0(x,y).w, View_mip_0(x+1,y).w, View_mip_0(x,y+1).w, View_mip_0(x+1,y+1).w represent variables introduced by the homogeneous coordinate transformation.

[0225] Then, the camera space coordinates of the four pixels in the 0th layer are averaged in the 3D camera coordinate system to obtain the camera space coordinates of the pixel (x, y) in the 1st layer:

[0226] View_mip_1(x,y)=(View_mip_0(x,y)+View_mip_0(x+1,y)+View_mip_0(x,y+1)+View_mip_0(x+1,y+1)) / 4.0.

[0227] Among them, View_mip_1(x,y) contains three components: View_mip_1(x,y).x, View_mip_1(x,y).y and View_mip_1(x,y).z.

[0228] Then, View_mip_1(x,y) is converted from the 3D camera coordinate system to the NDC coordinate system using the following equation:

[0229] NDC_mip_1(x,y)=View_mip_1(x,y)*projectionMatrix.

[0230] Among them, NDC_mip_1(x,y) includes four components: NDC_mip_1(x,y).x, NDC_mip_1(x,y).y, NDC_mip_1(x,y).z and NDC_mip_1(x,y).w.

[0231] Then perform homogeneous coordinate transformation to calculate the depth value corresponding to the first layer of the image pyramid:

[0232] Depth_mip_1(x,y)=NDC_mip_1(x,y).z / NDC_mip_1(x,y).w

[0233] The depth value Depth_mip_1(x,y) corresponding to the first layer of the image pyramid is calculated.

[0234] (2) Calculate the depth value corresponding to the second layer of the image pyramid (i is 2)

[0235] First, convert the depth value Depth_mip_1(x,y) corresponding to the first layer of the image pyramid to the three-dimensional camera coordinate system:

[0236] View_mip_1(x,y)=(x,y,Depth_mip_1(x,y),1.0)*Inverse(projectionMatrix).

[0237] View_mip_1(x,y) includes View_mip_1(x,y).x, View_mip_1(x,y).y and View_mip_1(x,y).z.

[0238] Then, through the homogeneous coordinate transformation, the camera space coordinates of the four pixels in the first layer are calculated in the three-dimensional camera coordinate system:

[0239] View_mip_1(x,y)=

[0240] (View_mip_1(x,y).x / View_mip_1(x,y).w,

[0241] View_mip_1(x,y).y / View_mip_1(x,y).w,

[0242] View_mip_1(x,y).z / View_mip_1(x,y).w, 1.0)

[0244] View_mip_1(x+1,y)=

[0245] (View_mip_1(x+1,y).x / View_mip_1(x+1,y).w,

[0246] View_mip_1(x+1,y).y / View_mip_1(x+1,y).w,

[0247] View_mip_1(x+1,y).z / View_mip_1(x+1,y).w, 1.0)

[0249] View_mip_1(x,y+1)=

[0250] (View_mip_1(x,y+1).x / View_mip_1(x,y+1).w,

[0251] View_mip_1(x,y+1).y / View_mip_1(x,y+1).w,

[0252] View_mip_1(x,y+1).z / View_mip_1(x,y+1).w, 1.0)

[0254] View_mip_1(x+1,y+1)=

[0255] (View_mip_1(x+1,y+1).x / View_mip_1(x+1,y+1).w,

[0256] View_mip_1(x+1,y+1).y / View_mip_1(x+1,y+1).w,

[0257] View_mip_1(x+1,y+1).z / View_mip_1(x+1,y+1).w, 1.0)

[0259] View_mip_1(x,y).w, View_mip_1(x+1,y).w, View_mip_1(x,y+1).w, View_mip_1(x+1,y+1).w represent variables introduced by the homogeneous coordinate transformation.

[0260] Then, the camera space coordinates of the four pixels in the first layer are averaged in the three-dimensional camera coordinate system to obtain the camera space coordinates of a pixel in the second layer image:

[0261] View_mip_2(x,y)=(View_mip_1(x,y)+View_mip_1(x+1,y)+View_mip_1(x,y+1)+View_mip_1(x+1,y+1)) / 4.0.

[0262] Among them, View_mip_2(x,y) contains three components: View_mip_2(x,y).x, View_mip_2(x,y).y and View_mip_2(x,y).z.

[0263] Then, View_mip_2(x,y) is converted from the 3D camera coordinate system to the DNC coordinate system by the following equation:

[0264] NDC_mip_2(x,y)=View_mip_2(x,y)*projectionMatrix.

[0265] Among them, NDC_mip_2(x,y) includes four components: NDC_mip_2(x,y).x, NDC_mip_2(x,y).y, NDC_mip_2(x,y).z and NDC_mip_2(x,y).w.

[0266] Then perform homogeneous coordinate transformation to calculate the depth value corresponding to the second layer of the image pyramid:

[0267] Depth_mip_2(x,y)=NDC_mip_2(x,y).z / NDC_mip_2(x,y).w

[0268] The depth value Depth_mip_2(x,y) corresponding to the second layer of the image pyramid is calculated.

[0269] (3) i is 3, ..., K in turn, and the depth values ​​corresponding to the 3rd, ..., Kth layers of the image pyramid are calculated.

[0270] According to the above method, the depth value Depth_mip_3 (x, y) corresponding to the 3rd layer of the image pyramid, ..., and the depth value Depth_mip_N (x, y) corresponding to the Kth layer can be calculated in sequence.

[0271] By the above method provided in the embodiment of the present application, the depth value corresponding to each layer of the image pyramid can be calculated. It should be noted that the depth values ​​(or depth information) of different levels calculated here can be applied to the following step of calculating corner points, which will be described in detail below.

[0272] Step 102: Calculate the corner points of the pyramid layer

[0273] It should be noted here that the corner point can be based on a pixel window as a basic unit, and the corner point can be defined as a window that will have a large response value change when moved in any direction.

[0274] In the embodiment of the present application, the brightness difference of each pixel of the color image in the surrounding area can be calculated, and the points with brightness difference greater than a threshold value can be regarded as corner points. It should be noted that, unlike the related art, the present application scheme not only judges the brightness difference based on the color image, but also judges the brightness difference based on the depth image, which can improve the recognition accuracy of the corner points.

[0275] Since corner points are points that are significantly different from surrounding pixels, they can be used as feature points in an image. By finding the corner points in an image, the feature points in the image can be identified based on the corner points.

[0276] In the embodiment of the present application, the color image (Color) and the depth image (DepthMipMap) of the corresponding pyramid level can be bound in the compute shader according to the specified parameters.

[0277] The specified parameters may include a zoom factor and an image ID.

[0278] For example, specifying the parameter glBindImage(ID=101,α=4) indicates that the image ID is 101 and the scaling factor α is 4. According to the specified parameters, the color image at the 4th level in the image pyramid and the ID 101 and the corresponding depth image can be bound in the compute shader.

[0279] Optionally, different zoom factors can be pre-set for games of different types or scenarios. For example, for type 1 games, the corresponding zoom factor parameter is 3, indicating that the third layer in the pyramid is taken; for type 2 games, the corresponding zoom factor parameter is 4, indicating that the fourth layer in the pyramid is taken. The specific setting can be based on actual usage requirements, and the embodiments of the present application are not limited. For ease of explanation, the embodiments of the present application take the zoom factor of 4 as an example for exemplary explanation.

[0280] For example, refer again to Figure 8 , assuming that the size of the original image is 256×256 pixels, an 8-layer pyramid is generated, and assuming that the scaling factor α=4, the fourth layer image of the pyramid can be bound in the compute shader. Among them, the size of the fourth layer image of the pyramid is 16×16 pixels. The present application can use the image brightness information of the 16×16 pixel block and its depth information (i.e., the depth value corresponding to the fourth level) to determine all corner points in the 16×16 pixel block area.

[0281] The following describes in detail the process of determining corner points according to image brightness information and depth information in an embodiment of the present application.

[0282] (1) The process of determining corner points based on image brightness information

[0283] In the compute shader of the GPU, the image brightness information may include the brightness value of each pixel in a 16×16 pixel block.

[0284] The brightness value of each pixel is calculated in the GPU's compute shader using the following equation 6:

[0285] Intensity = Color.r*0.299+Color.g*0.587+Color.b*0.114 (Equation 6)

[0286] Among them, Color.r, Color.g and Color.b represent the brightness value of a pixel in the color image (Color) on the R channel, G channel and B channel respectively. Intensity represents the brightness value of a pixel in the color image (Color). For the sake of convenience, the brightness value corresponding to the pixel point (x, y) is expressed as Intensity(x, y).

[0287] In an embodiment of the present application, the brightness difference between pixels is calculated in a compute shader of a GPU based on a pixel point (x, y) and a plurality of pixels that are at a preset distance from the pixel point (x, y).

[0288] It should be noted that the preset distance is an empirical value obtained through multiple experiments, and the preset distance can be determined according to factors such as application scenario, perspective, aperture, etc. For example, the preset distance can be 2 pixels.

[0289] For example, Fig.9A The positional relationship between the pixel point (x, y) and the pixel points (x-2, y), (x+2, y), (x, y-2) and (x, y+2) is shown.

[0290] As shown in Table 1 below, the brightness values ​​corresponding to the pixel points (x-2, y), (x+2, y), (x, y-2), and (x, y+2) are respectively expressed as Intensity(x-2, y), Intensity(x+2, y), Intensity(x, y-2), and Intensity(x, y+2).

[0291] Table 1

[0292] Pixels Brightness value (x,y) Intensity(x,y) (x-2,y) Intensity(x-2,y) (x+2,y) Intensity(x+2,y) (x,y-2) Intensity(x,y-2) (x,y+2) Intensity(x,y+2)

[0293] As shown in Table 2 below, calculate the absolute value of the brightness difference between Intensity(x,y) and Intensity(x-2,y), the absolute value of the brightness difference between Intensity(x,y) and Intensity(x+2,y), the absolute value of the brightness difference between Intensity(x,y) and Intensity(x,y-2), and the absolute value of the brightness difference between Intensity(x,y) and Intensity(x,y+2).

[0294] Table 2

[0295] Two pixels The absolute value of the brightness difference (x,y) and (x-2,y) ∣Intensity(x,y)-Intensity(x-2,y)∣ (x,y) and (x+2,y) ∣Intensity(x,y)-Intensity(x+2,y)∣ (x,y) and (x,y-2) ∣Intensity(x,y)-Intensity(x,y-2)∣ (x,y) and (x,y+2) ∣Intensity(x,y)-Intensity(x,y+2)∣

[0296] Then, in the calculation shader of the GPU, it is determined whether the absolute value of the difference in pixel brightness is greater than or equal to a preset brightness threshold, and the number of pixels whose absolute value of the difference in brightness is greater than or equal to the preset brightness threshold is counted.

[0297] If the counted number is greater than or equal to a preset number (eg, 3) and the preliminary selection condition is met, then it is determined that the current pixel may be a corner point, and the next step is performed to further determine whether the current pixel is a corner point.

[0298] If the counted number is less than the preset number, it is determined that the current pixel point cannot be a corner point and processing of the current pixel point is stopped.

[0299] It should be noted that, through step (1), the GPU's compute shader determines that some pixels in the image cannot be corner points, so these pixels can be eliminated. Then, the GPU's compute shader can make further judgments on the remaining pixels that may be corner points based on the depth information, thereby reducing the amount of calculation and improving recognition accuracy.

[0300] (2) The process of determining corner points based on depth information

[0301] The depth information is the depth value corresponding to the selected level of the image pyramid calculated in step S101 above.

[0302] The depth information corresponding to the depth image DepthMipMap may be Depth_mip_i(x, y). For example, i is 4, and the present application uses a 16×16 pixel block at the 4th layer of the image pyramid, and the depth information corresponding to the depth image DepthMipMap may be Depth_mip_4(x, y).

[0303] For the pixel points that meet the preliminary selection conditions determined in the previous step, sample the depth information (Depth_mip_x, Depth_mip_y and Depth_mip_z) corresponding to the depth image DepthMipMap according to the image coordinates (x, y) of the pixel point, and multiply the sampling result by the inverse matrix of the projection matrix to calculate the spatial coordinates of the pixel in the camera coordinate system:

[0304] Pos(x,y,z)=[(x,y)*2.0-1.0,Depth_mip_4(x,y),1.0]*Inverse(projectionMatrix) (Equation 7)

[0305] Among them, projectionMatrix represents the projection matrix, and Inverse() represents the inverse matrix.

[0306] Pos(x,y,z) consists of three components: Pos(x,y,z).x, Pos(x,y,z).y, and Pos(x,y,z).z.

[0307] Then, through homogeneous coordinate transformation, the spatial coordinates of Pos(x,y,z) can be expressed as:

[0308] Pos(x,y,z)=(Pos(x,y,z).x / Pos(x,y,z).w,Pos(x,y,z).y / Pos(x,y,z).w,Pos(x,y,z).z / Pos(x,y,z).w)

[0309] (Equation 8)

[0310] Among them, Pos(x,y,z).w represents the variable introduced by the homogeneous coordinate transformation.

[0311] Then, the spatial coordinates Pos(x, y, z) of the pixel point are converted into the two-dimensional image coordinates Pos(x, y), and the two-dimensional image coordinates Pos(x, y) of the pixel point are calculated based on the depth information of the pixel point and the coordinate conversion.

[0312] According to the depth information and coordinate conversion of the pixel point, the two-dimensional image coordinate Pos(x-2,y) of the pixel point (x-2,y) is calculated.

[0313] According to the depth information and coordinate conversion of the pixel point, the two-dimensional image coordinates Pos(x+2,y) of the pixel point (x+2,y) are calculated.

[0314] According to the depth information and coordinate conversion of the pixel point, the two-dimensional image coordinates Pos(x, y-2) of the pixel point (x, y-2) are calculated.

[0315] According to the depth information and coordinate conversion of the pixel point, the two-dimensional image coordinates Pos(x,y+2) of the pixel point (x,y+2) are calculated.

[0316] Then, calculate the absolute value of the difference in pixel brightness between the pixel points Pos(x,y) and Pos(x-2,y), Pos(x,y) and Pos(x+2,y), Pos(x,y) and Pos(x,y-2), and Pos(x,y) and Pos(x,y+2).

[0317] Next, according to an algorithm similar to step (1) above, the GPU's compute shader determines whether the absolute value of the difference in pixel brightness is greater than or equal to the preset distance threshold, and counts the number of pixels whose absolute value of the difference is greater than or equal to the preset pixel threshold. If the counted number is greater than or equal to the preset number (e.g., 3), that is, the condition is met, then it can be determined that the pixel is a corner point. If the counted number is less than the preset number (e.g., 3), then the current pixel cannot be a corner point, and processing of the current pixel stops.

[0318] In the embodiment of the present application, for each pixel point in the 16×16 pixel block area in the compute shader of the GPU, according to the above step (1), a part of the pixel points in the image is determined to be impossible to be corner points based on the image brightness information in the compute shader of the GPU, and these pixel points are eliminated. Then, according to step (2), the remaining pixel points that may be corner points are further judged based on the image depth information, thereby determining all the corner point pixels in the 16×16 pixel block area.

[0319] Step 103: Calculate the response function value of the corner pixel

[0320] The following describes in detail the process of calculating the response function value (or response value) for the corner pixel identified in step 102 in the calculation shader of the GPU.

[0321] In an embodiment of the present application, the response function value of each corner pixel is calculated based on the brightness difference and depth difference between each corner pixel (x, y) and the four adjacent pixels (x-1, y), (x+1, y), (x, y-1) and (x, y+1). Fig. 9B The positional relationship between the corner pixel (x, y) and the four adjacent pixels (x-1, y), (x+1, y), (x, y-1) and (x, y+1) is shown.

[0322] (1) Calculate the brightness values ​​of the four neighboring pixels (x-1, y), (x+1, y), (x, y-1), and (x, y+1) around the corner pixel (x, y), and calculate the depth values ​​of the four neighboring pixels around the corner pixel.

[0323] The brightness values ​​of the two pixels adjacent to the corner pixel (x, y) in the x direction: Intensity (x-1, y), Intensity (x+1, y).

[0324] The brightness values ​​of the two pixels adjacent to the corner pixel (x, y) in the y direction: Intensity (x, y-1), Intensity (x, y+1).

[0325] The depth values ​​of the two pixels adjacent to the corner pixel (x, y) in the x direction: Depth_mip(x-1, y), Depth_mip(x+1, y).

[0326] The depth values ​​of the two pixels adjacent to the corner pixel (x, y) in the y direction: Depth_mip(x, y-1), Depth_mip(x, y+1).

[0327] (2) Based on the brightness value calculated in (1) above, calculate the brightness difference of the corner pixel (x, y) in the x direction and the y direction.

[0328] Brightness difference of corner pixel (x, y) in x direction: Ix = Intensity (x+1, y) - Intensity (x-1, y);

[0329] The brightness difference of the corner pixel (x, y) in the y direction: Iy = Intensity (x, y + 1) - Intensity (x, y - 1).

[0330] (3) Based on the depth value calculated in (1) above, calculate the depth difference of the corner pixel (x, y) in the x direction and the y direction.

[0331] Depth difference of corner pixel (x, y) in x direction: dx = Depth_mip(x+1, y) - Depth_mip(x-1, y);

[0332] The depth difference of the corner pixel (x, y) in the y direction: dy = Depth_mip(x, y+1) - Depth_mip(x, y-1).

[0333] (4) Using Ix, Iy, dx and dy calculated in (3) above, the response function value (denoted as response) of the corner pixel (x, y) is calculated using an exponential function.

[0334] The response function value of the corner pixel (x, y): response = exp(Ix, Iy, dx, dy) = e (Ix,Iy,dx,dy) . (Equation 9)

[0335] In the embodiment of the present application, the response function value may be calculated for each corner point in the 16×16 pixel block area, and the response function values ​​of all corner point pixels in the 16×16 pixel block area may be calculated.

[0336] It should be noted that in the related art, the response function value of the pixel is calculated only by calculating the brightness difference between adjacent pixel points. Different from the related art, in the embodiment of the present application, not only the brightness difference between adjacent pixel points is calculated on the GPU side, but also the depth difference between adjacent pixel points is calculated on the GPU side. The GPU side calculates the response function value of each pixel based on the brightness difference between adjacent pixel points and the depth difference between adjacent pixel points, thereby improving the accuracy of calculating the response function value.

[0337] In the embodiment of the present application, the response function value response of each corner pixel can be compared with the preset response threshold to determine whether the corner pixel is a feature point. If the response function value response is greater than or equal to the preset response threshold, then the current pixel is a feature point, and step 4 is continued. If the response function value response is less than the preset response threshold, then the current pixel is not a feature point, and processing of the current pixel is stopped.

[0338] In this way, some pixels are selected from the corner pixels as feature points according to the response function values ​​of the corner pixels, which can reduce the amount of calculation and improve the calculation accuracy.

[0339] It should be noted that by screening the feature points of each 16×16 pixel block area, the following results are obtained: some 16×16 pixel block areas have one or more feature points, while some 16×16 pixel block areas have no feature points.

[0340] Therefore, through step 103, all feature points and their response function values ​​in each 16×16 pixel block area can be determined.

[0341] Step 104: Maximum suppression of feature points

[0342] Maximum suppression refers to searching for local maximum values ​​and eliminating similar feature points that are locally clustered.

[0343] Since there may be locally clustered similar feature points in the 16×16 pixel block area determined in step 103 , these similar feature points are redundant feature points, and therefore it is necessary to remove these similar feature points.

[0344] It should be noted that in the related art, maximum suppression is achieved by executing program code on the CPU through OpenCV, but unlike the related art, the present application uses shared cache variables to achieve maximum suppression.

[0345] In the embodiment of the present application, the shared cache variables of the thread workgroup of the compute shader can be used to take the maximum value of the response function value of all feature points in each 16×16 pixel block area to eliminate similar feature points that are locally clustered.

[0346] refer to Fig.10 In the embodiment of the present application, assuming that the size of the original image is W×H, then in the compute shader, W / 16×H / 16 thread work groups (or local work groups) may be used. Each thread work group includes 16×16 threads, and each thread corresponds to a feature point pixel.

[0347] In the embodiment of the present application, each thread work group defines two shared variables of the shared cache, which are respectively called shared variable 1 and shared variable 2. Shared variable 1 is used to store the maximum response value max_response in the current work group, and shared variable 2 is used to store the pixel coordinates max_response_index corresponding to the maximum response value.

[0348] Referring to Table 3 below, assuming that W×H is 256×256, W / 16×H / 16=256 thread work groups are used in the compute shader, each thread work group includes 16×16 threads, and each thread work group defines two shared variables of the shared cache. For each thread work group, the maximum response value among the 16×16 threads in the thread work group is determined and stored in shared variable 1, and the pixel coordinates corresponding to the maximum response value are stored in shared variable 2.

[0349] Table 3

[0350]

[0351] In the embodiment of the present application, for each thread work group, each feature point pixel calls the atomic operation atomicMax of the compute shader, calculates the maximum value between its own response value response and max_response, and assigns the calculated maximum value to max_response. And, if the maximum response value max_response is equal to its own response value response, its own image coordinates are assigned to max_response_index.

[0352] For example, assume that thread work group 1 has 16×16 threads, each thread corresponds to a feature point pixel, and three feature point pixels are used as an example for illustration. For example, the response values ​​of the three feature points are 1, 2, and 1.5 respectively. Compare the response value response of each feature point with max_response, and assign the maximum value to max_response. Among them, the maximum response value max_response stored in shared variable 1 is initialized to 0.

[0353] First, the atomic operation atomicMax(1,0)=1 of the compute shader is called to determine that the maximum value of the response value 1 and the initialization value 0 is 1, and the maximum value 1 is assigned to max_response, that is, the maximum response value max_response=1.

[0354] Then, the atomic operation atomicMax(2, 1)=2 of the compute shader is called again, and the maximum value can be determined to be 2. The maximum value 2 is assigned to max_response, that is, the maximum response value max_response=2.

[0355] Then, the atomic operation atomicMax(1.5, 2)=2 of the compute shader is called again, and the maximum value can be determined to be 2. The maximum value 2 is assigned to max_response, that is, the maximum response value max_response=2.

[0356] Thus, the local maximum response value can be determined. For thread work group 1, shared variable 1 stores the maximum response value in thread work group 1, and shared variable 2 stores the pixel coordinates corresponding to the maximum response value.

[0357] Through the above step 4, the shared cache variables of the thread work group of the compute shader are used to take the maximum value of the response function of all feature points in each 16×16 pixel block area to eliminate similar feature points that are locally clustered.

[0358] For example, refer again to Fig.10 For W / 16×H / 16 thread work groups (also referred to as local work groups), the local maximum response values ​​of the 16×16 threads in each thread work group can be determined according to step 104, so that W / 16×H / 16 local maximum response values ​​can be determined. The shared cache variables of the thread work groups are used to record the W / 16×H / 16 local maximum response values ​​and the pixel coordinates corresponding to the local maximum response values.

[0359] Step 105: Output feature point information

[0360] According to the multiple local maximum response values ​​and the pixel coordinates corresponding to the local maximum response values ​​respectively recorded by the shared cache variables of the thread working group, the multiple local maximum response values ​​are correspondingly written to the pixel coordinates in the mask image.

[0361] In the embodiment of the present application, the G channel of each pixel in the depth image DepthMipMap is initialized to 0.0. The G channel of the pixel can be used to save the maximum response value max_response of the thread working group. It should be noted that the G channel is used as an example for exemplary description here. In actual implementation, the R channel or B channel can also be used. The specific setting can be based on the use requirements, and the embodiment of the present application is not limited.

[0362] For each feature point, check the index gl_LocalInvocationIndex of the feature point pixel in the thread work group. If the index of feature point pixel 1 in thread work group 1 is gl_LocalInvocationIndex=0, then write the maximum response value max_response of thread work group 1 to the G channel at the max_response_index position of the depth image DepthMipMap, thereby generating image feature point information. At this point, the recognition of image feature points is completed. It should be noted that the thread position index is 0 as an example for illustrative explanation here. In actual implementation, the thread position index can be set to other values.

[0363] In the embodiment of the present application, the depth image with the maximum response value written therein is used as the mask image.

[0364] For example, Fig.11 As shown, by performing feature point recognition on the color image, five local maximum response values ​​are determined for the color image, namely feature point 1 to feature point 5, and the response values ​​and pixel coordinates of the five feature points are stored in a shared cache. Then, the response values ​​of the five feature points are written to the corresponding pixel coordinates of the mask image to obtain the mask image.

[0365] The mask image is the result of feature point recognition of the color image. All the identified feature points are marked in the mask image, and the pixel coordinates of each feature point are marked with the response value of the feature point.

[0366] It should be noted that the result of feature point recognition is simply illustrated here. It can be understood that in actual implementation, more or fewer feature points can be identified for a frame of image. The specific number can be determined based on actual conditions and is not limited in the embodiments of the present application.

[0367] The solution of the present application directly utilizes GPU-specific data to perform feature point recognition operations in the rendering pipeline in graphics rendering applications, greatly improving recognition efficiency and reducing computing power consumption while ensuring recognition accuracy.

[0368] The above describes the process of image feature point recognition by GPU provided by the embodiment of the present application in combination with specific embodiments, and the result of feature point recognition is used for feature point matching. The following describes the image feature matching method based on GPU provided by the embodiment of the present application in combination with specific embodiments.

[0369] In order to better understand the embodiments of the present application, feature point recognition and matching are first described in conjunction with the accompanying drawings.

[0370] After the feature points of each color image are recognized, the recognition result corresponding to each color image, i.e., the mask image, is obtained. A mask image is generated corresponding to each color image, and the mask image is marked with feature points and response values. For different color images, the corresponding mask images are used to perform feature point matching. For the convenience of explanation, the feature point recognition and matching of the first color image and the second color image will be used as an example for exemplary description.

[0371] For example, reference Fig. 12A and Fig. 12B , feature points of the first color image and the second color image are recognized respectively to obtain a first mask image and a second mask image, and then the feature points marked in the two mask images are compared to determine matching feature point pairs in the two images.

[0372] Specifically, the response values ​​of the feature points are marked in the first mask image, all the feature points in the first mask image are sorted according to the magnitude of the response values, and the largest N feature points are selected as the first feature point sequence. Then, a feature vector is calculated for each feature point in the first feature point sequence.

[0373] Similarly, the response values ​​of the feature points are marked in the second mask image, all the feature points in the second mask image are sorted according to the magnitude of the response values, and the largest N feature points are selected as the second feature point sequence. A feature vector is calculated for each feature point in the second feature point sequence.

[0374] Then, the distance between the feature vector of each feature point in the first feature point sequence and the feature vectors of all points in the second feature point sequence is calculated, where the two feature points with the smallest distance are the best matching points, thereby obtaining a set of pairwise corresponding feature point pairs [(x0, y0), (x1, y1)], where (x0, y0) is the image coordinate of the feature point in the first color image, and (x1, y1) is the image coordinate of the feature point matching (x0, y0) in the second color image.

[0375] pass Fig. 12A and Fig. 12B It can be seen that the embodiment of the present application performs feature point recognition on the two frames of images respectively, identifies the feature points of each of the two frames of images, and then performs feature point matching on the feature points of each of the two frames of images to determine the matching feature point pairs in the two frames of images.

[0376] The following is a brief description of the image feature point matching process provided in the embodiment of the present application. Fig.13 As shown, in the embodiment of the present application, the process of the electronic device implementing image feature point matching through the GPU may include the following steps 201 to 205:

[0377] Step 201: Calculate a random vector via GPU.

[0378] The first mask image corresponding to the first color image and the second mask image corresponding to the second color image are both loaded into the calculation shader of the GPU. The random number is calculated on the GPU using the image coordinates of the pixel to generate 256 4-dimensional integer random constants.

[0379] It should be noted that the mask image is an image obtained by performing feature point recognition on the color image (also called the real frame image), and the feature points and the response values ​​of the feature points are marked in the mask image. Fig.11 , the black area in the mask image is a non-feature point, the white area in the mask image is a feature point, and each feature point is marked with the response value of the feature point.

[0380] Step 202: Sort the feature points by GPU.

[0381] The first color image and the second color image are loaded into the calculation shader of the GPU. According to the feature points and response values ​​marked in the mask image, the feature points are sorted on the GPU according to the response value size, and the first N feature points with the largest response values ​​are selected.

[0382] Step 203: Calculate the feature vector of each feature point through the GPU.

[0383] The size of the global working group of the GPU's compute shader is set to 16×16 (i.e., 256). The feature vectors of the selected N feature points are calculated by using the shared cache and atomic operations between the thread groups of the GPU's compute shader and the 256 ivec4 random constants generated in step 1.

[0384] Step 204: Calculate the feature distance between the feature vectors in the two frames of images through the GPU.

[0385] Among them, the feature distances between the N feature vectors of the two frames of images are calculated, and the minimum feature distance is counted.

[0386] Step 205: Perform feature matching based on feature distance using the GPU.

[0387] Among them, if the minimum feature distance is less than the preset distance threshold, then the feature points to which the two feature vectors of the minimum feature distance belong are matched successfully, and the two successfully matched feature points constitute a feature point pair. If the minimum feature distance is greater than or equal to the preset distance threshold, then the matching fails. Thus, the matching of image feature points is completed.

[0388] This application proposes a technology for performing image feature matching in real time and with low power consumption on a mobile device through a GPU. By introducing the graphics rendering unit and graphics rendering pipeline of the GPU, especially introducing the GPU's unique functions such as compute shaders and inter-core shared caches and atomic operations of thread work groups, feature points of different images in the graphics rendering pipeline are matched, thereby greatly improving computing efficiency to reduce the time overhead of pixel-by-pixel calculations, while reducing the overhead of data copying and context switching between feature matching and the graphics rendering pipeline.

[0389] In an embodiment of the present application, in the process of matching image feature points generated by graphics rendering, the image resources are directly accessed inside the graphics rendering pipeline, and the copy operation through the memory between the GPU and the CPU is eliminated, thereby greatly reducing the heating of the mobile device caused by the memory bandwidth overhead. At the same time, the high-concurrency multi-threaded processing capability provided by the GPU and the thread atomic operation access to the shared cache between cores can be used to greatly accelerate the computational efficiency of feature point matching and achieve the purpose of high frame rate application.

[0390] The above briefly describes the process of image feature point matching provided in the embodiment of the present application. The following describes in detail the specific implementation method of the GPU-based image feature matching method provided in the embodiment of the present application in combination with the accompanying drawings.

[0391] Step 201: Calculate random vectors

[0392] In the embodiment of the present application, a readable and writable SSBO of size 256 and supporting ivec4 vectors is created, where ivec4 vectors represent four-component vectors of signed integers. For ease of explanation, this SSBO is denoted as SSBO1.

[0393] First of all, it should be noted that after the feature points of the color image are recognized, a mask image can be obtained. The mask image marks the image feature point pixels and the response function values ​​corresponding to the image feature point pixels. Among them, the first color image corresponds to the first mask image, and the second color image corresponds to the second mask image.

[0394] In the step of calculating the random number, the first mask image and the second mask image are loaded into the calculation shader of the GPU, such as Fig.14A As shown in the figure, the size of the local workgroup of the specified computing pipeline is 16×16, and the size of the global workgroup is (1,1,1). In other words, the global workgroup contains only one local workgroup, and 16×16 threads are used as a local workgroup. 16×16 threads are opened on the GPU, that is, 256 threads.

[0395] In the embodiment of the present application, a random number seed is generated in a compute shader. For example, the random number seed (denoted as r) is obtained by multiplying the location index (gl_LocalInvocationIndex) value by the scaling factor (scale) of the random number specified by the parameter through the following equation 10:

[0396] r = gl_LocalInvocationIndex × scale (Equation 10)

[0397] In the embodiment of the present application, each thread in the work group uses a combination of trigonometric functions and exponential functions to operate on a random number seed r to generate a 4-dimensional random vector (denoted as R).

[0398] R=e sin(r) (Equation 11)

[0399] Each vector in the 4-dimensional random vector is in the range [0.0, 1.0].

[0400] It should be noted that, in the embodiment of the present application, a total of 256 4-dimensional random vectors are generated through calculation.

[0401] In the embodiment of the present application, each 4-dimensional random vector (R) is adjusted according to an offset parameter (offset). Exemplarily, the random number can be adjusted by the following equation 12:

[0402] R' = R*20*offset (Equation 12)

[0403] It should be noted that in Equation 12, 20 is a value that can be set according to actual conditions, that is, the sampling size is limited to 20×20, and the offset parameter offset can be used to scale the sampling size based on the sampling size of 20×20, so that the scaled sampling size meets the requirements for sampling the image to be processed.

[0404] For example, assuming that the local working group size used for the image to be processed is 32×32, that is, the image block size is 32×32, then the offset parameter can be taken in the range of [1.5, 2.0]. For another example, assuming that the local working group size used for the image to be processed is 16×16, that is, the image block size is 16×16, then the offset parameter can be taken in the range of [0.25, 0.5].

[0405] In actual implementation, R'=(R*20*offset) may be a decimal form. For the convenience of calculation, the integer part of the operation result of R' can be taken. Through adjustment, the 4-dimensional random vectors are all natural numbers.

[0406] For ease of explanation, the 4-dimensional integer vector random number can be recorded as [A, B, C, D]. Among them, the two random numbers A and B represent the first random offset, and the two random numbers C and D represent the second random offset. The 4-dimensional integer vector random number can be used to calculate the feature vector of the feature point in the following steps, and the specific calculation process will be described in detail below.

[0407] Exemplarily, the adjusted 4-dimensional random vector [A, B, C, D]=[2, 2, 3, 5].

[0408] In the embodiment of the present application, adjustments are performed on 256 4-dimensional random vectors to obtain 256 adjusted 4-dimensional random vectors, and the 256 adjusted 4-dimensional random vectors constitute a random number queue.

[0409] like Fig. 14B As shown, in the embodiment of the present application, 256 adjusted 4-dimensional random vectors can be cached in SSBO1.

[0410] In summary, the present application loads two frames of mask images into the compute shader of the GPU, generates a random number seed for each frame of the mask image in the compute shader, and then uses trigonometric functions and exponential functions to operate on the random number seeds to generate 256 four-dimensional vector random numbers, then scales the four-dimensional vector random numbers, and then combines the adjusted 256 four-dimensional integer vector random numbers [A, B, C, D] to form a random number queue.

[0411] It should be noted that when calculating the feature vector of the feature point subsequently, each thread of the local working group can search for the corresponding random number in the random number queue and obtain the corresponding random offset according to its own index (gl_LocalInvocationIndex), and then use the thread coordinates and the random offset to sample the color image, obtain the RGB color value of the color image, calculate the brightness value of the color image according to the RGB color value of the color image, and then determine the feature vector of the feature point according to the brightness value of the color image.

[0412] Step 202: Feature point sorting

[0413] First, it should be noted that the mask image obtained by recognizing the image feature points has a certain width and height. For the sake of convenience, the width of the mask image is recorded as W, and the height of the mask image is recorded as H.

[0414] In the embodiment of the present application, a SSBO of a vec4 vector of size N×(W / 16)×(H / 16) is created. For ease of explanation, the SSBO is recorded as SSBO 2. The vec4 vector refers to a 4-component floating-point vector.

[0415] SSBO1 may include (W / 16)×(H / 16) segments, and each segment may store N items of data.

[0416] In the embodiment of the present application, SSBO 2 can be used to segment and save the feature point data of each of the (W / 16)×(H / 16) local working groups, where each local working group is a 16×16 image block. Since the number of feature points stored in a 16×16 image block is at most 16×16, N is less than or equal to 256.

[0417] It should be noted that N can be 8 or 12, which can be set according to actual use requirements, and is not limited in the embodiments of the present application. For the sake of convenience, the embodiments of the present application take N as 8 as an example for exemplary description. When N is 8, 0≤n≤8.

[0418] For example, it is assumed that the size of the mask image is 256×256, that is, the width W and the height H are both 256 pixels. As described above, in the embodiment of the present application, 16×16=256 threads are used as a local work group, so for a mask image with a size of 256×256, 256 local work groups can be divided.

[0419] Accordingly, the size of SSBO 2 can be:

[0420] N×(W / 16)×(H / 16)=N×(256 / 16)×(256 / 16)=256×N (Equation 13)

[0421] That is to say, SSBO 2 can store data of 256 local working groups in segments, and each segment can store N items of data.

[0422] In the feature point sorting step, two frames of mask images are loaded into the compute shader, and the size of the local workgroup of the specified computing pipeline is 16×16, and the size of the global workgroup is (W / 16, H / 16, 1), that is, the global workgroup includes W / 16*H / 16 local workgroups. In other words, the global workgroup contains W / 16×H / 16 local workgroups, and 16×16 threads are used as a local workgroup. 16×16 threads are opened on the GPU, that is, 256 threads.

[0423] For example, assuming that the width W and height H of the mask image are both 256 pixels, then Fig.15 As shown, the size of the global working group of the computing pipeline of the computing shader is (16, 16, 1), and the size of the local working group is 16×16.

[0424] In the embodiment of the present application, the feature point sorting is realized by using the shared cache mechanism between the thread work groups of the compute shader. For example, a shared cache of vec4 vectors of size N can be defined. The shared cache can store data of N feature points.

[0425] In the examples of this application, refer to Fig.16 , the global working group includes W / 16×H / 16 local working groups. Feature points are extracted from each local working group in the global working group. A local working group may contain one or more feature points, or may not contain any feature points. In order to improve the calculation rate, the number of feature points n in the local working group is controlled to be at most N, that is, the number of feature points n in each local working group is in the range [0,N].

[0426] Specifically, refer to Fig.17 , for each feature point corresponding to the local working group, the response value of the feature point can be extracted, and the atomic operation (atomicMax) can be used to count the n feature points with the largest response values ​​in the current local pixel area, and the n feature points with the largest response values ​​can be arranged in order from large to small. Then, the data of these n feature points are stored in the shared cache. Among them, in the thread whose position index (gl_LocalInvocationIndex) is 0, the data of the n feature points in the shared cache is copied to the position corresponding to SSBO 2. It should be noted that the thread with an index of 0 is used as an example for illustrative explanation here. In actual implementation, threads with indexes of other values ​​can also be selected, as long as any thread in the thread working group is selected each time to perform subsequent operations, to avoid errors caused by simultaneous execution of multiple threads.

[0427] The embodiment of the present application determines the N feature points with the largest response values ​​on the entire image by segmented sorting and segmented storage.

[0428] In one implementation, the entire image is divided into multiple local pixel regions, and the N feature points with the largest response values ​​in each local pixel region are sorted and stored in multiple segments of SSBO 2 respectively. Fig.18A , sort all feature points in all local pixel areas stored in multiple segments of SSBO 2, and select the first N feature points with the largest response values ​​in the entire image.

[0429] In another implementation, the entire image is divided into multiple local pixel regions, and the N feature points with the largest response values ​​in each local pixel region are sorted and stored in multiple segments of SSBO 2 respectively, and then the feature points in each local pixel region are sorted from large to small according to the response value. A feature point with the local maximum response value in each local pixel region is selected, and then all the selected feature points with the local maximum response value are sorted, thereby selecting the first N feature points with the local maximum response value in the entire image.

[0430] For example, reference Fig.18B , (W / 16)×(H / 16) sub-arrays can be stored in SSBO 2, and the length of each sub-array is N, wherein each sub-array can store N feature point data, and these N feature points are sorted from large to small according to the response value. In other words, the first position in each sub-array stores the value with the largest response value among the N feature points, that is, the local maximum value. For (W / 16)×(H / 16) sub-arrays, (W / 16)×(H / 16) local maxima can be extracted. Then, the (W / 16)×(H / 16) local maxima are sorted from large to small, and the globally largest N feature points are selected from the (W / 16)×(H / 16) local maxima as the N feature points with the largest response value on the entire image.

[0431] For example, assuming that N is 8, that is, the length of each subarray is 8, if there are 3 feature points, then these 3 feature points can be sorted from large to small according to the response value and stored in a subarray in SSBO 2, and 0 is written to the other 5 positions of the subarray.

[0432] refer to Fig.19 After identifying feature points of an image, a large number of feature points are identified in the image. In order to improve the efficiency of feature point matching, the embodiment of the present application uses a segmented sorting method of feature points to screen out the globally largest N feature points, that is, by sorting the feature points, only the N feature points with the largest response values ​​are retained, and the feature vectors of the N feature points with the largest response values ​​are subsequently calculated.

[0433] By using the feature point segmentation sorting method provided in the present application, the feature points of the first mask image and the second mask image are sorted respectively to obtain two sets of feature point sequences, namely, the first feature point sequence of the first mask image and the second feature point sequence of the second mask image. Exemplarily, N can be 8, that is, the first feature point sequence includes the 8 feature points with the largest response values ​​on the first mask image; the second feature point sequence includes the 8 feature points with the largest response values ​​on the second mask image.

[0434] Step 203: Calculate the feature vector of the feature point

[0435] For the first feature point sequence and the second feature point sequence generated in step 202, a feature vector (specialVector) of each feature point in the two feature point sequences is calculated.

[0436] refer to Fig. 20 After a large number of feature points in the mask image are sorted by segments, N (e.g., 8) feature points with the largest response values ​​are retained in the mask image. Then, a feature vector is calculated for each feature point: first, two pixels offset by a random offset relative to the feature point are determined, and then the feature vector of the feature point is calculated based on the brightness difference between the two pixels.

[0437] The process of calculating the feature vector of each feature point will be described in detail below.

[0438] In the embodiment of the present application, the first color image and the second color image are loaded into the compute shader, and the size of the local workgroup of the compute pipeline of the compute shader is specified to be 16×16, and the size of the global workgroup is (1, 1, 1). That is, the global workgroup contains only one local workgroup, and 16×16 threads are used as a local workgroup. 16×16 threads are opened on the GPU, that is, 256 threads.

[0439] In the compute shader of the GPU, each thread of the local workgroup uses its own location index (gl_LocalInvocationIndex) as the subscript, and searches for the 4-dimensional random vector corresponding to the subscript in the 256 4-dimensional random vectors cached in SSBO 1 as the random offset (offsetUV_i).

[0440] Among them, the random offset (offsetUV_i) can be expressed as a 4-dimensional random vector [A, B, C, D], A and B represent the first random offset, which can be recorded as offsetUV_i1; C and D represent the second random offset, which can be recorded as offsetUV_i2.

[0441] It should be noted that the reason why a 4-dimensional random vector is used in the embodiment of the present application is that two 2-dimensional random vectors are needed to do address offset. That is to say, when calculating the feature vector of a pixel, it is necessary to randomly select two neighboring pixels for the position of the current pixel. If a random neighboring pixel is selected, then a random offset in the horizontal and vertical directions is required, which requires a 2-dimensional random vector. Furthermore, if two random neighboring pixels are selected, two 2-dimensional random vectors are required, and the two 2-dimensional random vectors constitute a 4-dimensional random vector.

[0442] For example, refer to Fig.21 , assuming that the 4-dimensional random vector is [2, 2, 3, 5]. Among them, [2, 2] represents the first random offset, and [3, 5] represents the second random offset. The first offset of the feature point is made, the random offset is [2, 2], and the pixel image coordinates after the offset are (x+2, y+2). The second offset of the feature point is made, the random offset is [3, 4], and the pixel image coordinates after the offset are (x+3, y+5).

[0443] In the embodiment of the present application, for each thread, two random neighbor pixels at the current feature point pixel position are determined, the color values ​​of the two random neighbor pixels are sampled, and then the brightness of each random neighbor pixel is calculated according to the color value of each random neighbor pixel, and then the brightness of the two random neighbor pixels is compared, and the feature vector of the feature point is obtained according to the comparison result. The process of calculating the feature vector of the feature point will be described in detail below.

[0444] First, use the coordinates of each thread (gl_GlobalInvocationID.xy) and the random offset offsetUV_i1 to sample the first color image (Color1) and obtain the first RGB color value of the pixel corresponding to the thread:

[0445] Color1_i.rgba=texture(Color1,gl_GlobalInvocationID.xy+offsetUV_i1) (Equation 14)

[0446] Among them, the texture() function returns a vector of type vec4, which represents the color value of the texture at the specified position.

[0447] The above Color1_i.rgba represents the first RGB color value of the pixel. The first RGB color value is a 4-dimensional vector: Color1_i.r, Color1_i.g, Color1_i.b and Color1_i.a.

[0448] Similarly, the first color image (Color1) is sampled using the coordinates of each thread itself (gl_GlobalInvocationID.xy) and the random offset offsetUV_i2 to obtain the second RGB color value of the pixel corresponding to the thread:

[0449] Color1_i+1.rgba=texture(Color1,gl_GlobalInvocationID.xy+offsetUV_i2) (Equation 15)

[0450] The above Color1_i+1.rgba represents the second RGB color value of the pixel. The second RGB color value is a 4-dimensional vector: Color1_i+1.r, Color1_i+1.g, Color1_i+1.b and Color1_i+1.a.

[0451] Then, the first brightness value Intensity_i is calculated according to the first RGB color value of the pixel:

[0452] Intensity_i=Color1_i.r*0.299+Color1_i.g*0.587+Color1_i.b*0.114 (Equation 16)

[0453] Similarly, the second brightness value Intensity_i+1 is calculated according to the second RGB color value of the pixel:

[0454] Intensity_i+1=Color1_i+1.r*0.299+Color1_i+1.g*0.587+Color1_i+1.b*0.114 (Equation 17)

[0455] Then, each thread uses the atomic operation atomicOr() of the thread working group to calculate the relationship between the first brightness value Intensity_i and the second brightness value Intensity_i+1 corresponding to the thread, and obtains the calculation result result:

[0456] result=atomicOr(Intensity_i, Intensity_i+1) (Equation 18)

[0457] For example, if Intensity_i+1 is greater than Intensity_i, result is 1. If Intensity_i+1 is less than or equal to Intensity_i, result is 0.

[0458] In the embodiment of the present application, the above operations are performed on 16×16 threads respectively, and 16×16 results can be obtained.

[0459] Then, the 16×16 results are output to the gl_LocalInvocationIndex position of the feature vector (specialVector).

[0460] In the embodiment of the present application, multiple results can be written simultaneously through the Atomic() function, and the Atomic() function can be specifically expressed as:

[0461] Atomic(specialVector[gl_LocalInvocationIndex],result)

[0462] The Atomic() function can support writing multiple results at the same time, which can avoid read-write conflicts caused by read-write locks in the GPU.

[0463] In the embodiment of the present application, after all threads have completed calculation, a feature vector of a feature point is obtained.

[0464] In this way, a feature vector is calculated for each of the N feature points of the first color image (Color1), and finally N feature vectors are obtained. The feature vector of each feature point can be a 256-bit sequence of 0 or 1.

[0465] It should be noted that the above is an exemplary description of calculating the feature vector of each feature point in the first feature point sequence for the first color image (Color1). The process of calculating the feature vector of each feature point in the second feature point sequence for the second color image (Color2) is similar and will not be repeated. Therefore, through the algorithm provided in the embodiment of the present application, the feature vector of each feature point in the first feature point sequence can be calculated, and the feature vector of each feature point in the second feature point sequence can be calculated.

[0466] For example, referring to Table 4 below, the first feature point sequence includes N feature points with the largest response values ​​in the first color image, each of the N feature points corresponds to a feature vector, and the feature vector corresponding to each feature point is a sequence of 256 bits of 0 or 1. Similarly, the second feature point sequence includes N feature points with the largest response values ​​in the second color image, each of the N feature points corresponds to a feature vector, and the feature vector corresponding to each feature point is a sequence of 256 bits of 0 or 1.

[0467] Table 4

[0468]

[0469] Step 204: Calculate the feature distance between feature vectors

[0470] In the above step 202, a first feature point sequence (denoted as KeyPoint1) is generated for one frame image and a second feature point sequence (denoted as KeyPoint2) is generated for another frame image, and in step 203, a feature vector of each feature point in the first feature point sequence and a feature vector of each feature point in the second feature point sequence are calculated.

[0471] The two feature point sequences include N feature points respectively, each feature point corresponds to a feature vector, and the feature vector corresponding to each feature point is a 256-bit sequence of 0 or 1.

[0472] It should be noted that the int data type occupies 4 bytes in memory, that is, 32 bits, so 8 int variables can be used to represent 256 bits of data. Specifically for the present application, since the feature vector is a set of 256-bit sequences of 0 or 1, the present application uses 8 int variables to save the value of the feature vector in segments. Each int variable stores a 32-bit sequence of 0 or 1.

[0473] Among them, 8 int type variables can be represented as int svData[k], where k is 0, 1, …, 7.

[0474] Any feature vector in the first feature point sequence KeyPoint1 can be expressed as KeyPoint1.svData[0], KeyPoint1.svData[1], KeyPoint1.svData[2], KeyPoint1.svData[3], KeyPoint1.svData[4], KeyPoint1.svData[5], KeyPoint1.svData[6], KeyPoint1.svData[7].

[0475] Any feature vector in the second feature point sequence KeyPoint1 can be expressed as KeyPoint2.svData[0], KeyPoint2.svData[1, KeyPoint2.svData[2], KeyPoint2.svData[3], KeyPoint2.svData[4], KeyPoint2.svData[5], KeyPoint2.svData[6], KeyPoint2.svData[7].

[0476] In the embodiment of the present application, an XOR operation may be performed on each pair of feature vectors of the first feature point sequence KeyPoint1 and the second feature point sequence KeyPoint2:

[0477] Int result0=KeyPoint1.svData[0]^KeyPoint2.svData[0];

[0478] Int result1=KeyPoint1.svData[1]^KeyPoint2.svData[1];

[0479] Int result2=KeyPoint1.svData[2]^KeyPoint2.svData[2];

[0480] Int result3=KeyPoint1.svData[3]^KeyPoint2.svData[3];

[0481] Int result4=KeyPoint1.svData[4]^KeyPoint2.svData[4];

[0482] Int result5=KeyPoint1.svData[5]^KeyPoint2.svData[5];

[0483] Int result6=KeyPoint1.svData[6]^KeyPoint2.svData[6];

[0484] Int result7=KeyPoint1.svData[7]^KeyPoint2.svData[7].

[0485] Wherein, "^" is the symbol of the XOR operation. Table 5 shows the operation rules of the XOR operation.

[0486] Table 5

[0487]

[0488] In the embodiment of the present application, the feature vectors of the first feature point sequence KeyPoint1 and the second feature point sequence KeyPoint2 are segmented and saved using 8 int variables, and 8 XOR operations are performed on each of them, and 8 XOR results are obtained accordingly. Each XOR result is a 32-bit sequence of 0 or 1.

[0489] In the embodiment of the present application, the bitCount() instruction of the GPU can be used to count the number of 1s in each XOR result, and the number of 1s in the 8 XOR results is accumulated and summed, and the accumulated sum is used as the feature distance (svDistance) of the two feature vectors:

[0490] svDistance=bitCount(result0)+bitCount(result1)+bitCount(result2)+bitCount(result3)+bitCount(result4)+bitCount(result5)+bitCount(result6)+bitCount(result7).

[0491] The above describes a method for calculating the characteristic distance between two characteristic vectors. Based on the above method for calculating the characteristic distance between two characteristic vectors, the following describes a method for calculating the characteristic distance between each characteristic point in two characteristic point sequences provided by an embodiment of the present application.

[0492] refer to Fig.22A and Fig. 22B , the first feature point sequence (KeyPoint1) includes the N feature points (denoted as KeyPoint1_i) with the largest response value in the first color image, and each feature point corresponds to a feature vector. Similarly, the second feature point sequence (KeyPoint2) includes the N feature points (denoted as KeyPoint2_i) with the largest response value in the second color image, and each feature point corresponds to a feature vector.

[0493] In an embodiment of the present application, the feature distances between each feature point (KeyPoint1_i) in the first feature point sequence (KeyPoint1) are calculated based on the feature vectors of each feature point (KeyPoint1_i) in the second feature point sequence (Keypoint2) and the feature vectors of all feature points (Keypoint2_i, i is 1, 2, ..., N in sequence).

[0494] A total of N groups of calculations are required, and each group of calculations obtains N feature distances.

[0495] The first set of calculations: Fig.22A As shown in Table 6 below, the feature distances between each other can be calculated based on the feature vector of the first feature point (KeyPoint1_1) in the first feature point sequence (KeyPoint1) and the feature vectors of the N feature points Keypoint2_1, Keypoint2_2, ..., Keypoint2_N in the second feature point sequence (Keypoint2). For the feature point KeyPoint1_1, N feature distances can be obtained through calculation.

[0496] The second set of calculations: Fig. 22B As shown in Table 6 below, the feature distances between each other can be calculated based on the feature vector of the second feature point (KeyPoint1_2) in the first feature point sequence (KeyPoint1) and the feature vectors of the N feature points Keypoint2_1, Keypoint2_2, ..., Keypoint2_N in the second feature point sequence (Keypoint2). For the feature point KeyPoint1_2, N feature distances can be obtained through calculation.

[0497] Calculation of the Nth group: Referring to Table 6 again, it can be known that the feature distances between each other can be calculated based on the feature vector of the Nth feature point (KeyPoint1_N) in the first feature point sequence (KeyPoint1) and the feature vectors of the N feature points Keypoint2_1, Keypoint2_2, ..., Keypoint2_N in the second feature point sequence (Keypoint2). For the feature point KeyPoint1_N, N feature distances can be obtained through calculation.

[0498] Among them, Keypoint1_1, Keypoint1_2, ..., Keypoint1_N respectively represent the first feature point, the second feature point, ..., the Nth feature point of the first feature point sequence (Keypoint1). Keypoint2_1, Keypoint2_2, ..., Keypoint2_N respectively represent the first feature point, the second feature point, ..., the Nth feature point of the second feature point sequence (Keypoint2).

[0499] Table 6

[0500]

[0501] Step 205: Feature matching based on feature distance

[0502] In an embodiment of the present application, referring to Table 7 below, for each of the above N groups of calculations, find the minimum value of the N feature distances, and compare the minimum value of the N feature distances with the preset distance threshold (svDistance_Min). Based on the comparison result, determine whether the feature points match to improve the accuracy of feature point matching.

[0503] Table 7

[0504]

[0505] If the minimum distance value among the N feature distances is less than the preset distance threshold, then the two feature points corresponding to the minimum distance value are matched successfully. If the minimum distance value among the N feature distances is greater than or equal to the preset distance threshold, then the match fails.

[0506] When the matching operation of the feature points in the two frames of images is completed, a pair of feature points representing the same object in different images may be output.

[0507] For example, if Fig.23 As shown in the figure, assuming that the feature point (x1, y1) in the first color image and the feature point (x2, y2) in the second color image are two feature points that are successfully matched, then the feature point (x1, y1) and the feature point (x1, y1) constitute a feature point pair. Accordingly, the feature point pair can be recorded as [(x1, y1), (x2, y2)].

[0508] In actual implementation, after matching the feature points of two frames of images, the matching feature point pairs between the two frames of images can be determined, and then the position information of the matching feature point pairs can be used for motion estimation, image stitching, background segmentation, etc.

[0509] The following describes an application scenario of feature point recognition and matching based on a GPU provided by an embodiment of the present application. The following uses a game scenario as an example for exemplary description.

[0510] The feature point recognition and matching based on GPU provided in the embodiment of the present application can be applied to the inter-frame prediction of game images. Fig.24 , a frame of image can be predicted based on two consecutive frames of game images as the intermediate frame of game image between the two frames of game images. The two consecutive frames of game images are called real frame N-1 and real frame N+1 respectively, and the predicted intermediate frame of game image is called predicted frame N.

[0511] In an embodiment of the present application, a predicted frame can be generated based on two rendering instruction streams acquired successively. "Two rendering instruction streams acquired successively" means that rendering instruction stream 1 is acquired first, and then rendering instruction stream 2 is acquired. After acquiring rendering instruction stream 1 and before acquiring rendering instruction stream 2, the GPU has not acquired other rendering instruction streams. Specifically, the GPU can acquire two rendering instruction streams successively, one rendering instruction stream is used to draw a real frame image, and feature points are identified for each real frame image. After identifying the feature points of each real frame image, the feature points in the two real frame images are matched, and the motion vector (for example, including translation and / or rotation) of each pixel is calculated based on the image coordinate information of the matched feature point pairs, and then the 3D scene image corresponding to a real frame is offset pixel by pixel based on the motion vector, thereby generating the final predicted frame.

[0512] Assuming that the GPU generates real frame N-1 according to the rendering instruction stream 1 obtained first, generates real frame N+1 according to the rendering instruction stream 2 obtained later, and generates predicted frame N according to rendering instruction streams 1 and 2, the GPU can send these frames to the queue to be displayed in the order of real frame N-1, predicted frame N and real frame N+1. Then the display screen plays real frame N-1, predicted frame N and real frame N+1 in sequence.

[0513] It should be understood that since the resource consumption of generating predicted frames using inter-frame prediction technology is less than the resource consumption of rendering real frames by game applications, by reducing the number of real frames generated by game applications and using inter-frame prediction technology to generate predicted frames, not only the total amount of game frames can be guaranteed, but also the power consumption load of the chip can be reduced.

[0514] In the above-mentioned inter-frame prediction process, the embodiment of the present application respectively identifies and matches the feature points of the two frames of images, and then calculates the motion vector (for example, including translation and / or rotation) of each pixel based on the image coordinate information of the matched feature point pair, and then performs pixel-by-pixel offset on the 3D scene image corresponding to a certain real frame based on the motion vector to generate a predicted frame. It should be noted that the embodiment of the present application identifies and matches the feature points of different images in the graphics rendering pipeline by performing image feature recognition and matching on the GPU in real time and with low power consumption, and by introducing the GPU's graphics rendering unit and graphics rendering pipeline, especially introducing the GPU's compute shader and inter-core shared cache and atomic operations of the thread work group, which are unique to the GPU, greatly improves the computing efficiency to reduce the time overhead of pixel-by-pixel calculation, and at the same time reduces the overhead of data copying and context switching between feature recognition and matching and the graphics rendering pipeline.

[0515] This application utilizes some of the features of the GPU and the logic of processing feature point matching directly in the graphics rendering pipeline, which can achieve the feature point matching operation of the rendered image with high efficiency and low power consumption, and can achieve high frame rate image feature point matching function on the platform with limited computing power on the mobile terminal. Through the solution of this application, the power consumption of mobile terminal devices in feature matching and machine vision algorithms can be effectively reduced, and the application can be extended to graphics rendering application scenarios such as VR / AR, which improves image processing efficiency and calculation accuracy, and reduces computing power consumption.

[0516] It should be noted that, in the embodiments of the present application, "greater than" can be replaced by "greater than or equal to", "less than or equal to" can be replaced by "less than", or "greater than or equal to" can be replaced by "greater than", and "less than" can be replaced by "less than or equal to".

[0517] The various embodiments described herein may be independent solutions or may be combined according to internal logic, and all of these solutions fall within the protection scope of this application.

[0518] The above mainly describes the solution provided by the embodiment of the present application from the perspective of method steps. It is understandable that, in order to achieve the above functions, the electronic device implementing the method includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should be aware that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of protection of the present application.

[0519] The present application also provides a chip, which is coupled to a memory and is used to read and execute computer programs or instructions stored in the memory to execute the methods in the above embodiments.

[0520] The present application also provides an electronic device, which includes a chip, and the chip is used to read and execute a computer program or instruction stored in a memory, so that the method in each embodiment is executed.

[0521] This embodiment also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the GPU-based image feature matching method in the above-mentioned embodiment.

[0522] This embodiment also provides a computer program product, wherein the computer-readable storage medium stores program code. When the computer program product is run on a computer, the computer executes the above-mentioned related steps to implement the GPU-based image feature matching method in the above-mentioned embodiment.

[0523] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory so that the chip executes the GPU-based image feature matching method in the above-mentioned method embodiments.

[0524] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0525] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0526] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A GPU-based image feature matching method, characterized in that: include: Acquire feature vectors corresponding to feature points of a color image; wherein one feature point corresponds to one feature vector; when the color image is a first color image, N feature points of the first color image correspond to N feature vectors; when the color image is a second color image, N feature points of the second color image correspond to N feature vectors; Determine N feature distances between each feature point of the first color image and the N feature points of the second color image, respectively, according to the N feature vectors corresponding to the N feature points of the first color image and the N feature vectors corresponding to the N feature points of the second color image; If the minimum feature distance among the N feature distances is less than a preset distance threshold, the feature points to which the two feature vectors of the minimum feature distance belong are determined as a matching feature point pair; The step of obtaining a feature vector corresponding to a feature point of a color image includes: A mask image is loaded in a first computing shader of the GPU, and M four-dimensional random constants are generated according to image coordinate data of feature point pixels in the mask image; the mask image is an image obtained by performing feature point recognition on the color image, and the mask image marks a plurality of feature point pixels and corresponding response function values; Loading the mask image in the second computing shader of the GPU, sorting a plurality of feature point pixels marked in the mask image according to the magnitude of the response function value, and determining the first N feature point pixels with the largest response function values ​​in the mask image; The color image is loaded into the third compute shader of the GPU, and the brightness difference of adjacent pixels of the feature point pixels in the color image is determined according to the M four-dimensional random constants, and the feature vectors of the first N feature points with the largest response function values ​​are determined according to the brightness difference of adjacent pixels of the feature point pixels in the color image, as the N feature vectors corresponding to the N feature points of the color image.

2. The method according to claim 1, characterized in that Each of the M four-dimensional random constants includes a first random offset and a second random offset; The step of determining the brightness difference of adjacent pixels of the feature point pixels in the color image according to the M four-dimensional random constants includes: In a third computing shader of the GPU, for each thread of the local working group, determining a first pixel color value after being offset by the first random offset and a second pixel color value after being offset by the second random offset relative to a first feature point pixel; and calculating the first pixel brightness value according to the first pixel color value, and calculating the second pixel brightness value according to the second pixel color value; A brightness difference between the second pixel brightness value and the first pixel brightness value is calculated.

3. The method according to claim 2, characterized in that The step of determining the feature vectors of the first N feature points with the largest response function values ​​according to the brightness differences of adjacent pixels of the feature point pixels in the color image comprises: If the brightness difference is greater than 0, the output result is 1; If the brightness difference is less than or equal to 0, the output result is 0; A sequence of 0 or 1 is obtained according to the output result as a feature vector corresponding to the first feature point pixel.

4. The method according to claim 3, characterized in that When the local working group includes 16×16 threads, the feature vector corresponding to the first feature point pixel is a 256-bit sequence of 0s or 1s.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The feature vector of a feature point pixel of the first color image is stored in segments according to S integer variables; The feature vector of a feature point pixel of the second color image is stored in segments according to S integer variables; Performing an XOR operation on the i-th integer variable of the segmented storage of the first color image and the i-th integer variable of the segmented storage of the second color image, where i is 1, ..., S; Counting the number of 1s in the XOR operation result using the bitCount instruction of the GPU; The number of 1s obtained by counting is used as the feature distance between a feature point pixel of the first color image and a feature point pixel of the second color image.

6. The method according to claim 5, characterized in that When the feature vector is a 256-bit sequence of 0 or 1 and an integer variable is 32 bits, S is 8.

7. The method according to any one of claims 1 to 6, characterized in that The value of M is equal to the local workgroup size of the computing pipeline in the third computing shader of the GPU.

8. The method according to any one of claims 1 to 7, characterized in that The size of the local workgroup of the computing pipeline in the third computing shader of the GPU is equal to the size of the local workgroup of the computing pipeline in the first computing shader of the GPU; and the size of the global workgroup of the computing pipeline in the third computing shader of the GPU is equal to the size of the global workgroup of the computing pipeline in the first computing shader of the GPU.

9. The method according to any one of claims 1 to 8, characterized in that The product of the size of the local working group of the computing pipeline in the second computing shader of the GPU and the size of the global working group is equal to the size of the mask image; The size of the local workgroup of the computing pipeline in the second computing shader of the GPU is equal to the size of the local workgroup of the computing pipeline in the third computing shader of the GPU.

10. The method according to any one of claims 1 to 9, characterized in that In the first computing shader of the GPU, the size of the local working group of the computing pipeline is 16×16, and the size of the global working group is (1, 1, 1).

11. The method according to any one of claims 1 to 10, characterized in that In the second compute shader of the GPU, the size of the local working group of the compute pipeline is 16×16, the size of the global working group is (W / 16, H / 16, 1), and the size of the mask image is W×H pixels.

12. The method according to any one of claims 1 to 11, characterized in that In the third compute shader of the GPU, the size of the local work group of the compute pipeline is 16×16, and the size of the global work group is (1, 1, 1).

13. The method according to any one of claims 1 to 12, characterized in that The generating M four-dimensional random constants according to the image coordinate data of the feature point pixels in the mask image includes: Multiplying the position index value of the feature point pixel in the mask image by the specified random number scaling coefficient to obtain a random number seed; In a first computing shader of the GPU, for each thread in a local working group, a random number seed is operated using a trigonometric function and an exponential function to generate a four-dimensional random constant, and the four-dimensional random constant is adjusted according to an offset parameter to obtain an adjusted four-dimensional random constant; When the local working group includes 16×16 threads, 256 adjusted four-dimensional random constants are obtained, and M is equal to 256.

14. The method according to any one of claims 1 to 13, characterized in that After generating M four-dimensional random constants according to the image coordinate data of pixels in the mask image, the method further includes: M four-dimensional random constants are stored in a first shader storage buffer.

15. The method according to any one of claims 1 to 14, characterized in that In the second compute shader of the GPU, the global work group includes W / 16×H / 16 local work groups; The step of sorting the plurality of feature point pixels marked in the mask image according to the magnitude of the response function values ​​to determine the first N feature point pixels with the largest response function values ​​in the mask image includes: For W / 16×H / 16 local working groups, the atomic operation atomicMax of the GPU is used to determine the first N feature points with the largest response function values ​​in each local working group; the first N feature points with the largest response function values ​​are arranged from large to small according to the response function values; For W / 16×H / 16 local working groups, the first N feature points with the largest response function values ​​in the global working group are determined from the N×W / 16×H / 16 feature points as the first N feature points with the largest response function values ​​in the mask image.

16. The method according to claim 15, characterized in that The method of determining the first N feature points with the largest response function values ​​in the global working group from the N×W / 16×H / 16 feature points for the W / 16×H / 16 local working groups includes: For W / 16×H / 16 local working groups, the first N feature points with the largest response function values ​​in each local working group are sequentially stored in the second shader storage buffer; Determine the first N feature points with the largest response function values ​​in the global working group from the N×W / 16×H / 16 feature points stored in the second shader storage buffer.

17. The method according to claim 16, characterized in that For W / 16×H / 16 local working groups, the first N feature points with the largest response function values ​​in each local working group are sequentially stored in the second shader storage buffer, including: For W / 16×H / 16 local working groups, the first N feature points with the largest response function values ​​in each local working group are stored in the shared buffer between working groups according to the position index; The feature points in the shared cache are copied to corresponding positions of the second shader storage buffer according to the position index.

18. The method according to any one of claims 1 to 17, characterized in that The matched feature point pair includes a first feature point and a second feature point; After determining the feature points to which the two feature vectors with the minimum feature distance belong as a matching feature point pair, the method further includes: extracting coordinate parameters of the first feature point from a mask image corresponding to the first color image; The coordinate parameters of the second feature point are extracted from the mask image corresponding to the second color image.

19. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory, wherein the processor is configured to execute the computer program so that the electronic device implements the method according to any one of claims 1 to 18.

20. A chip comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that: The processor is configured to execute the computer program to implement the method according to any one of claims 1 to 18.

21. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 18.

Images