GPU-based image feature matching methods, electronic devices, and storage media

Abstract

The application provides a GPU-based image feature matching method, an electronic device and a storage medium, and relates to the technical field of image processing. A mask image (in which feature points and response function values are marked) is loaded in a calculation shader of a GPU to generate a four-dimensional random vector. Feature point pixels with the maximum N response function values in the mask image are determined in the calculation shader of the GPU. A color image is loaded in the calculation shader of the GPU, and feature vectors of feature points with the maximum N response function values in the color image are determined according to the four-dimensional random constant. Feature distances are calculated based on two-by-two feature vectors, and feature point matching is performed according to the feature distances. According to the application, image resources are directly accessed inside a graphics rendering pipeline for feature matching, so that the operation of copying data between the GPU and the CPU through memory is avoided, power consumption is reduced, and the calculation efficiency of feature point matching is greatly improved by using the high-concurrency multi-thread processing capability provided by the GPU.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a GPU-based image feature matching method, electronic device, and storage medium. Background Technology

[0002] Currently, in the fields of image processing and machine vision, as well as graphics rendering applications such as autonomous driving and VR / AR, computational efficiency and accuracy have always been the main focus of the industry. Among them, image recognition, matching, and stereo vision based on feature points have received much attention due to their wide range of applications.

[0003] Current feature point matching algorithms require extensive pixel computation, and the higher the image resolution, the more pixels need to be calculated, resulting in increased computational resources and time consumption. Therefore, achieving efficient and low-power feature point matching for rendered images is a pressing issue that needs to be addressed. Summary of the Invention

[0004] This application provides a GPU-based image feature matching method, electronic device, and storage medium, which solves the problems of large computational resources and long computation time required for image feature point matching.

[0005] In a first aspect, this application provides a GPU-based image feature matching method, which includes:

[0006] Obtain the feature vectors corresponding to the feature points of the color image; wherein, one feature point corresponds to one feature vector; when the color image is a first color image, N feature points of the first color image correspond to N feature vectors; when the color image is a second color image, N feature points of the second color image correspond to N feature vectors.

[0007] Based on the N feature vectors corresponding to the N feature points of the first color image and the N feature vectors corresponding to the N feature points of the second color image, determine the N feature distances between each feature point of the first color image and the N feature points of the second color image.

[0008] If the minimum feature distance among the N feature distances is less than a preset distance threshold, then the feature points to which the two feature vectors of the minimum feature distance belong are determined as a matching feature point pair;

[0009] The step of obtaining the feature vector corresponding to the feature points of the color image includes:

[0010] A mask image is loaded in the first computation shader of the GPU, and M four-dimensional random constants are generated based on the image coordinate data of the feature point pixels in the mask image; the mask image is an image obtained by feature point recognition of the color image, and the mask image marks multiple feature point pixels and their corresponding response function values;

[0011] The mask image is loaded in the second computation shader of the GPU, and the multiple feature point pixels marked in the mask image are sorted according to the size of the response function value to determine the top N feature point pixels with the largest response function value in the mask image;

[0012] The color image is loaded in the third compute shader of the GPU. The brightness difference between adjacent pixels of the feature point pixels in the color image is determined according to the M four-dimensional random constants. The feature vectors of the feature points with the largest response function values ​​in the first N feature points are determined according to the brightness difference between adjacent pixels of the feature point pixels in the color image. These feature vectors are used as the N feature vectors corresponding to the N feature points of the color image.

[0013] The GPU-based image feature matching method provided in this application loads a mask image (marked with feature points and response function values) into the GPU's computational shader and generates a four-dimensional random vector. The mask image is then loaded into the GPU's computational shader, and the top N feature point pixels with the largest response function values ​​in the mask image are determined. A color image is loaded into the GPU's computational shader, and the feature vectors of the top N feature points with the largest response function values ​​in the color image are determined based on a four-dimensional random constant. Feature distances are calculated based on pairwise feature vectors, and feature point matching is performed based on these distances. This application performs feature matching by directly accessing image resources within the graphics rendering pipeline, avoiding memory copying operations between the GPU and CPU, significantly reducing memory bandwidth overhead and resulting in overheating of mobile devices. Simultaneously, leveraging the high-concurrency multi-threaded processing capabilities provided by the GPU, the computational efficiency of feature point matching can be significantly accelerated, achieving the goal of high frame rate applications.

[0014] It should be noted that, firstly, feature point identification is performed on the rendered image (or color image) generated in the rendering pipeline, and then image feature matching is performed based on the feature points of the two images. After feature point identification on the color image, a mask image is obtained. This mask image marks the image feature point pixels and their corresponding response function values. Specifically, the first color image corresponds to the first mask image, and the second color image corresponds to the second mask image.

[0015] In some possible implementations, generating M four-dimensional random constants based on the image coordinate data of feature point pixels in the mask image includes: multiplying the position index value of the feature point pixels in the mask image by a specified random number scaling factor to obtain a random number seed; in the first computation shader of the GPU, for each thread in the local workgroup, using trigonometric and exponential functions to operate on the random number seed to generate a four-dimensional random constant, and adjusting the four-dimensional random constant according to the offset parameter to obtain an adjusted four-dimensional random constant; in the case where the local workgroup includes 16×16 threads, 256 adjusted four-dimensional random constants are obtained, where M equals 256.

[0016] For example, in the step of calculating random constants, the first and second mask images are loaded into the GPU's computation shaders, specifying the size of the local workgroup of the computation pipeline as 16×16, and the size of the global workgroup as (1,1,1). That is, the global workgroup contains only one local workgroup, with 16×16 threads forming one local workgroup. This results in 16×16 threads being started on the GPU, or 256 threads.

[0017] In this embodiment, a random number seed is generated in the computation shader. For example, the random number seed is obtained by multiplying the position index (gl_LocalInvocationIndex) value by the scaling factor (scale) of the random number specified by the parameter using Equation 11 below. Each thread in the working group uses trigonometric functions and exponential functions to perform calculations on the random number seed r to generate a 4-dimensional random vector. Then, each 4-dimensional random vector is adjusted according to the offset parameter. Adjustments are performed on all 256 4-dimensional random vectors to obtain 256 adjusted 4-dimensional random vectors, which constitute a random number queue.

[0018] In some possible implementations, each of the M four-dimensional random constants includes a first random offset and a second random offset. In this case, determining the brightness difference between adjacent pixels of a feature point pixel in the color image based on the M four-dimensional random constants includes: in the third computation shader of the GPU, for each thread of the local workgroup, determining a first pixel color value offset by the first random offset relative to the first feature point pixel and a second pixel color value offset by the second random offset; calculating the first pixel brightness value based on the first pixel color value, and calculating the second pixel brightness value based on the second pixel color value; and calculating the brightness difference between the second pixel brightness value and the first pixel brightness value.

[0019] In some possible implementations, determining the feature vector of the first N feature points with the largest response function values ​​based on the brightness difference between adjacent pixels of the feature point pixels in the color image includes: if the brightness difference is greater than 0, the output result is 1; if the brightness difference is less than or equal to 0, the output result is 0; and obtaining a sequence of 0 or 1 based on the output result as the feature vector corresponding to the first feature point pixel.

[0020] In some possible implementations, where the local working group comprises 16×16 threads, the feature vector corresponding to the first feature point pixel is a sequence of 256 bits of 0s or 1s.

[0021] In the embodiments of this application, during the matching of image feature points generated by graphics rendering, by directly accessing image resources within the graphics rendering pipeline, data copying operations between the GPU and CPU through memory can be avoided, significantly reducing the heat generation of mobile devices caused by memory bandwidth overhead. At the same time, by utilizing the high-concurrency multi-threaded processing capabilities provided by the GPU and the atomic operation of threads to access the shared cache between cores, the computational efficiency of feature point matching can be greatly accelerated, achieving the goal of high frame rate applications.

[0022] In some possible implementations, the method further includes: storing the feature vector of a feature point pixel of the first color image in segments of S integer variables; storing the feature vector of a feature point pixel of the second color image in segments of S integer variables; performing an XOR operation on the i-th integer variable of the segmented storage of the first color image and the i-th integer variable of the segmented storage of the second color image, where i takes the values ​​1, ..., S; using the bitCount instruction of the GPU to count the number of 1s in the XOR operation result; and using the counted number of 1s as the feature distance between a feature point pixel of the first color image and a feature point pixel of the second color image.

[0023] For example, two feature point sequences each include N feature points, each feature point corresponds to a feature vector, and the feature vector corresponding to each feature point is a sequence of 256 0s or 1s.

[0024] In some possible implementations, the int data type occupies 4 bytes (32 bits) in memory, so 8 int variables can represent 256 bits of data. Specifically, in this application, since the feature vector is a 256-bit sequence of 0s and 1s, this application uses 8 int variables to segment and store the feature vector values. Each int variable stores a 32-bit sequence of 0s and 1s. That is, S can be 8. By segmenting the feature vectors of both feature point sequences using 8 int variables, and performing 8 XOR operations on each pair of features, 8 XOR results are obtained. Each XOR result is a 32-bit sequence of 0s and 1s. The GPU's bitCount() instruction can be used to count the number of 1s in each XOR result, and the sum of the 1 counts in the 8 XOR results is used as the feature distance between the two feature vectors.

[0025] In this embodiment, a first feature point sequence is generated for one frame of image and a second feature point sequence is generated for another frame of image. Then, the feature vectors of each feature point in the first feature point sequence and the second feature point sequence are calculated. Next, based on the feature vectors of each feature point in the first feature point sequence and the feature vectors of all feature points in the second feature point sequence, the pairwise feature distances are calculated. The minimum feature distance is compared with a preset distance threshold, and the comparison result is used to determine whether the feature points match, thereby improving the accuracy of feature point matching.

[0026] In some possible implementations, the value of M is equal to the local workgroup size of the compute pipeline in the third compute shader of the GPU. For example, if M is 256, the local workgroup size of the compute pipeline in the third compute shader of the GPU is 16×16.

[0027] In some possible implementations, the size of the local workgroup of the computation pipeline in the third computation shader of the GPU is equal to the size of the local workgroup of the computation pipeline in the first computation shader of the GPU; and the size of the global workgroup of the computation pipeline in the third computation shader of the GPU is equal to the size of the global workgroup of the computation pipeline in the first computation shader of the GPU. For example, the size of the local workgroup of the computation pipeline is 16×16, and the size of the global workgroup of the computation pipeline is (1, 1, 1).

[0028] In some possible implementations, the product of the size of the local workgroup and the size of the global workgroup in the second computation shader of the GPU is equal to the size of the mask image. The size of the local workgroup in the second computation shader of the GPU is equal to the size of the local workgroup in the third computation shader of the GPU. For example, the size of the local workgroup in each computation pipeline is 16×16.

[0029] In some possible implementations, in the first compute shader of the GPU, the size of the local workgroup of the compute pipeline is 16×16, and the size of the global workgroup is (1, 1, 1).

[0030] In some possible implementations, in the second compute shader of the GPU, the size of the local workgroup of the compute pipeline is 16×16, the size of the global workgroup is (W / 16, H / 16, 1), and the size of the mask image is W×H pixels.

[0031] In some possible implementations, in the third compute shader of the GPU, the local workgroup size of the compute pipeline is 16×16, and the global workgroup size is (1, 1, 1).

[0032] In some possible implementations, after generating M four-dimensional random constants based on the image coordinate data of pixels in the mask image, the method further includes storing the M four-dimensional random constants in a first shader storage buffer (SSBO 1). In this embodiment, a readable and writable SSBO of size 256 supporting ivec4 vectors is created. By using the shader storage buffer, it is beneficial to utilize the high-concurrency multi-threaded processing capabilities provided by the GPU, as well as the atomic operation of threads accessing the shared cache between cores, which can significantly accelerate the computational efficiency of feature point matching and achieve the goal of high frame rate applications.

[0033] In some possible implementations, the global workgroup in the second computation shader of the GPU comprises W / 16 × H / 16 local workgroups. In this case, the above-described method of sorting the marked feature point pixels in the mask image according to their response function values ​​and determining the top N feature point pixels with the largest response function values ​​in the mask image includes: for the W / 16 × H / 16 local workgroups, using the atomicMax operation of the GPU, determining the top N feature points with the largest response function values ​​in each local workgroup; the top N feature points with the largest response function values ​​are arranged in descending order of their response function values; for the W / 16 × H / 16 local workgroups, determining the top N feature points with the largest response function values ​​in the global workgroup from the N × W / 16 × H / 16 feature points, and using these as the top N feature points with the largest response function values ​​in the mask image.

[0034] In some possible implementations, the above-mentioned method of determining the top N feature points with the largest response function values ​​in the global workgroup from N×W / 16×H / 16 feature points for W / 16×H / 16 local workgroups includes: for W / 16×H / 16 local workgroups, storing the top N feature points with the largest response function values ​​in each local workgroup sequentially in a second shader storage buffer; and determining the top N feature points with the largest response function values ​​in the global workgroup from the N×W / 16×H / 16 feature points stored in the second shader storage buffer.

[0035] In one implementation, the entire image is divided into multiple local pixel regions. The N feature points with the highest response values ​​within each local pixel region are sorted and stored in multiple segments of the second shader's storage buffer. All feature points within all local pixel regions stored in the multiple segments of the first shader's storage buffer are then sorted, and the N feature points with the highest response values ​​in the entire image are selected.

[0036] In another implementation, the entire image is divided into multiple local pixel regions. The N feature points with the largest response values ​​within each local pixel region are sorted and stored in multiple segments of the first shader's storage buffer. Then, the feature points within each local pixel region are sorted in descending order of their response values. A feature point with the largest local response value is selected from each local pixel region, and all selected feature points with the largest local response value are sorted. This process yields the N feature points with the largest local response values ​​in the entire image.

[0037] In some possible implementations, the above-mentioned method of storing the top N feature points with the largest response function values ​​in each local workgroup in the second shader storage buffer for W / 16×H / 16 local workgroups includes: storing the top N feature points with the largest response function values ​​in each local workgroup in a shared buffer between workgroups according to the position index; and copying the feature points in the shared buffer to the corresponding positions in the second shader storage buffer according to the position index.

[0038] In some possible implementations, the matched feature point pair includes a first feature point and a second feature point. In this case, after determining the feature points to which the two feature vectors with the minimum feature distance belong as the matched feature point pair, the method further includes: extracting the coordinate parameters of the first feature point from the mask image corresponding to the first color image; and extracting the coordinate parameters of the second feature point from the mask image corresponding to the second color image.

[0039] When the feature point matching operation is completed in two frames of images, the feature point pairs representing the same object in different images can be output. For example, suppose feature point (x1, y1) in the first color image and feature point (x2, y2) in the second color image are two successfully matched feature points, then feature point (x1, y1) and feature point (x2, y2) constitute a feature point pair. Accordingly, this feature point pair can be denoted as [(x1, y1), (x2, y2)].

[0040] In practice, after matching feature points between two frames of images, the matching feature point pairs between the two frames can be determined. Then, based on the position information of the matching feature point pairs, they can be used for purposes such as motion estimation, image stitching, and background segmentation.

[0041] This application proposes a technique for real-time and low-power image feature matching in mobile devices using GPUs. By introducing GPU graphics rendering units and graphics rendering pipelines, especially GPU-specific features such as compute shaders, inter-core shared caches, and atomic operations of thread workgroups, feature points of different images in the graphics rendering pipeline are matched, significantly improving computational efficiency and reducing the time overhead of pixel-by-pixel calculations. At the same time, it reduces the overhead of data copying and context switching between feature matching and the graphics rendering pipeline.

[0042] Secondly, this application provides a GPU-based image feature matching apparatus, which includes units for performing the method described in the first aspect above. This apparatus can correspond to performing the method described in the first aspect above; for a detailed description of the units within the apparatus, please refer to the description in the first aspect above, and for the sake of brevity, will not be repeated here.

[0043] The method described in the first aspect above can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a processing module or unit, a display module or unit, etc.

[0044] Thirdly, this application provides an electronic device, which includes a processor, a computer program or instructions stored in a memory, wherein the processor is used to execute the computer program or instructions to cause the method in the first aspect to be performed.

[0045] Fourthly, this application provides a computer-readable storage medium having a computer program (also referred to as instructions or code) stored thereon for implementing the method of the first aspect. For example, when the computer program is executed by a computer, it enables the computer to perform the method of the first aspect.

[0046] Fifthly, this application provides a chip including a processor. The processor is used to read and execute a computer program stored in a memory to perform the methods in the first aspect and any possible implementation thereof. Optionally, the chip further includes a memory connected to the processor via a circuit or wire.

[0047] Sixthly, this application provides a chip system including a processor. The processor is used to read and execute a computer program stored in a memory to perform the methods in the first aspect and any possible implementation thereof. Optionally, the chip system further includes a memory connected to the processor via a circuit or wire.

[0048] In a seventh aspect, this application provides a computer program product comprising a computer program (also referred to as instructions or code), which, when executed by an electronic device, causes the electronic device to implement the method in the first aspect.

[0049] It is understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the image feature recognition and matching process in the embodiments of this application;

[0051] Figure 2 This is a schematic diagram of coordinate system transformation in an embodiment of this application;

[0052] Figure 3 This is a schematic diagram of the global workgroup and local workgroup in the embodiments of this application;

[0053] Figure 4A This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0054] Figure 4B This is a schematic diagram of the CPU and GPU structure provided in the embodiments of this application;

[0055] Figure 5 A schematic diagram of the software architecture of an electronic device provided in an embodiment of this application;

[0056] Figure 6 A schematic diagram of an image pyramid provided for an embodiment of this application;

[0057] Figure 7 A flowchart of GPU-based image feature recognition provided in this application embodiment;

[0058] Figure 8 This is a schematic diagram illustrating the calculation of corner points using an image pyramid, provided in an embodiment of this application.

[0059] Figure 9A This is a schematic diagram illustrating the positional relationship between a pixel and its neighboring pixels, provided in an embodiment of this application.

[0060] Figure 9B This is a schematic diagram illustrating the positional relationship between corner points and adjacent pixels provided in an embodiment of this application.

[0061] Figure 10 This is a schematic diagram illustrating the acquisition of multiple local maximum response values ​​for a mask image, provided in an embodiment of this application.

[0062] Figure 11 A schematic diagram of the mask image after feature point recognition provided in an embodiment of this application;

[0063] Figure 12A A flowchart illustrating feature point matching between two frames of images provided in this application embodiment. Figure 1 ;

[0064] Figure 12B A flowchart illustrating feature point matching between two frames of images provided in this application embodiment. Figure 2 ;

[0065] Figure 13 A schematic flowchart illustrating the GPU-based image feature matching method provided in this application embodiment;

[0066] Figure 14A This is a schematic diagram illustrating the specification of the global and local workgroups of the computation pipeline when computing random variables through a computation shader in an embodiment of this application;

[0067] Figure 14B This is a schematic diagram illustrating the storage of random variables when calculating random variables using a computation shader, as described in an embodiment of this application.

[0068] Figure 15 This is a schematic diagram illustrating the specification of the global and local workgroups of the computation pipeline when sorting feature points using a computation shader, as described in an embodiment of this application.

[0069] Figure 16 This is a schematic diagram illustrating the extraction of feature points in units of local workgroups when sorting feature points using a computation shader in an embodiment of this application.

[0070] Figure 17 This is a schematic diagram illustrating the segmented storage of feature points of each local working group when sorting feature points using a compute shader, as described in this embodiment of the application.

[0071] Figure 18A This is a schematic diagram illustrating the segmented sorting of feature points in each local working group during feature point sorting using a compute shader in an embodiment of this application.

[0072] Figure 18B This is a schematic diagram illustrating the segmented sorting of feature points in each local working group during feature point sorting using a compute shader in an embodiment of this application.

[0073] Figure 19 This is a schematic diagram of retaining the N feature points with the largest response values ​​in the mask image after sorting feature points by a computational shader in an embodiment of this application.

[0074] Figure 20 This is a schematic diagram illustrating the calculation of feature vectors for the N feature points with the largest response values ​​in the mask image in an embodiment of this application;

[0075] Figure 21 This is a schematic diagram illustrating the calculation of feature vectors based on the brightness difference of feature point offsets in an embodiment of this application.

[0076] Figure 22A This is a schematic diagram illustrating the calculation of feature distance between feature vectors in an embodiment of this application. Figure 1 ;

[0077] Figure 22B This is a schematic diagram illustrating the calculation of feature distance between feature vectors in an embodiment of this application. Figure 2 ;

[0078] Figure 23 This is a schematic diagram illustrating feature point matching based on feature distance in an embodiment of this application;

[0079] Figure 24 This is a schematic diagram illustrating an application scenario of the GPU-based image feature matching method provided in this application embodiment. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0081] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0082] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. In the description of embodiments in this application, unless otherwise stated, "multiple" means two or more; for example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0083] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0084] Currently, in the fields of image processing and machine vision, as well as graphics rendering applications such as autonomous driving and VR / AR, computational efficiency and accuracy have always been the main focus of the industry. Among them, image recognition, matching, and stereo vision based on feature points have received much attention due to their wide range of applications.

[0085] The following explains the current process of image feature recognition and matching. (Refer to...) Figure 1 As shown, the process of image feature recognition and matching may include the following steps:

[0086] (1) Feature recognition: also known as feature detection, is the process of finding feature points with significant properties in an image, such as corner points, edge points, or spots. These feature points are usually located in the salient structures of the image and have good stability and repeatability.

[0087] (2) Feature Description: For each detected feature point, its descriptor is calculated. A descriptor is a vector or matrix used to represent local image information surrounding the feature point. For example... Figure 1 As shown, in this step, the feature vector of each feature point in the two frames of images will be calculated separately.

[0088] (3) Feature matching: By comparing the descriptors of feature points in different images, the most similar pair of feature points is found. Various distance or similarity metrics, such as Euclidean distance, Hamming distance, or correlation measures, can be used for feature matching. A threshold is usually set to filter out matching pairs that meet the requirements. Figure 1 As shown, in this step, image feature matching is performed based on the feature vectors between feature points of the two frames.

[0089] Based on the identified image feature points and by matching feature points between different images, applications such as motion estimation, image stitching, and background segmentation can be achieved. These matching algorithms require a large number of pixel calculations; the higher the image resolution, the more pixels need to be calculated, thus consuming more computational resources and time.

[0090] It should be noted that the image feature point matching scheme mainly utilizes multi-threading on the CPU side for computation, but the degree of parallelism is low and the computation time is relatively long. Furthermore, since image data in the rendering pipeline needs to be copied or mapped in graphics rendering applications, and the CPU accesses the image data in the rendering pipeline for corresponding processing, the processing time is long and the memory power consumption is high, making it difficult for mobile applications to run at high frame rates with limited computing power.

[0091] With the emergence of more customized computing units such as GPUs and DSPs in mobile devices, there is a desire to achieve efficient and low-power feature matching on these devices. However, even with the current hardware acceleration of these computing units, feature matching and machine vision on mobile devices still face efficiency and power consumption issues. In particular, when applying feature matching and machine vision to graphics rendering fields such as VR / AR applications, how to efficiently interact with the graphics rendering pipeline has become a major performance bottleneck.

[0092] Based on this, this application provides a GPU-based image feature matching method and electronic device. In the process of matching image feature points generated by graphics rendering, the image resources are accessed directly inside the graphics rendering pipeline, eliminating the operation of copying from GPU memory space to CPU via memory (e.g., DDR). This significantly reduces the heat generation of mobile devices caused by DDR bandwidth overhead. At the same time, by utilizing the high-concurrency multi-threaded processing capabilities provided by the GPU and the atomic operation of threads to access the shared cache between cores, the computational efficiency of feature point matching can be greatly accelerated, achieving the purpose of high frame rate applications.

[0093] Compared with the solutions of the above-mentioned related technologies, the improvement of the solution of this application is that: the solution of this application directly uses GPU-specific data to perform feature point matching operations in the rendering pipeline in graphics rendering applications, which greatly improves matching efficiency and reduces computational power consumption while ensuring matching accuracy.

[0094] In other words, by utilizing some features of the GPU and the logic of directly processing feature point matching in the graphics rendering pipeline, the feature point matching operation of the rendered image can be implemented efficiently and with low power consumption. Even on mobile platforms with limited computing power, high frame rate image feature point matching can be achieved.

[0095] To facilitate understanding of the embodiments of this application, some terms used in the embodiments of this application are explained below, so that those skilled in the art can understand them.

[0096] Image coordinate system: a two-dimensional coordinate system, such as... Figure 2 As shown, the origin (0,0) of the image coordinate system is located at the center, and the coordinates are in pixels. Based on the image coordinate system, any pixel on the image can be located.

[0097] World coordinate system: a three-dimensional coordinate system, the origin of which can be determined as needed, such as... Figure 2 As shown, the world coordinate system can represent objects in space, with units of length, such as millimeters (mm).

[0098] Specifically, regarding the proposed solution in this application, such as Figure 2 As shown, during image processing, coordinate mapping can be performed between the image coordinate system and the world coordinate system. For example, mapping the image coordinates of a pixel to the vertex coordinates of an object will cause the entire texture to be applied to the surface of the 3D object model during rendering, thereby achieving a more detailed texture representation effect.

[0099] Homogeneous coordinates: Homogeneous coordinates are a very useful fundamental concept in computer graphics. By adding an extra dimension W to the Cartesian coordinate system, they can be used for matrix transformations of geometry, such as scaling, rotation, translation, and perspective projection. A point (X, Y) in Cartesian coordinates becomes (x, y, w) in homogeneous coordinates. To convert homogeneous coordinates (x, y, w) to Cartesian coordinates, simply divide x and y by w. Therefore, X and Y in Cartesian coordinates can be rewritten as X = x / w, Y = y / w.

[0100] Computational shaders are shaders that can flexibly utilize the high-speed computing of the GPU to process non-graphics tasks. They are shader programs that run on the GPU outside of the ordinary rendering pipeline and are used for massively parallel general-purpose graphics processor algorithms, which can improve the rendering speed of some games.

[0101] Global and Local Workgroups: Calls to compute shaders take place within a global workgroup, which contains local workgroups (also called local groups). Each local workgroup contains execution units. For example, such as... Figure 3 As shown, the size of the global workgroup is (4, 4, 1), and the size of the local workgroup is 4×4.

[0102] The compute pipeline is the pipeline in the GPU that controls the compute shader stage. It is a single-process pipeline with built-in variables that can determine the location (index) of local workgroups and execution units.

[0103] gl_LocalInvocationID: Indicates the location of the execution unit within the local workgroup.

[0104] gl_GlobalInvocationID: Represents the position of the current execution unit in the global workgroup; it is a three-dimensional variable.

[0105] gl_LocalInvocationIndex is a one-dimensional index obtained by converting the three-dimensional variables of the execution unit. It is a flattened form of gl_LocalInvocationID.

[0106] The rendering pipeline is the process of converting a 3D scene model into the screen's pixel space for output. The graphics rendering pipeline accepts a set of 3D coordinates and transforms them into a colored 2D image on the screen.

[0107] It should be noted that shader code can be used to instruct the GPU how to draw patterns. The shader code determines "how to draw the specific effects," while the rendering pipeline controls "what is drawn" and "which shaders are used for drawing."

[0108] SSBO: Shader Storage Buffer Object. SSBOs are used to store byte data and upload it to graphics memory so that shader programs can process the data. For example, SSBOs are used to store vertex data, element data (vertex indices), uniform block data, etc. In practice, SSBOs can be used to transfer processed data between two compute shaders.

[0109] This application employs two SSBOs, designated SSBO1 and SSBO2. For example, in the random number calculation step, the calculated random number queue is stored in SSBO1. In the feature point sorting step, the N feature point data with the largest response values ​​are copied to SSBO2.

[0110] Multi-threaded shared cache: A shared cache allows two or more processes or threads to access it. Specifically, in the feature point sorting step, the top N feature points with larger response values ​​are selected from a local pixel region, and the data of these N feature points are sorted in descending order of response values. The sorted feature point data is then stored in a shared cache.

[0111] This application proposes a software-level image processing acceleration scheme. Utilizing certain features of the GPU and logic for directly processing feature point matching within the graphics rendering pipeline, it enables efficient and low-power feature point matching operations for rendered images. Even on mobile platforms with limited computing power, it achieves high frame rate image feature point matching. This solution effectively reduces power consumption in feature matching and machine vision algorithms on mobile devices, expanding its application to graphics rendering scenarios such as VR / AR, improving image processing efficiency and computational accuracy while reducing computational power consumption.

[0112] The GPU-based image feature matching method provided in this application can be applied to electronic devices with display functions. These electronic devices may include mobile phones, smart TVs, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, and so on. The embodiments of this application do not limit the specific technology or device form used in the terminal devices.

[0113] First, an exemplary electronic device 100 provided in the embodiments of this application is introduced. In some embodiments, the electronic device 100 may be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device and / or smart city device. The embodiments of this application do not impose any special limitations on the specific type of the electronic device 100.

[0114] Figure 4AThis is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Figure 4A As shown, the electronic device 100 may include a processor 110 and a display screen 120.

[0115] Processor 110 may include one or more processing units, such as a central processing unit (CPU) 111 and a graphics processing unit (GPU) 112. These different processing units may be independent devices or integrated into one or more processors.

[0116] See Figure 4B The CPU includes a controller, one or more arithmetic and logic units (ALUs), and a memory unit. The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. The ALU is used to perform arithmetic and logical operations. The memory unit is used to store instructions and data. In some embodiments, the memory unit can be a cache memory. This memory unit can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from the memory unit. This avoids repeated accesses, reduces the processor 110's waiting time, and thus improves system efficiency.

[0117] GPU 112 is a microprocessor for processing images, connected to CPU 111 and connected to display screen 120. See also Figure 4B The GPU 112 is equipped with a corresponding video memory unit. The GPU 112 can receive image data sent by the CPU, store the image data in the video memory unit, perform graphics rendering and other processing on the image data, and then send the processed image to the video memory unit. The display screen 120 can then read the processed image from the video memory unit and display the processed image.

[0118] Display screen 120 is used to display images rendered by the GPU. Display screen 120 includes a display panel. Exemplarily, the display panel may be an organic light-emitting diode (OLED).

[0119] The electronic device 100 implements image display function through CPU 111, GPU 112 and display screen 120.

[0120] Taking a game scene as an example, the rendering of images in a game requires the joint work of the CPU and GPU. The CPU is responsible for handling some global tasks, such as determining the objects and lights that need to be provided to the GPU in a draw call, setting up the GPU's rendering pipeline, and then notifying the GPU to perform the rendering. The GPU is responsible for the detailed tasks of image drawing.

[0121] In actual implementation, the rendering process is initiated by the CPU. The game application uses the CPU to determine which objects need to be rendered. After finding the objects that need to be rendered, it calls the corresponding graphics API of the system one by one. Each object will call a Drawcall, which is the process by which the CPU notifies the CPU to start rendering. When the CPU receives the Drawcall, it will draw the image of the specified object and then display the drawn 2D image on the screen.

[0122] Typically, CPUs and GPUs have different strengths in image processing.

[0123] On one hand, the advantages of a CPU lie in its computing power and parallel processing capabilities, enabling it to quickly process large amounts of data and perform complex calculations. A CPU can also perform operations such as matrix operations and vectorization, which are crucial in computer vision. Furthermore, a CPU can execute time-consuming tasks such as sorting and searching.

[0124] On the other hand, the advantage of GPUs lies in their graphics processing power, enabling them to perform image processing and computation quickly. GPUs can also perform parallel computing, which is particularly useful for large-scale image processing tasks. GPUs can also perform texture merging and shader operations, which can increase the accuracy of image matching.

[0125] Specifically, in order to address the efficiency and power consumption issues currently faced by feature matching and machine vision on mobile devices, this application embodiment, during the matching process of image feature points generated by graphics rendering, directly accesses image resources within the graphics rendering pipeline, enabling more efficient interaction with the graphics rendering pipeline. This reduces the power consumption of mobile devices in feature matching and machine vision algorithms. For example, when applied to graphics rendering application scenarios such as VR / AR, it can significantly improve matching efficiency while ensuring matching accuracy and reducing computational power consumption.

[0126] The above is a detailed description of the embodiments of this application using electronic device 100 as an example. It should be understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on electronic device 100. Electronic device 100 may have more or fewer components than shown in the figures, may combine two or more components, or may have different component configurations. The various components shown in the figures can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0127] In addition, an operating system runs on top of these components. Examples include Apple's iOS, Google's Android, and Microsoft's Windows. Applications can be installed and run on this operating system.

[0128] The operating system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0129] Figure 5 This is a software structure block diagram of an electronic device 100 according to an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: applications, application framework, Android runtime module and system libraries, and kernel.

[0130] The application layer may include a series of application packages. For example, the application layer may include video applications, game applications, autonomous driving applications, AR / VR applications, etc., and this application embodiment does not impose any limitations on this.

[0131] The application framework layer provides an application programming interface (API) and programming framework for applications within the application layer. The application framework layer includes some predefined functions. For example, the application framework layer may include a window manager, content provider, view system, resource manager, etc., but this application embodiment does not impose any limitations on this.

[0132] The Android runtime module is responsible for scheduling and managing the Android system. It consists of a virtual machine and core libraries. The application layer and application framework layer run within the virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is responsible for managing object lifecycles, stack management, thread management, security and exception management, and garbage collection. The core libraries contain the functionalities required by the Java language.

[0133] The system library can include multiple functional modules. Examples include: a surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), and 2D graphics engines (e.g., SGL). The surface manager manages the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

[0134] The kernel layer is the layer between hardware and software. The kernel layer contains at least display drivers and sensor drivers.

[0135] For ease of explanation, Figure 5 The document also includes a hardware layer that interacts with the aforementioned software architecture. For example, the hardware layer may include a CPU, GPU, and display screen.

[0136] It should be noted that although the embodiments of this application are described using the Android system as an example, the basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

[0137] The execution subject of the GPU-based image feature matching method provided in this application embodiment can be the aforementioned electronic device, or it can be a functional module and / or functional entity within the electronic device capable of implementing the GPU-based image feature matching method. Furthermore, the solution of this application can be implemented through hardware and / or software, and the specific implementation can be determined according to actual usage requirements; this application embodiment does not impose any limitations. The following description uses an electronic device as an example, combined with the accompanying drawings, to exemplarily illustrate the GPU-based image feature matching method provided in this application embodiment.

[0138] Before introducing the GPU-based image feature matching method provided in the embodiments of this application, we will first introduce the GPU-based image feature matching method provided in the embodiments of this application in conjunction with specific embodiments.

[0139] To better understand the embodiments of this application, the image feature point recognition process provided in the embodiments of this application will be briefly described below. In the embodiments of this application, the process of image feature point recognition implemented by an electronic device through a GPU may include:

[0140] Step 101: Create an image pyramid. This involves creating an image pyramid for the images to be processed in the rendering pipeline. (Reference) Figure 6 The pyramid contains image data across N levels. The image data for one pixel in each level of the pyramid is the average of the four pixels at the corresponding position in the next level. This continues until the top level of the pyramid contains only one pixel, whose value is the average of all pixels in the entire image.

[0141] Step 102: Calculate the corner points of the pyramid layer. This involves using the GPU's compute shader to bind the corresponding levels of the image pyramid based on the scaling factor parameters. For example, binding the level of a 16×16 pixel block in the image pyramid and calculating the corner points within that 16×16 pixel block using image and depth information.

[0142] Step 103: Calculate the response function value (or simply response value) of the corner point. Specifically, calculate the response function value for each pixel at the corner point location. If the response function value is greater than or equal to a preset response threshold, then the corner point is determined to be a feature point.

[0143] Step 104: Feature point local maxima suppression. This involves using the shared cache variable of the compute shader's thread workgroup to maximize the response function values ​​of all feature points within a 16×16 pixel block region, thus eliminating locally clustered similar feature points.

[0144] Step 105: Output feature point information. This involves recording the function value and pixel coordinates of the local maxima, and writing the response function value to the corresponding coordinate position in the mask image.

[0145] This completes the identification of image feature points. The proposed solution directly utilizes GPU-specific data within the rendering pipeline in graphics rendering applications to perform feature point identification, significantly improving identification efficiency and reducing computational power consumption while maintaining accuracy.

[0146] The following is combined with Figure 7 The image feature point recognition process of the embodiments of this application is described in detail.

[0147] Step 101: Creating an Image Pyramid

[0148] An image pyramid is a collection of images composed of multiple sub-images of different resolutions from a single image. This set of images is generated by repeatedly downsampling individual images, with the smallest image possibly containing only a single pixel.

[0149] In this embodiment, the color image (or rendered image) generated by the rendering pipeline is bound to the currently operated image by the texture binding instruction (glBindTexture function), and then the texture mapping instruction (glGenerateMipmap function) is called to generate the image pyramid.

[0150] The `glBindTexture` function is used to bind a texture. Its function is to replace the currently bound texture object with the texture object specified in the parameters, so that subsequent operations will be applied to that texture object. For example, after binding the color image `texture101`, image processing operations can be further performed on that color image `texture 101`.

[0151] In addition, this application embodiment creates a depth image (also called a texture image) DepthMipMap in R16FG16F format and with the same size as the color image described above. The depth image DepthMipMap is also bound to the operation image through the texture binding instruction (glBindTexture function), and the texture mapping instruction (glGenerateMipmap function) is called to generate the pyramid.

[0152] It should be noted that the related technologies perform image recognition using color images on the CPU and GPU. The improvement of this application's solution compared to the related technologies lies in the following: on the GPU side, in addition to binding the color image as the operation image and generating the corresponding image pyramid, it is also possible to bind the depth image as the operation image and generate the corresponding image pyramid. This allows for the determination of corner points in the image and the calculation of the corner point response function values ​​based on image data from both the color image and the depth image, as well as the recognition of feature points based on the corner point response function values.

[0153] In this embodiment, the 0th layer of the pyramid is the original image, and the data of the sampled corresponding depth image is saved to the R channel of the DepthMipMap. In this embodiment, the original image refers to the depth image with the same size as the color image described above.

[0154] In this embodiment, a for loop is used to calculate the depth value corresponding to the color image for each i-th layer of the image pyramid, starting from the first layer. This allows the acquisition of the depth value corresponding to each layer of the image pyramid.

[0155] It should be noted that the process of calculating the image data of one pixel in the i-th layer based on the image data of four pixels in the (i-1)-th layer may include: first, transforming the depth values ​​of the four pixels in the (i-1)-th layer of the image pyramid to the camera coordinate system; then, averaging the camera space coordinates of these four pixels in the camera coordinate system to obtain the camera space coordinates of the i-th layer of the image pyramid; and then transforming the camera space coordinates of the i-th layer of the image pyramid to obtain the depth value of one pixel corresponding to the i-th layer of the image pyramid.

[0156] First, the depth values ​​of the (x, y) pixels in the (i-1)th layer of the image pyramid are transformed into the 3D camera coordinate system using Equation 1 below.

[0157] View_mip_i-1(x,y)=(x,y,Depth_mip_i-1(x,y),1.0)*Inverse(projectionMatrix) (Equation 1)

[0158] Here, Depth_mip_i(x,y) represents the depth value corresponding to the (i-1)th layer in the image pyramid. projectionMatrix represents the projection matrix, and Inverse() represents the inverse matrix.

[0159] View_mip_i-1(x,y) includes View_mip_i-1(x,y).x, View_mip_i-1(x,y), and View_mip_i-1(x,y).z. Then, a homogeneous coordinate transformation is performed using Equation 2 to calculate the camera space coordinates of the (x,y) pixel at the (i-1)th layer of the image pyramid:

[0160] View_mip_i-1(x,y)=(View_mip_i-1(x,y).x / View_mip_i-1(x,y).w,View_mip_i-1(x, y).y / View_mip_i-1(x,y).w,View_mip_i-1(x,y).z / View_mip_i-1(x,y).w,1.0) (Equation 2)

[0161] Here, View_mip_i-1(x,y).w represents the variable introduced by the homogeneous coordinate transformation.

[0162] Here, View_mip_i-1(x,y) represents the camera space coordinates of the pixel (x,y) at the (i-1)th level of the image pyramid in the 3D camera coordinate system.

[0163] It should be noted that the depth value of the pixel (x,y) is a one-dimensional coordinate. The depth value of the pixel (x,y) is transformed into a three-dimensional camera coordinate system, and then the camera space coordinates of the pixel (x,y) are obtained through homogeneous coordinate transformation. These camera space coordinates are three-dimensional coordinates.

[0164] Based on Equations 1 and 2 above, the depth value of the (i-1)th layer pixel (x,y) in the image pyramid is transformed to the three-dimensional camera coordinate system to obtain the camera space coordinates View_mip_i(x,y) corresponding to the (i-1)th layer pixel (x,y) in the image pyramid.

[0165] View_mip_i-1(x,y) includes the following three components:

[0166] (1)x component: X1=View_mip_i-1(x,y).x / View_mip_i-1(x,y).w;

[0167] (2)y component: Y1=View_mip_i-1(x,y).y / View_mip_i-1(x,y).w;

[0168] (3) z component: Z1=View_mip_i-1(x,y).z / View_mip_i-1(x,y).w.

[0169] Similarly, according to Equations 1 and 2 above, the depth value of the pixel (x+1, y) at the (i-1)th layer of the image pyramid can be transformed to the 3D camera coordinate system, obtaining the camera space coordinates View_mip_i-1(x+1, y) of the pixel (x+1, y) at the (i-1)th layer in the 3D camera coordinate system. View_mip_i-1(x+1, y) includes the following three components:

[0170] (1)x component: X2=View_mip_i-1(x+1,y).x / View_mip_i-1(x+1,y).w;

[0171] (2)y component: Y2=View_mip_i-1(x+1,y).y / View_mip_i-1(x+1,y).w;

[0172] (3) z component: Z2=View_mip_i-1(x+1,y).z / View_mip_i-1(x+1,y).w.

[0173] Similarly, according to Equations 1 and 2 above, the depth value of the pixel (x, y+1) at the (i-1)th layer of the image pyramid can be transformed to the 3D camera coordinate system, obtaining the camera space coordinates View_mip_i-1(x, y+1) of the pixel (x, y+1) at the (i-1)th layer in the 3D camera coordinate system. View_mip_i-1(x, y+1) includes the following three components:

[0174] (1)x component: X3=View_mip_i-1(x,y+1).x / View_mip_i-1(x,y+1).w;

[0175] (2)y component: Y3=View_mip_i-1(x,y+1).y / View_mip_i-1(x,y+1).w;

[0176] (3) z component: Z3=View_mip_i-1(x,y+1).z / View_mip_i-1(x,y+1).w.

[0177] Similarly, according to Equations 1 and 2 above, the depth value of the pixel (x+1, y+1) at the (i-1)th layer of the image pyramid can be transformed to the 3D camera coordinate system, obtaining the camera space coordinates View_mip_i-1(x+1, y+1) of the pixel (x+1, y+1) at the (i-1)th layer in the 3D camera coordinate system. View_mip_i-1(x+1, y+1) includes the following three components:

[0178] (1)x component: X4=View_mip_i-1(x+1,y+1).x / View_mip_i-1(x+1,y+1).w;

[0179] (2)y component: Y4=View_mip_i-1(x+1,y+1).y / View_mip_i-1(x+1,y+1).w;

[0180] (3) z component: Z4=View_mip_i-1(x+1,y+1).z / View_mip_i-1(x+1,y+1).w.

[0181] Therefore, the camera space coordinates of the four pixels (x,y), (x+1,y), (x,y+1), and (x+1,y+1) of the (i-1)th level pyramid can be calculated using Equations 1 and 2 above.

[0182] By calculating the average of the camera space coordinates of the four pixels (x,y), (x+1,y), (x,y+1), and (x+1,y+1) at the (i-1)th level of the image pyramid using Equation 3 below, the camera space coordinates of the pixel (x,y) at the i-th level can be obtained.

[0183] View_mip_i(x,y)=(View_mip_i-1(x,y)+View_mip_i-1(x+1,y)+View_mip_i-1(x,y+1)+View_mip_i-1(x+1,y+1)) / 4.0 (Equation 3)

[0184] View_mip_i-1(x,y), View_mip_i-1(x+1,y), View_mip_i-1(x,y+1), and View_mip_i-1(x+1,y+1) represent the camera space coordinates of four pixels (x,y), (x+1,y), (x,y+1), and (x+1,y+1) at the (i-1)th level of the image pyramid in the 3D camera coordinate system, respectively. This can be understood as summing the camera space coordinates of these four pixels and then dividing by 4 to obtain their camera space coordinates.

[0185] View_mip_i(x,y) represents the camera space coordinates of the i-th pixel in the 3D camera coordinate system, obtained by averaging the camera space coordinates of the four pixels at level i-1. View_mip_i(x,y) includes the following three components:

[0186] (1)x component: View_mip_i(x,y).x=(X1+X2+X3+X4) / 4;

[0187] (2)y component: View_mip_i(x,y).y=(Y1+Y2+Y3+Y4) / 4;

[0188] (3) z component: View_mip_i(x,y).z=(Z1+Z2+Z3+Z4) / 4.

[0189] Then, by multiplying the pixel camera space coordinates View_mip_i(x,y) obtained in the 3D camera coordinate system by the projection matrix using Equation 4 below, the NDC coordinates, which serve as an intermediate result, are obtained:

[0190] NDC_mip_i(x,y)=View_mip_i(x,y)*projectionMatrix (Equation 4)

[0191] Here, NDC_mip_i(x,y) represents the NDC coordinates of the pixel at the i-th level of the image pyramid in the 3D camera coordinate system. NDC coordinates are four-dimensional coordinates, including x, y, z, and w components. NDC_mip_i(x,y) includes NDC_mip_i(x,y).x, NDC_mip_i(x,y).y, NDC_mip_i(x,y).z, and NDC_mip_i(x,y).w. NDC_mip_i(x,y).w represents the variable introduced by the homogeneous coordinate transformation.

[0192] Then, by using Equation 5 below and performing homogeneous coordinate transformation, the ratio between NDC_mip_i(x,y).z and NDC_mip_i(x,y).w can be obtained to determine the depth value corresponding to the i-th level of the image pyramid.

[0193] Depth_mip_i(x,y)=NDC_mip_i(x,y).z / NDC_mip_i(x,y).w (Equation 5)

[0194] Where Depth_mip_i(x,y) represents the depth value corresponding to the i-th level of the image pyramid.

[0195] In this embodiment of the application, by using Equations 1 to 5 above, the depth values ​​of the four pixels at the (i-1)th level of the image pyramid are first transformed to the camera coordinate system. Then, the average of the camera space coordinates of these four pixels is calculated in the camera coordinate system to obtain the camera space coordinates at the i-th level of the image pyramid. Then, the camera space coordinates at the i-th level of the image pyramid are transformed to obtain the pixel depth value corresponding to the i-th level of the image pyramid.

[0196] In equations 1 to 5 above, i takes the values ​​1, 2, ..., K in sequence, and the depth value corresponding to each color image in the K-layer image pyramid can be calculated.

[0197] The following example illustrates the process of calculating the depth value corresponding to the color image for each layer of the image pyramid.

[0198] For example, such as Figure 8 As shown, the image pyramid consists of 8 layers. Using a for loop, starting from the first layer, the depth value corresponding to the color image is calculated for each layer of the image pyramid.

[0199] (1) Calculate the depth value corresponding to the first layer of the image pyramid (i is 1).

[0200] First, the color image is transformed from the two-dimensional image coordinate system to the three-dimensional camera coordinate system using the following equation:

[0201] View_mip_0(x,y) = (x,y,Depth_mip_0(x,y),1.0) * Inverse(projectionMatrix).

[0202] View_mip_0(x,y) includes View_mip_0(x,y).x, View_mip_0(x,y).y, and View_mip_0(x,y).z.

[0203] Then, through homogeneous coordinate transformation, calculate the camera space coordinates of the four pixel points in the 0th layer in the three-dimensional camera coordinate system:

[0204] View_mip_0(x,y) =

[0205] (View_mip_0(x,y).x / View_mip_0(x,y).w,

[0206] View_mip_0(x,y).y / View_mip_0(x,y).w,

[0207] View_mip_0(x,y).z / View_mip_0(x,y).w, 1.0)

[0209] View_mip_0(x + 1,y) =

[0210] (View_mip_0(x + 1,y).x / View_mip_0(x + 1,y).w,

[0211] View_mip_0(x + 1,y).y / View_mip_0(x + 1,y).w,

[0212] View_mip_0(x + 1,y).z / View_mip_0(x + 1,y).w, 1.0)

[0214] View_mip_0(x,y + 1) =

[0215] (View_mip_0(x,y + 1).x / View_mip_0(x,y + 1).w,

[0216] View_mip_0(x,y + 1).y / View_mip_0(x,y + 1).w,

[0217] View_mip_0(x,y + 1).z / View_mip_0(x,y + 1).w, 1.0)

[0219] View_mip_0(x+1,y+1)=

[0220] (View_mip_0(x+1,y+1).x / View_mip_0(x+1,y+1).w,

[0221] View_mip_0(x+1,y+1).y / View_mip_0(x+1,y+1).w,

[0222] View_mip_0(x+1,y+1).z / View_mip_0(x+1,y+1).w, 1.0)

[0224] View_mip_0(x,y).w, View_mip_0(x+1,y).w, View_mip_0(x,y+1).w, and View_mip_0(x+1,y+1).w represent the variables introduced by the homogeneous coordinate transformation.

[0225] Then, by averaging the camera space coordinates of the four pixels in the 3D camera coordinate system, the camera space coordinates (x, y) of the pixel in the 1st layer are obtained:

[0226] View_mip_1(x,y)=(View_mip_0(x,y)+View_mip_0(x+1,y)+View_mip_0(x,y+1)+View_mip_0(x+1,y+1)) / 4.0.

[0227] View_mip_1(x,y) contains three components: View_mip_1(x,y).x, View_mip_1(x,y).y, and View_mip_1(x,y).z.

[0228] Then, View_mip_1(x,y) is transformed from the 3D camera coordinate system to the NDC coordinate system using the following equation:

[0229] NDC_mip_1(x,y)=View_mip_1(x,y)*projectionMatrix.

[0230] NDC_mip_1(x,y) includes four components: NDC_mip_1(x,y).x, NDC_mip_1(x,y).y, NDC_mip_1(x,y).z and NDC_mip_1(x,y).w.

[0231] Then, a homogeneous coordinate transformation is performed to calculate the depth value corresponding to the first layer of the image pyramid:

[0232] Depth_mip_1(x,y)=NDC_mip_1(x,y).z / NDC_mip_1(x,y).w

[0233] This allows us to calculate the depth value Depth_mip_1(x,y) corresponding to the first layer of the image pyramid.

[0234] (2) Calculate the depth value corresponding to the second layer of the image pyramid (i is 2).

[0235] First, transform the depth value Depth_mip_1(x,y) corresponding to the first layer of the image pyramid to the 3D camera coordinate system:

[0236] View_mip_1(x,y)=(x,y,Depth_mip_1(x,y),1.0)*Inverse(projectionMatrix).

[0237] View_mip_1(x,y) includes View_mip_1(x,y).x, View_mip_1(x,y).y and View_mip_1(x,y).z.

[0238] Then, through homogeneous coordinate transformation, the camera space coordinates of the four pixels in the first layer are calculated in the 3D camera coordinate system:

[0239] View_mip_1(x,y)=

[0240] (View_mip_1(x,y).x / View_mip_1(x,y).w,

[0241] View_mip_1(x,y).y / View_mip_1(x,y).w,

[0242] View_mip_1(x,y).z / View_mip_1(x,y).w, 1.0)

[0244] View_mip_1(x+1,y)=

[0245] (View_mip_1(x+1,y).x / View_mip_1(x+1,y).w,

[0246] View_mip_1(x+1,y).y / View_mip_1(x+1,y).w,

[0247] View_mip_1(x+1,y).z / View_mip_1(x+1,y).w, 1.0)

[0249] View_mip_1(x,y+1)=

[0250] (View_mip_1(x,y+1).x / View_mip_1(x,y+1).w,

[0251] View_mip_1(x,y+1).y / View_mip_1(x,y+1).w,

[0252] View_mip_1(x,y+1).z / View_mip_1(x,y+1).w, 1.0)

[0254] View_mip_1(x+1,y+1)=

[0255] (View_mip_1(x+1,y+1).x / View_mip_1(x+1,y+1).w,

[0256] View_mip_1(x+1,y+1).y / View_mip_1(x+1,y+1).w,

[0257] View_mip_1(x+1,y+1).z / View_mip_1(x+1,y+1).w, 1.0)

[0259] View_mip_1(x,y).w, View_mip_1(x+1,y).w, View_mip_1(x,y+1).w, and View_mip_1(x+1,y+1).w represent the variables introduced by the homogeneous coordinate transformation.

[0260] Then, by averaging the camera space coordinates of the four pixels in the first layer in the 3D camera coordinate system, the camera space coordinates of one pixel in the second layer image are obtained:

[0261] View_mip_2(x,y)=(View_mip_1(x,y)+View_mip_1(x+1,y)+View_mip_1(x,y+1)+View_mip_1(x+1,y+1)) / 4.0.

[0262] View_mip_2(x,y) contains three components: View_mip_2(x,y).x, View_mip_2(x,y).y, and View_mip_2(x,y).z.

[0263] Then, View_mip_2(x,y) is transformed from the 3D camera coordinate system to the DNC coordinate system using the following equation:

[0264] NDC_mip_2(x,y)=View_mip_2(x,y)*projectionMatrix.

[0265] NDC_mip_2(x,y) includes four components: NDC_mip_2(x,y).x, NDC_mip_2(x,y).y, NDC_mip_2(x,y).z and NDC_mip_2(x,y).w.

[0266] Then, a homogeneous coordinate transformation is performed to calculate the depth value corresponding to the second layer of the image pyramid:

[0267] Depth_mip_2(x,y)=NDC_mip_2(x,y).z / NDC_mip_2(x,y).w

[0268] This allows us to calculate the depth value Depth_mip_2(x,y) corresponding to the second layer of the image pyramid.

[0269] (3) i is successively taken as 3, ..., K, and the depth values ​​corresponding to the 3rd, ..., Kth layers of the image pyramid are calculated.

[0270] Following the above method, the depth values ​​Depth_mip_3(x,y) corresponding to the 3rd layer of the image pyramid, ..., and the depth value Depth_mip_N(x,y) corresponding to the Kth layer can be calculated sequentially.

[0271] Using the method described in the embodiments of this application, the depth value corresponding to each layer of the image pyramid can be calculated. It should be noted that the depth values ​​(or depth information) calculated at different levels can be applied to the corner point calculation steps described below, which will be detailed in detail below.

[0272] Step 102: Calculate the corner points of the pyramid layer.

[0273] It should be noted that corner points can be defined in pixels as the basic unit. A corner point can be defined as a window that will show a large change in response value when moved in any direction.

[0274] In this embodiment, the brightness difference of each pixel in the color image within its surrounding area can be calculated, and points with a brightness difference greater than a threshold can be identified as corner points. It should be noted that, unlike related technologies, this solution not only judges brightness differences based on the color image but also on the depth image, which can improve the accuracy of corner point identification.

[0275] Since corner points are points that differ significantly from surrounding pixels, they can be used as feature points in an image. By finding the corner points in an image, feature points can be identified based on them.

[0276] In this embodiment of the application, the color image and depth map of the corresponding pyramid level can be bound in the computation shader according to specified parameters.

[0277] The specified parameters may include the scaling factor and the image ID.

[0278] For example, specifying the parameters glBindImage(ID=101, α=4) indicates that the image identifier is 101 and the scaling factor α is 4. Based on these specified parameters, the fourth layer of the image pyramid, the color image with the identifier 101, and the corresponding depth image can be bound in the compute shader.

[0279] Optionally, different scaling factors can be pre-set for different types or scenarios of games. For example, for type 1 games, the corresponding scaling factor parameter is 3, indicating that the third layer in the pyramid is selected; for type 2 games, the corresponding scaling factor parameter is 4, indicating that the fourth layer in the pyramid is selected. The specific settings can be configured according to actual usage requirements, and this embodiment does not limit the specific settings. For ease of explanation, this embodiment uses a scaling factor of 4 as an example for illustrative purposes.

[0280] For example, refer to again Figure 8 Assuming the original image is 256×256 pixels, and an 8-layer pyramid is generated, with a scaling factor α = 4, the fourth layer image of the pyramid can be bound in the computation shader. The fourth layer image of the pyramid is 16×16 pixels in size. This application can utilize the image brightness information and depth information (i.e., the depth value corresponding to the fourth layer) of the 16×16 pixel block to determine all corner points within the 16×16 pixel block area.

[0281] The following details the process by which corner points are determined based on image brightness and depth information in the embodiments of this application.

[0282] (1) The process of determining corner points based on image brightness information

[0283] In the GPU's compute shader, image brightness information can include the brightness value of each pixel in a 16×16 pixel block.

[0284] The luminance value of each pixel is calculated in the GPU's compute shader using the following equation 6:

[0285] Intensity = Color.r * 0.299 + Color.g * 0.587 + Color.b * 0.114 (Equation 6)

[0286] Where Color.r, Color.g, and Color.b represent the brightness values ​​of a pixel in the color image (Color) on the R, G, and B channels, respectively. Intensity represents the brightness value of a pixel in the color image (Color). For ease of explanation, the brightness value corresponding to pixel (x,y) is represented as Intensity(x,y).

[0287] In this embodiment of the application, the brightness difference between pixels is calculated in the GPU's computation shader based on the pixel (x,y) and multiple pixels at a preset distance from the pixel (x,y).

[0288] It should be noted that the preset distance is an empirical value obtained from multiple trials and can be determined based on factors such as the application scenario, perspective, and aperture. For example, the preset distance can be 2 pixels.

[0289] For example, Figure 9A The positional relationship between pixel (x,y) and pixels (x-2,y), (x+2,y), (x,y-2), and (x,y+2) is shown.

[0290] As shown in Table 1 below, the brightness values ​​corresponding to the pixels (x-2,y), (x+2,y), (x,y-2) and (x,y+2) are represented as Intensity(x-2,y), Intensity(x+2,y), Intensity(x,y-2), and Intensity(x,y+2), respectively.

[0291] Table 1

[0292] pixel Brightness value (x,y) Intensity(x,y) (x-2,y) Intensity(x-2,y) (x+2,y) Intensity(x+2,y) (x, y-2) Intensity(x, y-2) (x,y+2) Intensity(x, y+2)

[0293] As shown in Table 2 below, calculate the absolute value of the brightness difference between Intensity(x,y) and Intensity(x-2,y), the absolute value of the brightness difference between Intensity(x,y) and Intensity(x+2,y), the absolute value of the brightness difference between Intensity(x,y) and Intensity(x,y-2), and the absolute value of the brightness difference between Intensity(x,y) and Intensity(x,y+2).

[0294] Table 2

[0295] Two pixels absolute value of the difference in brightness (x,y) and (x-2,y) ∣Intensity(x,y)-Intensity(x-2,y)∣ (x,y) and (x+2,y) ∣Intensity(x,y)-Intensity(x+2,y)∣ (x,y) and (x,y-2) ∣Intensity(x,y)-Intensity(x,y-2)∣ (x,y) and (x,y+2) ∣Intensity(x,y)-Intensity(x,y+2)∣

[0296] Then, in the GPU's compute shader, it is determined whether the absolute value of the difference in pixel brightness is greater than or equal to a preset brightness threshold, and the number of pixels whose absolute value of brightness difference is greater than or equal to the preset brightness threshold is counted.

[0297] If the count is greater than or equal to the preset count (e.g., 3), the initial selection condition is met. Then, it is determined that the current pixel may be a corner point, and the next step is executed to further determine whether the current pixel is a corner point.

[0298] If the count is less than the preset number, then it is determined that the current pixel cannot be a corner point, and processing of the current pixel is stopped.

[0299] It should be noted that, through step (1), the GPU's computation shader determines that some pixels in the image cannot be corners, so these pixels can be removed. Then, the GPU's computation shader can further judge the remaining pixels that may be corners based on the depth information, so as to reduce the amount of computation and improve the recognition accuracy.

[0300] (2) The process of determining corner points based on depth information

[0301] The depth information is the depth value corresponding to the selected layer of the image pyramid calculated in step S101 above.

[0302] The depth information corresponding to the depth image DepthMipMap can be Depth_mip_i(x,y). For example, i is 4. In this application, a 16×16 pixel block of the 4th layer of the image pyramid is used, and the depth information corresponding to the depth image DepthMipMap can be Depth_mip_4(x,y).

[0303] For the pixels that meet the initial selection criteria identified in the previous step, the depth information (Depth_mip_x, Depth_mip_y, and Depth_mip_z) corresponding to the DepthMipMap is sampled based on the pixel's image coordinates (x, y). The sampling result is then multiplied by the inverse of the projection matrix to calculate the pixel's spatial coordinates in the camera coordinate system.

[0304] Pos(x,y,z)=[(x,y)*2.0-1.0,Depth_mip_4(x,y),1.0]*Inverse(projectionMatrix) (Equation 7)

[0305] Here, projectionMatrix represents the projection matrix, and Inverse() represents finding the inverse matrix.

[0306] Pos(x,y,z) consists of three components: Pos(x,y,z).x, Pos(x,y,z).y, and Pos(x,y,z).z.

[0307] Then, through homogeneous coordinate transformation, the spatial coordinates of Pos(x,y,z) can be expressed as:

[0308] Pos(x,y,z)=(Pos(x,y,z).x / Pos(x,y,z).w,Pos(x,y,z).y / Pos(x,y,z).w,Pos(x,y,z).z / Pos(x,y,z).w) (Equation 8)

[0309] Where Pos(x,y,z).w represents the variable introduced by the homogeneous coordinate transformation.

[0310] Then, the spatial coordinates Pos(x,y,z) of the pixel are converted into two-dimensional image coordinates Pos(x,y). Based on the depth information of the pixel and the coordinate transformation, the two-dimensional image coordinates Pos(x,y) of the pixel are calculated.

[0311] Based on the depth information and coordinate transformation of the pixel, the two-dimensional image coordinates Pos(x-2,y) of the pixel (x-2,y) are then calculated.

[0312] Based on the depth information and coordinate transformation of the pixel, the two-dimensional image coordinates Pos(x+2,y) of the pixel (x+2,y) are then calculated.

[0313] Based on the depth information and coordinate transformation of the pixel, the two-dimensional image coordinates Pos(x,y-2) of the pixel (x,y-2) are then calculated.

[0314] Based on the depth information and coordinate transformation of the pixel, the two-dimensional image coordinates Pos(x,y+2) of the pixel (x,y+2) are then calculated.

[0315] Then, calculate the absolute value of the difference in pixel brightness between pixels Pos(x,y) and Pos(x-2,y), Pos(x,y) and Pos(x+2,y), Pos(x,y) and Pos(x,y-2), and Pos(x,y) and Pos(x,y+2).

[0316] Next, following an algorithm similar to step (1) above, the absolute value of the difference in pixel brightness is determined in the GPU's compute shader to be greater than or equal to a preset distance threshold, and the number of pixels whose absolute difference is greater than or equal to the preset pixel threshold is counted. If the count is greater than or equal to the preset number (e.g., 3), the condition is met, and the pixel can be determined to be a corner point. If the count is less than the preset number (e.g., 3), the current pixel cannot be a corner point, and processing of the current pixel is stopped.

[0317] In this embodiment of the application, for each pixel in the 16×16 pixel block area in the GPU's computation shader, according to the above steps (1), it is determined in the GPU's computation shader that some pixels in the image cannot be corner points based on the image brightness information, and these pixels are removed. Then, according to steps (2), the remaining pixels that may be corner points are further judged based on the image depth information, thereby determining all corner pixels in the 16×16 pixel block area.

[0318] Step 103: Calculate the response function values ​​of corner pixels.

[0319] The following details the process of calculating the response function value (or response value) for the corner pixels identified in step 102 in the GPU's computational shader.

[0320] In the embodiments of this application, the response function value of each corner pixel is calculated based on the brightness difference and depth difference between each corner pixel (x,y) and its four adjacent pixels (x-1,y), (x+1,y), (x,y-1) and (x,y+1). Figure 9B The positional relationship between the corner pixel (x,y) and its four neighboring pixels (x-1,y), (x+1,y), (x,y-1), and (x,y+1) is shown.

[0321] (1) Calculate the brightness values ​​of the four neighboring pixels (x-1,y), (x+1,y), (x,y-1) and (x,y+1) around the corner pixel (x,y), and calculate the depth values ​​of the four neighboring pixels around the corner pixel.

[0322] The brightness values ​​of the two adjacent pixels in the x-direction of the corner pixel (x,y): Intensity(x-1,y), Intensity(x+1,y).

[0323] The brightness values ​​of the two adjacent pixels in the y-direction of the corner pixel (x,y): Intensity(x,y-1), Intensity(x,y+1).

[0324] The depth values ​​of the corner pixel (x,y) between two adjacent pixels in the x-direction: Depth_mip(x-1,y), Depth_mip(x+1,y).

[0325] The depth values ​​of the two adjacent pixels in the y direction of the corner pixel (x,y): Depth_mip(x,y-1), Depth_mip(x,y+1).

[0326] (2) Based on the brightness value calculated in (1) above, calculate the brightness difference of the corner pixel (x,y) in the x and y directions.

[0327] The brightness difference of a corner pixel (x,y) in the x-direction: Ix = Intensity(x+1,y) - Intensity(x-1,y);

[0328] The brightness difference of the corner pixel (x,y) in the y direction: Iy = Intensity(x,y+1) - Intensity(x,y-1).

[0329] (3) Based on the depth value calculated in (1) above, calculate the depth difference between the x and y directions of the corner pixel (x,y).

[0330] Depth difference of corner pixel (x,y) in the x direction: dx = Depth_mip(x+1,y) - Depth_mip(x-1,y);

[0331] Depth difference of corner pixel (x,y) in the y direction: dy = Depth_mip(x,y+1) - Depth_mip(x,y-1).

[0332] (4) Using the Ix, Iy, dx and dy calculated in (3) above, calculate the response function value (denoted as response) of the corner pixel (x,y) using the exponential function.

[0333] The response function value of the corner pixel (x, y): response = exp(Ix, Iy, dx, dy) = e (Ix,Iy,dx,dy) (Equation 9)

[0334] In this embodiment of the application, the response function value can be calculated for each corner point within a 16×16 pixel block area, thereby calculating the response function values ​​of all corner pixels within the 16×16 pixel block area.

[0335] It should be noted that, in related technologies, the response function value of a pixel is calculated only by calculating the brightness difference between adjacent pixels. In contrast, in this embodiment, not only is the brightness difference between adjacent pixels calculated on the GPU side, but the depth difference between adjacent pixels is also calculated on the GPU side. The response function value of each pixel is then calculated on the GPU side based on the brightness difference and the depth difference between adjacent pixels, thereby improving the accuracy of the calculated response function value.

[0336] In this embodiment, the response function value (response) of each corner pixel can be compared with a preset response threshold to determine whether the corner pixel is a feature point. If the response function value (response) is greater than or equal to the preset response threshold, then the current pixel is a feature point, and step 4 continues. If the response function value (response) is less than the preset response threshold, then the current pixel is not a feature point, and processing of the current pixel stops.

[0337] In this way, by selecting some pixels from the corner pixels as feature points based on the response function values ​​of the corner pixels, the amount of computation can be reduced and the accuracy of computation can be improved.

[0338] It should be noted that by filtering feature points in each 16×16 pixel block region, the following results are obtained: some 16×16 pixel block regions contain one or more feature points, while some 16×16 pixel block regions do not contain any feature points.

[0339] Therefore, step 103 can determine all feature points and their response function values ​​within each 16×16 pixel block region.

[0340] Step 104: Maximum suppression of feature points

[0341] Maximum suppression refers to searching for local maxima and eliminating similar feature points that cluster together locally.

[0342] Since there may be locally clustered similar feature points within the 16×16 pixel block area determined by step 103, and these similar feature points are redundant, it is necessary to remove these similar feature points.

[0343] It should be noted that, unlike related technologies, which use OpenCV to execute program code on the CPU to achieve maximum suppression, this application uses shared cache variables to achieve maximum suppression.

[0344] In this embodiment, the shared cache variable of the thread workgroup of the compute shader can be used to take the maximum value of the response function of all feature points in each 16×16 pixel block area, so as to remove locally clustered similar feature points.

[0345] refer to Figure 10 In this embodiment, assuming the original image size is W×H, the computation shader can employ W / 16×H / 16 thread workgroups (or local workgroups). Each thread workgroup comprises 16×16 threads, and each thread corresponds to a feature point pixel.

[0346] In this embodiment, each thread workgroup defines two shared variables for the shared cache, referred to as shared variable 1 and shared variable 2. Shared variable 1 is used to store the maximum response value max_response in the current workgroup, and shared variable 2 is used to store the pixel coordinate max_response_index corresponding to the maximum response value.

[0347] Referring to Table 3 below, assuming W×H is 256×256, the compute shader uses W / 16×H / 16 = 256 thread workgroups, each consisting of 16×16 threads. Each thread workgroup defines two shared variables for a shared cache. For each thread workgroup, the maximum response value among the 16×16 threads is determined and stored in shared variable 1, while the pixel coordinates corresponding to the maximum response value are stored in shared variable 2.

[0348] Table 3

[0349]

[0350] In this embodiment, for each thread workgroup, each feature point pixel calls the atomic Max operation of the compute shader to calculate the maximum value between its own response value and max_response, and assigns the calculated maximum value to max_response. Furthermore, if the maximum response value max_response equals its own response value, then its own image coordinates are assigned to max_response_index.

[0351] For example, suppose thread workgroup 1 has 16×16 threads, each thread corresponding to a feature point pixel. Let's take three feature points as an example. For instance, the response values ​​for the three feature points are 1, 2, and 1.5. The response value `response` for each feature point is compared with `max_response`, and the maximum value is assigned to `max_response`. The maximum response value `max_response` stored in shared variable 1 is initialized to 0.

[0352] First, calling the atomic operation atomicMax(1, 0) = 1 in the compute shader determines that the maximum value between the response value 1 and the initial value 0 is 1. The maximum value 1 is then assigned to max_response, i.e., the maximum response value max_response = 1.

[0353] Then, by calling the atomic operation atomicMax(2, 1) = 2 in the compute shader, the maximum value is determined to be 2. The maximum value 2 is then assigned to max_response, that is, the maximum response value max_response = 2.

[0354] Then, by calling the atomic operation atomicMax(1.5, 2) = 2 in the compute shader, the maximum value of 2 can be determined. The maximum value of 2 is then assigned to max_response, that is, the maximum response value max_response = 2.

[0355] Therefore, the local maximum response value can be determined. For thread workgroup 1, shared variable 1 stores the maximum response value in thread workgroup 1, and shared variable 2 stores the pixel coordinates corresponding to the maximum response value.

[0356] Through step 4 above, the shared cache variable of the thread workgroup of the computation shader is used to take the maximum value of the response function value of all feature points in each 16×16 pixel block area, so as to eliminate locally clustered similar feature points.

[0357] For example, refer to again Figure 10 For W / 16×H / 16 thread workgroups (also called local workgroups), step 104 determines the local maximum response value of the 16×16 threads in each thread workgroup, thus determining W / 16×H / 16 local maximum response values. The shared cache variables of the thread workgroups record the W / 16×H / 16 local maximum response values ​​and their corresponding pixel coordinates.

[0358] Step 105: Output feature point information

[0359] Based on the shared cache variables of the thread workgroup, which record multiple local maximum response values ​​and their corresponding pixel coordinates, the multiple local maximum response values ​​are written to the pixel coordinates in the mask image.

[0360] In this embodiment, the G channel of each pixel in the DepthMipMap is initialized to 0.0. The G channel of a pixel can be used to store the maximum response value max_response of the thread workgroup. It should be noted that the G channel is used as an example here. In actual implementation, the R channel or B channel can also be used. The specific setting can be determined according to the usage requirements, and this embodiment does not limit it.

[0361] For each feature point, check the index of the feature point pixel in the thread workgroup: gl_LocalInvocationIndex. If the index of feature point pixel 1 in thread workgroup 1 is gl_LocalInvocationIndex = 0, then the maximum response value max_response of thread workgroup 1 is written to the G channel at the max_response_index position of the depth image DepthMipMap, thereby generating image feature point information. This completes the recognition of image feature points. It should be noted that this example uses a thread position index of 0; in actual implementations, the thread position index can be set to other values.

[0362] In this embodiment, the depth image with the maximum response value written into it is used as the mask image.

[0363] For example, such as Figure 11 As shown, by performing feature point recognition on the color image, five local maximum response values ​​are determined for the color image, namely feature point 1 to feature point 5. The response values ​​and pixel coordinates of these five feature points are stored in the shared cache. Then, the response values ​​of these five feature points are written to the corresponding pixel coordinates of the mask image to obtain the mask image.

[0364] The mask image is the result of feature point recognition on the color image. All the identified feature points are marked in the mask image, and the response value of the feature point is marked at the pixel coordinates of each feature point.

[0365] It should be noted that the result of feature point recognition is simply illustrated here. It can be understood that in actual implementation, more or fewer feature points can be identified for a single frame of image. The specific number can be determined according to the actual situation, and this application embodiment does not limit it.

[0366] The proposed solution directly utilizes GPU-specific data to perform feature point recognition operations within the rendering pipeline in graphics rendering applications, significantly improving recognition efficiency and reducing computational power consumption while ensuring recognition accuracy.

[0367] The above describes the process of image feature point recognition using a GPU provided in this application, with specific embodiments, and the results of feature point recognition are used for feature point matching. The following describes the GPU-based image feature matching method provided in this application, with specific embodiments.

[0368] To better understand the embodiments of this application, feature point recognition and matching will first be described in conjunction with the accompanying drawings.

[0369] After feature point recognition is performed on each color image, a recognition result, i.e., a mask image, is obtained for each color image. A mask image is generated for each color image, and the mask image is marked with feature points and response values. For different color images, feature point matching is performed using the corresponding mask image. For ease of explanation, feature point recognition and matching of the first color image and the second color image will be used as an example for illustration.

[0370] For example, refer to Figure 12A and Figure 12B Feature point recognition is performed on the first color image and the second color image respectively to obtain the first mask image and the second mask image. Then, the marked feature points in the two mask images are compared to determine the matching feature point pairs in the two images.

[0371] Specifically, the response values ​​of feature points are marked in the first mask image. All feature points in the first mask image are sorted according to the magnitude of their response values, and the N largest feature points are selected as the first feature point sequence. Then, a feature vector is calculated for each feature point in the first feature point sequence.

[0372] Similarly, the response values ​​of feature points are marked in the second mask image. All feature points in the second mask image are sorted according to the magnitude of their response values, and the N largest feature points are selected as the second feature point sequence. A feature vector is calculated for each feature point in the second feature point sequence.

[0373] Then, the distance between the feature vector of each feature point in the first feature point sequence and the feature vector of all points in the second feature point sequence is calculated. The two feature points with the smallest distance are the best matching points, thus obtaining a pair of corresponding feature point pairs [(x0,y0),(x1,y1)], where (x0,y0) are the image coordinates of the feature points in the first color image, and (x1,y1) are the image coordinates of the feature points in the second color image that match (x0,y0).

[0374] pass Figure 12A and Figure 12B As can be seen, in this embodiment of the application, feature point recognition is performed on two frames of images respectively, the feature points of each frame of images are identified, and then feature point matching is performed on the feature points of each frame of images to determine the matching feature point pairs in the two frames of images.

[0375] The following is a brief description of the image feature point matching process provided in the embodiments of this application. For example... Figure 13 As shown in this embodiment, the process of image feature point matching by the electronic device through the GPU may include the following steps 201-205:

[0376] Step 201: Calculate random vectors using GPU.

[0377] Specifically, the first mask image corresponding to the first color image and the second mask image corresponding to the second color image are both loaded into the GPU's computation shader. Random numbers are calculated on the GPU using the image coordinates of the pixels, generating 256 4D integer random constants.

[0378] It should be noted that the mask image is obtained by performing feature point recognition on a color image (which can also be called a real frame image). The mask image marks the feature points and their response values. For example, refer to the reference... Figure 11 In the mask image, the black areas are non-feature points, and the white areas are feature points. Each feature point is labeled with its response value.

[0379] Step 202: Sort the feature points using the GPU.

[0380] The first and second color images are loaded into the GPU's computational shader. Based on the marked feature points and their response values ​​in the mask image, the feature points are sorted on the GPU according to their response values, and the top N feature points with the largest response values ​​are selected.

[0381] Step 203: Calculate the feature vector for each feature point using the GPU.

[0382] Specifically, the global workgroup size of the GPU's compute shader is set to 16×16 (i.e., 256). Utilizing the shared cache and atomic operations between thread groups of the GPU's compute shader, and combined with the 256 random constants of ivec4 generated in step 1, the feature vectors of the selected N feature points are calculated.

[0383] Step 204: Calculate the feature distance between feature vectors in two frames of images using the GPU.

[0384] Specifically, the feature distance between each pair of N feature vectors of two frames is calculated, and the minimum feature distance is calculated.

[0385] Step 205: Perform feature matching based on feature distance using the GPU.

[0386] Specifically, if the minimum feature distance is less than a preset distance threshold, then the feature points belonging to the two feature vectors with that minimum feature distance are successfully matched, and the two successfully matched feature points form a feature point pair. If the minimum feature distance is greater than or equal to the preset distance threshold, then the matching fails. This completes the matching of image feature points.

[0387] This application proposes a technique for real-time and low-power image feature matching in mobile devices using GPUs. By introducing GPU graphics rendering units and graphics rendering pipelines, especially GPU-specific features such as compute shaders, inter-core shared caches, and atomic operations of thread workgroups, feature points of different images in the graphics rendering pipeline are matched, significantly improving computational efficiency and reducing the time overhead of pixel-by-pixel calculations. At the same time, it reduces the overhead of data copying and context switching between feature matching and the graphics rendering pipeline.

[0388] In this embodiment, during the matching of image feature points generated by graphics rendering, image resources are accessed directly within the graphics rendering pipeline, eliminating the memory copying operation between the GPU and CPU. This significantly reduces the heat generated by mobile devices due to memory bandwidth overhead. At the same time, by utilizing the high-concurrency multi-threaded processing capabilities provided by the GPU and the atomic operation of threads to access the shared cache between cores, the computational efficiency of feature point matching can be greatly accelerated, achieving the goal of high frame rate applications.

[0389] The above briefly describes the image feature point matching process provided in the embodiments of this application. The following describes in detail the specific implementation of the GPU-based image feature matching method provided in the embodiments of this application with reference to the accompanying drawings.

[0390] Step 201: Calculate the random vector

[0391] In this embodiment, a readable and writable SSBO of size 256 that supports ivec4 vectors is created. An ivec4 vector represents a four-component vector of signed integers. For ease of explanation, this SSBO is denoted as SSBO1.

[0392] First, it should be noted that after feature point recognition of the color image, a mask image is obtained. This mask image marks the image feature point pixels and their corresponding response function values. Specifically, the first color image corresponds to the first mask image, and the second color image corresponds to the second mask image.

[0393] In the step of calculating random numbers, the first mask image and the second mask image are loaded into the GPU's computation shader, such as... Figure 14A As shown, the size of the local workgroup for the specified computation pipeline is 16×16, and the size of the global workgroup is (1,1,1). That is, the global workgroup contains only one local workgroup, with 16×16 threads forming one local workgroup. Running 16×16 threads on the GPU results in 256 threads.

[0394] In this embodiment, a random number seed is generated in the compute shader. For example, the random number seed (denoted as r) is obtained by multiplying the position index (gl_LocalInvocationIndex) value by the scaling factor (scale) of the random number specified by the parameter using the following Equation 10:

[0395] r = gl_LocalInvocationIndex × scale (Equation 10)

[0396] In this embodiment, each thread in the working group uses trigonometric and exponential functions to perform operations on the random number seed r to generate a 4-dimensional random vector (denoted as R).

[0397] R = e sin(r) (Equation 11)

[0398] Each vector in the 4-dimensional random vector is in the range [0.0, 1.0].

[0399] It should be noted that, through calculation, a total of 256 4-dimensional random vectors are generated in this embodiment of the application.

[0400] In this embodiment, each 4-dimensional random vector (R) is adjusted according to an offset parameter. For example, the random number can be adjusted using the following Equation 12:

[0401] R' = R * 20 * offset (Equation 12)

[0402] It should be noted that in Equation 12, 20 is a value that can be set according to the actual situation, that is, the sampling size is limited to 20×20. The offset parameter offset can be used to scale the sampling size based on 20×20 so that the scaled sampling size meets the requirements of sampling the image to be processed.

[0403] For example, assuming the local workgroup size used for the image to be processed is 32×32, that is, the image patch size is 32×32, then the offset parameter can take values ​​in the range of [1.5, 2.0]. As another example, assuming the local workgroup size used for the image to be processed is 16×16, that is, the image patch size is 16×16, then the offset parameter can take values ​​in the range of [0.25, 0.5].

[0404] In actual implementation, R' = (R * 20 * offset) may be in decimal form. To facilitate calculation, the result of the operation on R' can be rounded down to the integer part. By adjusting, the 4-dimensional random vector is made up of natural numbers.

[0405] For ease of explanation, the 4D integer vector random number can be denoted as [A, B, C, D]. Here, A and B represent the first random offset, and C and D represent the second random offset. This 4D integer vector random number can be used to calculate the eigenvectors of feature points in the following steps; the specific calculation process will be described in detail below.

[0406] For example, the adjusted 4-dimensional random vector [A, B, C, D] = [2, 2, 3, 5].

[0407] In this embodiment of the application, all 256 4-dimensional random vectors are adjusted to obtain 256 adjusted 4-dimensional random vectors, which constitute a random number queue.

[0408] like Figure 14B As shown in this embodiment, 256 adjusted 4D random vectors can be cached in SSBO1.

[0409] In summary, this application loads two mask images into the GPU's computation shader, generates a random number seed for each mask image in the computation shader, and then uses trigonometric and exponential functions to operate on the random number seed to generate 256 four-dimensional vector random numbers. The four-dimensional vector random numbers are then scaled and adjusted, and the adjusted 256 four-dimensional integer vector random numbers [A, B, C, D] are combined to form a random number queue.

[0410] It should be noted that when calculating the feature vectors of feature points later, each thread in the local working group can search for the corresponding random number in the random number queue based on its own index (gl_LocalInvocationIndex) and obtain the corresponding random offset. Then, it uses the thread coordinates and random offset to sample the color image, obtain the RGB color value of the color image, calculate the brightness value of the color image based on the RGB color value of the color image, and then determine the feature vector of the feature point based on the brightness value of the color image.

[0411] Step 202: Feature Point Ranking

[0412] First, it should be noted that the mask image obtained by recognizing image feature points has a certain width and height. For ease of explanation, let's denote the width of the mask image as W and the height as H.

[0413] In this embodiment, an SSBO with a vec4 vector of size N×(W / 16)×(H / 16) is created. For ease of explanation, this SSBO is denoted as SSBO 2. Here, the vec4 vector refers to a 4-component floating-point vector.

[0414] SSBO1 can include (W / 16)×(H / 16) segments, and each segment can store N data items.

[0415] In this embodiment, SSBO 2 can be used to segment and store the feature point data of each of (W / 16)×(H / 16) local workgroups, where each local workgroup is a 16×16 image block. Since the maximum number of feature points stored in a 16×16 image block is 16×16, N is less than or equal to 256.

[0416] It should be noted that N can be 8 or 12, and the specific value can be set according to actual usage requirements. This application embodiment does not limit this. For ease of explanation, this application embodiment uses N=8 as an example for illustrative purposes. When N=8, 0≤n≤8.

[0417] For example, assume the mask image size is 256×256, that is, the width W and height H are both 256 pixels. As described above, in this embodiment, 16×16=256 threads are treated as a local workgroup, so for a mask image of size 256×256, 256 local workgroups can be divided.

[0418] Accordingly, the size of SSBO 2 can be:

[0419] N×(W / 16)×(H / 16)=N×(256 / 16)×(256 / 16)=256×N (Equation 13)

[0420] In other words, SSBO 2 can store data from 256 local workgroups in segments, with each segment storing N data items.

[0421] In the feature point sorting step, two frames of mask images are loaded into the computation shader. The size of the local workgroup in the computation pipeline is specified as 16×16, and the size of the global workgroup is (W / 16, H / 16, 1), meaning the global workgroup includes W / 16*H / 16 local workgroups. In other words, the global workgroup contains W / 16×H / 16 local workgroups, with 16×16 threads forming one local workgroup. On the GPU, 16×16 threads are enabled, resulting in 256 threads.

[0422] For example, assuming the width W and height H of the mask image are both 256 pixels, then as follows Figure 15 As shown, the global workgroup size of the computational pipeline of the computational shader is (16, 16, 1), and the local workgroup size is 16×16.

[0423] In this embodiment, feature point sorting is achieved using a shared cache mechanism among thread workgroups of the computation shader. For example, a shared cache of size N for the vec4 vector can be defined. The shared cache can store data for N feature points.

[0424] In the embodiments of this application, reference is made to Figure 16 The global working group comprises W / 16 × H / 16 local working groups. Feature points are extracted from each local working group within the global working group. A local working group may contain one or more feature points, or it may not contain any feature points. To improve computational speed, the number of feature points n in each local working group is controlled to be at most N, meaning the number of feature points n in each local working group is within the range [0, N].

[0425] Specifically, refer to Figure 17 For each feature point in the local workgroup, the response value of the feature point can be extracted. The atomic operation (atomicMax) is used to count the n feature points with the largest response values ​​within the current local pixel region, and these n feature points are arranged in descending order. Then, the data of these n feature points is stored in a shared cache. Specifically, in the thread with position index (gl_LocalInvocationIndex) 0, the data of the n feature points in the shared cache is copied to the corresponding position in SSBO 2. It should be noted that this example uses the thread with index 0; in actual implementation, threads with other index values ​​can also be selected, as long as it is ensured that only one thread in the thread workgroup is selected to execute subsequent operations each time, avoiding errors caused by multiple threads executing simultaneously.

[0426] This application embodiment determines the N feature points with the largest response values ​​on the entire image by segmenting and sorting and storing them in segments.

[0427] In one implementation, the entire image is divided into multiple local pixel regions. The N feature points with the highest response values ​​within each local pixel region are sorted and then stored in multiple segments of the SSBO 2 algorithm. (Reference) Figure 18A Sort all feature points in all local pixel regions stored in multiple segments of SSBO 2, and select the N feature points with the largest response values ​​in the whole image.

[0428] In another implementation, the entire image is divided into multiple local pixel regions. The N feature points with the largest response values ​​within each local pixel region are sorted and stored in multiple segments of SSBO 2. Then, the feature points within each local pixel region are sorted from largest to smallest response value. A single feature point with the largest local response value is selected from each local pixel region, and all selected feature points with the largest local response value are sorted. This process yields the N feature points with the largest local response values ​​in the entire image.

[0429] For example, refer to Figure 18B In SSBO 2, (W / 16) × (H / 16) subarrays can be stored, each with a length of N. Each subarray stores N feature point data points, which are sorted in descending order of response value. That is, the first position of each subarray stores the largest response value among the N feature points, i.e., the local maximum. For (W / 16) × (H / 16) subarrays, (W / 16) × (H / 16) local maxima can be extracted. Then, these (W / 16) × (H / 16) local maxima are sorted in descending order, and the N feature points with the largest global response values ​​are selected from them as the N feature points with the largest response values ​​in the entire image.

[0430] For example, assuming N is 8, that is, the length of each subarray is 8, if there are 3 feature points, then these 3 feature points can be sorted in descending order of response value and stored in a subarray of SSBO 2, and the other 5 positions of the subarray are written with 0.

[0431] refer to Figure 19 After feature point recognition of the image, a large number of feature points are identified. In order to improve the efficiency of feature point matching, this application embodiment adopts the feature point segmentation sorting method to filter out the N largest feature points globally. That is, by sorting the feature points, only the N feature points with the largest response values ​​are retained. Subsequently, the feature vectors of the N feature points with the largest response values ​​will be calculated.

[0432] Using the feature point segmentation and sorting method provided in this application, feature points are sorted for the first mask image and the second mask image respectively, resulting in two sets of feature point sequences: the first feature point sequence of the first mask image and the second feature point sequence of the second mask image. For example, N can be 8, meaning the first feature point sequence includes the 8 feature points with the largest response values ​​on the first mask image; the second feature point sequence includes the 8 feature points with the largest response values ​​on the second mask image.

[0433] Step 203: Calculate the eigenvectors of the feature points.

[0434] For the first feature point sequence and the second feature point sequence generated in step 202, calculate the feature vector (specialVector) of each feature point in the two feature point sequences.

[0435] refer to Figure 20 After segmenting and sorting a large number of feature points in the mask image, the N (e.g., 8) feature points with the largest response values ​​are retained in the mask image. Then, a feature vector is calculated for each feature point: first, two pixels offset relative to the feature point by a random offset are determined, and then the feature vector of the feature point is calculated based on the brightness difference between these two pixels.

[0436] The process of calculating the eigenvector of each feature point will be explained in detail below.

[0437] In this embodiment, the first color image and the second color image are loaded into the computation shader. The size of the local workgroup of the computation shader's computation pipeline is specified as 16×16, and the size of the global workgroup is (1, 1, 1). That is, the global workgroup contains only one local workgroup, and 16×16 threads form one local workgroup. 16×16 threads are started on the GPU, resulting in 256 threads.

[0438] In the GPU's compute shader, each thread in the local workgroup uses its own position index (gl_LocalInvocationIndex) as an index to search for the 4D random vector corresponding to that index among the 256 4D random vectors cached in SSBO 1, which is then used as the random offset (offsetUV_i).

[0439] The random offset (offsetUV_i) can be represented as a 4-dimensional random vector [A, B, C, D]. A and B represent the first random offset, which can be denoted as offsetUV_i1; C and D represent the second random offset, which can be denoted as offsetUV_i2.

[0440] It should be noted that the reason for using a 4-dimensional random vector in this embodiment is that two 2-dimensional random vectors are needed for address offset. That is, when calculating the feature vector of a pixel, it is necessary to take two random neighboring pixels at the current pixel's position. If only one random neighboring pixel is taken, then random offsets in the horizontal and vertical directions are required, necessitating a 2-dimensional random vector. Furthermore, taking two random neighboring pixels requires two 2-dimensional random vectors, which together form a 4-dimensional random vector.

[0441] For example, refer to Figure 21 Assume a 4-dimensional random vector is [2, 2, 3, 5]. Here, [2, 2] represents the first random offset, and [3, 5] represents the second random offset. The feature point is shifted first by [2, 2], resulting in pixel coordinates of (x+2, y+2). The feature point is then shifted second by [3, 4], resulting in pixel coordinates of (x+3, y+5).

[0442] In this embodiment, for each thread, two random neighbor pixels at the current feature point pixel position are determined, and the color values ​​of these two random neighbor pixels are sampled. Then, the brightness of each random neighbor pixel is calculated based on its color value. Finally, the brightness of the two random neighbor pixels is compared, and the feature vector of the feature point is obtained based on the comparison result. The process of calculating the feature vector of the feature point will be described in detail below.

[0443] First, the first color image (Color1) is sampled using the coordinates (gl_GlobalInvocationID.xy) of each thread and a random offset UV_i1 to obtain the first RGB color value of the pixel corresponding to the thread:

[0444] Color1_i.rgba=texture(Color1,gl_GlobalInvocationID.xy+offsetUV_i1) (Equation 14)

[0445] The texture() function returns a vector of type vec4, representing the color value of the texture at the specified location.

[0446] The above Color1_i.rgba represents the first RGB color value of the pixel. The first RGB color value is a 4-dimensional vector: Color1_i.r, Color1_i.g, Color1_i.b, and Color1_i.a.

[0447] Similarly, the first color image (Color1) is sampled using each thread's own coordinates (gl_GlobalInvocationID.xy) and a random offset UV_i2 to obtain the second RGB color value of the pixel corresponding to the thread:

[0448] Color1_i+1.rgba=texture(Color1,gl_GlobalInvocationID.xy+offsetUV_i2) (Equation 15)

[0449] The above Color1_i+1.rgba represents the second RGB color value of the pixel. The second RGB color value is a 4-dimensional vector: Color1_i+1.r, Color1_i+1.g, Color1_i+1.b, and Color1_i+1.a.

[0450] Then, the first luminance value Intensity_i is calculated based on the first RGB color value of the pixel:

[0451] Intensity_i=Color1_i.r*0.299+Color1_i.g*0.587+Color1_i.b*0.114 (Equation 16)

[0452] Similarly, the second brightness value Intensity_i+1 is calculated based on the second RGB color value of the pixel:

[0453] Intensity_i+1=Color1_i+1.r*0.299+Color1_i+1.g*0.587+Color1_i+1.b*0.114 (Equation 17)

[0454] Then, each thread uses the atomicOr() operation of the thread workgroup to calculate the relationship between the first brightness value Intensity_i and the second brightness value Intensity_i+1 corresponding to the thread, and obtains the calculation result:

[0455] result=atomicOr(Intensity_i, Intensity_i+1) (Equation 18)

[0456] For example, if Intensity_i+1 is greater than Intensity_i, then the result is 1. If Intensity_i+1 is less than or equal to Intensity_i, then the result is 0.

[0457] In this embodiment of the application, the above calculations are performed on 16×16 threads respectively, resulting in 16×16 results.

[0458] Then, the 16×16 results are output to the gl_LocalInvocationIndex position of the feature vector (specialVector).

[0459] In this embodiment, multiple results can be written simultaneously using the Atomic() function. The Atomic() function can be specifically represented as follows:

[0460] Atomic(specialVector[gl_LocalInvocationIndex],result)

[0461] The `Atomic()` function allows for simultaneous writing of multiple results. This avoids read / write conflicts caused by read / write locks in the GPU.

[0462] In this embodiment of the application, after all threads have completed their calculations, a feature vector of a feature point is obtained.

[0463] In this way, a feature vector is calculated for each of the N feature points in the first color image (Color1), resulting in N feature vectors. Each feature vector can be a 256-bit sequence of 0s or 1s.

[0464] It should be noted that the above explanation is based on the example of calculating the feature vector of each feature point in the first feature point sequence for the first color image (Color1). The process of calculating the feature vector of each feature point in the second feature point sequence for the second color image (Color2) is similar and will not be repeated here. Therefore, the algorithm provided in this application embodiment can calculate the feature vector of each feature point in the first feature point sequence and the feature vector of each feature point in the second feature point sequence.

[0465] For example, referring to Table 4 below, the first feature point sequence includes the N feature points with the largest response values ​​in the first color image. Each of the N feature points corresponds to a feature vector, and the feature vector corresponding to each feature point is a sequence of 256 bits of 0s or 1s. Similarly, the second feature point sequence includes the N feature points with the largest response values ​​in the second color image. Each of the N feature points corresponds to a feature vector, and the feature vector corresponding to each feature point is a sequence of 256 bits of 0s or 1s.

[0466] Table 4

[0467]

[0468] Step 204: Calculate the feature distance between feature vectors

[0469] In step 202 above, a first feature point sequence (denoted as KeyPoint1) is generated for one frame of image and a second feature point sequence (denoted as KeyPoint2) is generated for another frame of image. In step 203, the feature vector of each feature point in the first feature point sequence and the feature vector of each feature point in the second feature point sequence are calculated.

[0470] These two feature point sequences each include N feature points, each feature point corresponds to a feature vector, and the feature vector corresponding to each feature point is a 256-bit sequence of 0s or 1s.

[0471] It should be noted that the int data type occupies 4 bytes (32 bits) in memory, therefore 8 int variables can represent 256 bits of data. Specifically, in this application, since the feature vector is a 256-bit sequence of 0s and 1s, this application uses 8 int variables to segment and store the feature vector values. Each int variable stores a 32-bit sequence of 0s and 1s.

[0472] Among them, the 8 int type variables can be represented as int svData[k], where k takes the values ​​0, 1, ..., 7.

[0473] Any feature vector in the first feature point sequence KeyPoint1 can be represented as KeyPoint1.svData[0], KeyPoint1.svData[1], KeyPoint1.svData[2], KeyPoint1.svData[3], KeyPoint1.svData[4], KeyPoint1.svData[5], KeyPoint1.svData[6], KeyPoint1.svData[7].

[0474] Any feature vector in the second feature point sequence KeyPoint1 can be represented as KeyPoint2.svData[0], KeyPoint2.svData[1], KeyPoint2.svData[2], KeyPoint2.svData[3], KeyPoint2.svData[4], KeyPoint2.svData[5], KeyPoint2.svData[6], KeyPoint2.svData[7].

[0475] In this embodiment of the application, an XOR operation can be performed on the pairwise feature vectors of the first feature point sequence KeyPoint1 and the second feature point sequence KeyPoint2:

[0476] Int result0=KeyPoint1.svData[0]^KeyPoint2.svData[0];

[0477] Int result1=KeyPoint1.svData[1]^KeyPoint2.svData[1];

[0478] Int result2=KeyPoint1.svData[2]^KeyPoint2.svData[2];

[0479] Int result3=KeyPoint1.svData[3]^KeyPoint2.svData[3];

[0480] Int result4=KeyPoint1.svData[4]^KeyPoint2.svData[4];

[0481] Int result5=KeyPoint1.svData[5]^KeyPoint2.svData[5];

[0482] Int result6=KeyPoint1.svData[6]^KeyPoint2.svData[6];

[0483] Int result7=KeyPoint1.svData[7]^KeyPoint2.svData[7].

[0484] Here, "^" represents the XOR operation symbol. Table 5 shows the rules for XOR operation.

[0485] Table 5

[0486]

[0487] In this embodiment, the feature vectors of the first feature point sequence KeyPoint1 and the second feature point sequence KeyPoint2 are each stored in segments using eight int variables. Eight XOR operations are then performed on each pair of these segments to obtain eight XOR results. Each XOR result is a 32-bit sequence of 0s or 1s.

[0488] In this embodiment, the GPU's bitCount() instruction can be used to count the number of 1s in each XOR result, and the counts of 1s in the eight XOR results can be summed. The sum is then used as the feature distance (svDistance) between the two feature vectors.

[0489] svDistance=bitCount(result0)+bitCount(result1)+bitCount(result2)+bitCount(result3)+bitCount(result4)+bitCount(result5)+bitCount(result6)+bitCount(result7).

[0490] The above describes the method for calculating the feature distance between two feature vectors. Based on the above method for calculating the feature distance between two feature vectors, the following describes the method provided in this application for calculating the feature distance between each feature point in two feature point sequences.

[0491] refer to Figure 22A and Figure 22B The first feature point sequence (KeyPoint1) includes the N feature points (denoted as KeyPoint1_i) with the largest response values ​​in the first color image, and each feature point corresponds to a feature vector. Similarly, the second feature point sequence (KeyPoint2) includes the N feature points (denoted as KeyPoint2_i) with the largest response values ​​in the second color image, and each feature point corresponds to a feature vector.

[0492] In this embodiment of the application, the feature distance between each pair of feature points is calculated based on the feature vector of each feature point (KeyPoint1_i) in the first feature point sequence (KeyPoint1) and the feature vector of all feature points (Keypoint2_i, i takes the values ​​1, 2, ..., N) in the second feature point sequence (Keypoint2).

[0493] A total of N sets of calculations are required, and each set of calculations yields N feature distances.

[0494] First set of calculations: (e.g.) Figure 22A As shown in Table 6 below, the pairwise feature distances between the first feature point (KeyPoint1_1) in the first feature point sequence (KeyPoint1) and the feature vectors of the N feature points Keypoint2_1, Keypoint2_2, ..., Keypoint2_N in the second feature point sequence (Keypoint2) can be calculated. For feature point KeyPoint1_1, N feature distances can be obtained through calculation.

[0495] Second set of calculations: (e.g.) Figure 22B As shown in Table 6 below, the pairwise feature distances between the second feature point (KeyPoint1_2) in the first feature point sequence (KeyPoint1) and the feature vectors of the N feature points Keypoint2_1, Keypoint2_2, ..., Keypoint2_N in the second feature point sequence (Keypoint2) can be calculated. For feature point KeyPoint1_2, N feature distances can be obtained through calculation.

[0496] The Nth group calculation: Referring to Table 6, we can calculate the pairwise feature distances between the Nth feature point (KeyPoint1_N) in the first feature point sequence (KeyPoint1) and the feature vectors of the N feature points Keypoint2_1, Keypoint2_2, ..., Keypoint2_N in the second feature point sequence (Keypoint2). For feature point KeyPoint1_N, N feature distances can be obtained through calculation.

[0497] Wherein, Keypoint1_1, Keypoint1_2, ..., Keypoint1_N represent the first feature point, the second feature point, ..., the Nth feature point of the first feature point sequence (Keypoint1), respectively. Keypoint2_1, Keypoint2_2, ..., Keypoint2_N represent the first feature point, the second feature point, ..., the Nth feature point of the second feature point sequence (Keypoint2), respectively.

[0498] Table 6

[0499]

[0500] Step 205: Perform feature matching based on feature distance

[0501] In this embodiment of the application, referring to Table 7 below, for each of the above N sets of calculations, the minimum value among the N feature distances is found, and the minimum value among the N feature distances is compared with a preset distance threshold (svDistance_Min). Based on the comparison result, it is determined whether the feature points match, so as to improve the accuracy of feature point matching.

[0502] Table 7

[0503]

[0504] If the minimum distance among the N feature distances is less than a preset distance threshold, then the two feature points corresponding to the minimum distance are successfully matched. If the minimum distance among the N feature distances is greater than or equal to the preset distance threshold, then the matching fails.

[0505] When the feature point matching operation in two frames of images is completed, feature point pairs representing the same object in different images can be output.

[0506] For example, such as Figure 23 As shown, assuming that feature point (x1, y1) in the first color image and feature point (x2, y2) in the second color image are two successfully matched feature points, then feature point (x1, y1) and feature point (x2, y2) constitute a feature point pair. Accordingly, this feature point pair can be denoted as [(x1, y1), (x2, y2)].

[0507] In practice, after matching feature points between two frames of images, the matching feature point pairs between the two frames can be determined. Then, based on the position information of the matching feature point pairs, they can be used for purposes such as motion estimation, image stitching, and background segmentation.

[0508] The following describes the application scenarios of GPU-based feature point recognition and matching provided in the embodiments of this application. A game scenario is used as an example for illustrative purposes.

[0509] The GPU-based feature point recognition and matching provided in this application can be applied to inter-frame prediction of game images. (Reference) Figure 24 A game can predict one frame from two consecutive game frames, which serves as the intermediate frame between the two frames. The two consecutive game frames are called the real frame N-1 and the real frame N+1, and the predicted intermediate frame is called the predicted frame N.

[0510] In this embodiment, a predicted frame can be generated based on two sequentially acquired rendering instruction streams. "Two sequentially acquired rendering instruction streams" means that rendering instruction stream 1 is acquired first, followed by rendering instruction stream 2. After acquiring rendering instruction stream 1 and before acquiring rendering instruction stream 2, the GPU has not acquired any other rendering instruction streams. Specifically, the GPU can acquire two rendering instruction streams sequentially. One rendering instruction stream is used to draw a real frame image. Feature point recognition is performed on each real frame image. After recognizing the feature points of each real frame image, the feature points in the two real frame images are matched. Based on the image coordinate information of the matched feature point pairs, the motion vector of each pixel (e.g., including translation and / or rotation) is calculated. Then, based on the motion vector, a pixel-by-pixel offset is performed on the 3D scene image corresponding to a certain real frame, thereby generating the final predicted frame.

[0511] Assuming the GPU generates N-1 real frames based on the first obtained rendering instruction stream 1, generates N+1 real frames based on the second obtained rendering instruction stream 2, and generates N predicted frames based on both rendering instruction streams 1 and 2, the GPU can send these frames into the display queue in the order of real frames N-1, predicted frames N, and real frames N+1. The display screen then plays real frames N-1, predicted frames N, and real frames N+1 sequentially.

[0512] It should be understood that since the resource consumption of generating predicted frames using inter-frame prediction technology is less than the resource consumption of rendering real frames in a game application, by reducing the number of real frames generated by the game application and using inter-frame prediction technology to generate predicted frames, not only can the total number of game frames be guaranteed, but the power consumption load of the chip can also be reduced.

[0513] In the aforementioned inter-frame prediction process, this embodiment performs feature point identification and matching on two frames of images respectively. Then, based on the image coordinate information of the matched feature point pairs, it calculates the motion vector (e.g., translation and / or rotation) for each pixel. Finally, based on the motion vector, it performs pixel-by-pixel offset on the 3D scene image corresponding to a real frame to generate a predicted frame. It should be noted that this embodiment performs image feature identification and matching on the GPU in real-time and with low power consumption. By introducing the GPU's graphics rendering unit and graphics rendering pipeline, especially by introducing GPU-specific functions such as compute shaders, inter-core shared caches, and atomic operations of thread workgroups, it identifies and matches feature points of different images in the graphics rendering pipeline. This significantly improves computational efficiency and reduces the time overhead of pixel-by-pixel calculations, while also reducing the overhead of data copying and context switching between feature identification and matching and the graphics rendering pipeline.

[0514] This application leverages certain features of the GPU and logic for directly processing feature point matching within the graphics rendering pipeline to achieve efficient and low-power feature point matching operations for rendered images. Even on mobile platforms with limited computing power, it enables high-frame-rate image feature point matching. This solution effectively reduces power consumption in feature matching and machine vision algorithms on mobile devices, expanding its application to graphics rendering scenarios such as VR / AR, improving image processing efficiency and computational accuracy while reducing computational power consumption.

[0515] It should be noted that in the embodiments of this application, "greater than" can be replaced with "greater than or equal to", "less than or equal to" can be replaced with "less than", or "greater than or equal to" can be replaced with "greater than", and "less than" can be replaced with "less than or equal to".

[0516] The various embodiments described herein can be independent solutions or combinations thereof based on their inherent logic, and all such solutions fall within the protection scope of this application.

[0517] The foregoing mainly describes the solutions provided by the embodiments of this application from the perspective of method steps. It is understood that, in order to achieve the above functions, the electronic device implementing this method includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of protection of this application.

[0518] This application also provides a chip coupled to a memory, which is used to read and execute computer programs or instructions stored in the memory to perform the methods in the above embodiments.

[0519] This application also provides an electronic device including a chip for reading and executing computer programs or instructions stored in a memory, causing the methods in the various embodiments to be performed.

[0520] This embodiment also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the GPU-based image feature matching method in the above embodiment.

[0521] This embodiment also provides a computer program product, which is a computer-readable storage medium storing program code. When the computer program product is run on a computer, it causes the computer to perform the above-mentioned related steps to implement the GPU-based image feature matching method in the above embodiment.

[0522] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the GPU-based image feature matching method in the above method embodiments.

[0523] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0524] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0525] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for GPU-based image feature matching, characterized in that, The method comprises: obtaining feature vectors corresponding to feature points of a color image; wherein one feature point corresponds to one feature vector; when the color image is a first color image, N feature points of the first color image correspond to N feature vectors; when the color image is a second color image, N feature points of the second color image correspond to N feature vectors; determining N feature distances between each feature point of the first color image and N feature points of the second color image respectively according to N feature vectors corresponding to N feature points of the first color image and N feature vectors corresponding to N feature points of the second color image; if a minimum feature distance in the N feature distances is less than a preset distance threshold, determining feature points to which two feature vectors of the minimum feature distance belong as a matched feature point pair. The method further comprises: loading a mask image in a first computing shader of the GPU, and generating M four-dimensional random constants according to image coordinate data of feature point pixels in the mask image; the mask image is an image obtained by performing feature point identification on the color image, and the mask image marks a plurality of feature point pixels and corresponding response function values; loading the mask image in a second computing shader of the GPU, sorting the plurality of feature point pixels marked in the mask image according to response function value sizes, and determining N feature point pixels with maximum response function values in the mask image; loading the color image in a third computing shader of the GPU, determining a neighboring pixel brightness difference of a feature point pixel in the color image according to the M four-dimensional random constants, and determining a feature vector of the N feature point pixels with the maximum response function values in the color image as N feature vectors corresponding to N feature points of the color image.

2. The method of claim 1, wherein, Each of the M four-dimensional random constants comprises a first random offset and a second random offset. The method further comprises: in the third computing shader of the GPU, for each thread of a local workgroup, determining a first pixel color value after offsetting a first feature point pixel by the first random offset and a second pixel color value after offsetting the first feature point pixel by the second random offset, and calculating a first pixel brightness value according to the first pixel color value and a second pixel brightness value according to the second pixel color value; calculating a brightness difference value between the second pixel brightness value and the first pixel brightness value.

3. The method of claim 2, wherein, The method further comprises: if the brightness difference value is greater than 0, outputting a result of 1; if the brightness difference value is less than or equal to 0, outputting a result of 0; obtaining a sequence of 0 or 1 according to the output result as a feature vector corresponding to the first feature point pixel.

4. The method of claim 3, wherein, In the case that the local workgroup includes 16x16 threads, the feature vector corresponding to the feature point pixel of the first color image is a sequence of 256 bits of 0 or 1.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: storing the feature vector of a feature point pixel of the first color image in S integer variables in segments; storing the feature vector of a feature point pixel of the second color image in S integer variables in segments; performing XOR operation on the i-th integer variable stored in segments of the first color image and the i-th integer variable stored in segments of the second color image, i being 1, …, S; counting the number of 1s in the result of the XOR operation by using the bitCount instruction of the GPU; taking the counted number of 1s as the feature distance between the feature point pixel of the first color image and the feature point pixel of the second color image.

6. The method of claim 5, wherein, In the case that the feature vector is a sequence of 256 bits of 0 or 1, and one integer variable is 32 bits, S is 8.

7. The method according to any one of claims 1 to 6, characterized in that, The value of M is equal to the size of the local workgroup of the calculation pipeline in the third calculation shader of the GPU.

8. The method according to any one of claims 1 to 7, characterized in that, The size of the local workgroup of the calculation pipeline in the third calculation shader of the GPU is equal to the size of the local workgroup of the calculation pipeline in the first calculation shader of the GPU; and the size of the global workgroup of the calculation pipeline in the third calculation shader of the GPU is equal to the size of the global workgroup of the calculation pipeline in the first calculation shader of the GPU.

9. The method according to any one of claims 1 to 8, characterized in that, The product of the size of the local workgroup and the size of the global workgroup of the calculation pipeline in the second calculation shader of the GPU is equal to the size of the mask image; The size of the local workgroup of the calculation pipeline in the second calculation shader of the GPU is equal to the size of the local workgroup of the calculation pipeline in the third calculation shader of the GPU.

10. The method according to any one of claims 1 to 9, characterized in that, In the first calculation shader of the GPU, the size of the local workgroup of the calculation pipeline is 16x16, and the size of the global workgroup of the calculation pipeline is (1, 1, 1).

11. The method according to any one of claims 1 to 10, characterized in that, In the second calculation shader of the GPU, the size of the local workgroup of the calculation pipeline is 16x16, the size of the global workgroup of the calculation pipeline is (W / 16, H / 16, 1), and the size of the mask image is WxH pixels.

12. The method according to any one of claims 1 to 11, characterized in that, In the third calculation shader of the GPU, the size of the local workgroup of the calculation pipeline is 16x16, and the size of the global workgroup of the calculation pipeline is (1, 1, 1).

13. The method according to any one of claims 1 to 12, characterized in that, The generating of the M four-dimensional random constants according to the image coordinate data of the feature point pixels in the mask image comprises: multiplying the position index value of the feature point pixel in the mask image by a specified random number scaling factor to obtain a random number seed; in the first calculation shader of the GPU, for each thread in the local workgroup, generating one four-dimensional random constant by using trigonometric functions and exponential functions on the random number seed, and adjusting the one four-dimensional random constant according to an offset parameter to obtain an adjusted four-dimensional random constant; in the case that the local workgroup includes 16x16 threads, 256 adjusted four-dimensional random constants are obtained, and M is equal to 256.

14. The method according to any one of claims 1 to 13, characterized in that, After the M four-dimensional random constants are generated according to the image coordinate data of the pixels in the mask image, the method further comprises: storing the M four-dimensional random constants in a first shader storage buffer.

15. The method according to any one of claims 1 to 14, characterized in that, In the second computing shader of the GPU, a global workgroup comprises W / 16×H / 16 local workgroups; The sorting of the plurality of feature point pixels marked in the mask image according to the response function value size, and the determination of the first N feature point pixels with the largest response function value in the mask image, comprises: For the W / 16×H / 16 local workgroups, the first N feature points with the largest response function value in each local workgroup are determined by using the atomic operation atomicMax of the GPU; the first N feature points with the largest response function value are arranged in descending order of the response function value; For the W / 16×H / 16 local workgroups, the first N feature points with the largest response function value in the global workgroup are determined from the N×W / 16×H / 16 feature points, as the first N feature points with the largest response function value in the mask image.

16. The method of claim 15, wherein, The determination of the first N feature points with the largest response function value in the global workgroup from the N×W / 16×H / 16 feature points for the W / 16×H / 16 local workgroups, comprises: For the W / 16×H / 16 local workgroups, the first N feature points with the largest response function value in each local workgroup are sequentially stored in a second shader storage buffer; The first N feature points with the largest response function value in the global workgroup are determined from the N×W / 16×H / 16 feature points stored in the second shader storage buffer.

17. The method of claim 16, wherein, The sequential storage of the first N feature points with the largest response function value in each local workgroup in the second shader storage buffer for the W / 16×H / 16 local workgroups, comprises: For the W / 16×H / 16 local workgroups, the first N feature points with the largest response function value in each local workgroup are stored in a shared buffer between workgroups according to a position index; The feature points in the shared buffer are copied to the corresponding positions of the second shader storage buffer according to the position index.

18. The method of any one of claims 1 to 17, wherein, The matched feature point pair comprises a first feature point and a second feature point; After the two feature vectors belonging to the feature points with the minimum feature distance are determined as the matched feature point pair, the method further comprises: extracting the coordinate parameters of the first feature point from the mask image corresponding to the first color image; extracting the coordinate parameters of the second feature point from the mask image corresponding to the second color image.

19. An electronic device, comprising: The electronic device comprises a processor, a memory, and a computer program stored in the memory, and the processor is configured to execute the computer program to enable the electronic device to implement the method according to any one of claims 1 to 18.

20. A chip comprising a processor, a memory, and a computer program stored on the memory and loadable on the processor, characterized in that, The processor is configured to execute the computer program to implement the method according to any one of claims 1 to 18.

21. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and when the computer program runs on an electronic device, the electronic device executes the method according to any one of claims 1 to 18.

Images