Real-time SuperPoint feature extraction and description method and device

Abstract

The invention relates to a real-time SuperPoint network feature extraction and description method and a real-time SuperPoint network feature extraction and description device. The method comprises the following steps: performing feature extraction on an input image by using a SuperPoint convolutional neural network module, and generating a first feature tensor for performing feature point extraction and a second feature tensor for performing descriptor calculation; constructing a feature point extraction module to extract feature point coordinates based on the first feature tensor, and outputting the feature point coordinates and scanning coordinates in real time; and constructing a descriptor calculation module to obtain the feature point coordinates and the scanning coordinates in real time, and selecting tensor data of a corresponding position from the second feature tensor according to the feature point coordinates and the scanning coordinates to perform descriptor calculation. According to the method, parallel processing of feature point extraction and descriptor calculation can be realized, and the instantaneity of SuperPoint network feature extraction and description is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a real-time SuperPoint network feature extraction and description method and device. Background Art

[0002] Feature points refer to locations with obvious changes in images, usually local areas with good distinguishing ability, and can exist stably under different scales and rotation transformations. Descriptors are methods of numerically describing feature points, usually represented by vectors or matrices, and used to describe local areas around feature points. Feature point extraction and descriptor calculation are important components of SLAM. In recent years, convolutional neural networks have been used for feature point extraction to improve the accuracy of feature point extraction. As one of the most advanced feature point extraction methods, SuperPoint has a self-supervised learning framework, isomorphic adaptation to enhance rich point detection and robust descriptors. It includes feature point detection steps and descriptor generation steps in a single forward pass, surpassing traditional manual methods in accuracy.

[0003] Feature point extraction and descriptor calculation generally involve extracting feature points first and then calculating descriptors based on the feature points. The order in which these two parts are processed makes parallel processing difficult, which in turn increases processing delays and reduces real-time performance. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a real-time SuperPoint network feature extraction and description method and device, which can process feature point extraction and descriptor calculation with high parallelism.

[0005] The technical solution adopted by the present invention to solve the technical problem is: to provide a real-time SuperPoint network feature extraction and description method, comprising:

[0006] Using the SuperPoint convolutional neural network module to extract features from the input image, and generate a first feature tensor for feature point extraction and a second feature tensor for descriptor calculation;

[0007] Constructing a feature point extraction module to extract feature point coordinates based on the first feature tensor, and outputting the feature point coordinates and scanning coordinates in real time;

[0008] A descriptor calculation module is constructed to obtain the feature point coordinates and the scanning coordinates in real time, and select tensor data at corresponding positions from the second feature tensor according to the feature point coordinates and the scanning coordinates to perform descriptor calculation.

[0009] Furthermore, the extracting feature point coordinates based on the first feature tensor and outputting the feature point coordinates and scanning coordinates in real time includes:

[0010] Performing Softmax processing and Reshape operations on the first feature tensor in sequence to obtain a feature point probability map;

[0011] The NMS algorithm is used to perform target detection on the feature point probability map, and the coordinates of the feature points in the current detection window scanned are output. When the last data in the current detection window is scanned, the coordinates at this resolution are output as the scanning coordinates.

[0012] Furthermore, each convolutional layer of the convolutional neural network module is provided with an input cache and an output cache to realize parallel pipeline operations between the convolutional layers.

[0013] Furthermore, an input cache and an output cache are provided in the descriptor computing module, and its input cache is connected to the output cache of the second convolutional layer of the descriptor decoder in the convolutional neural network module through a stream interface for obtaining the second feature tensor.

[0014] Further, the selecting tensor data at a corresponding position from the second feature tensor according to the feature point coordinates and the scanning coordinates to perform descriptor calculation includes:

[0015] Check whether the feature point coordinates and scanning coordinates are currently received;

[0016] When there are neither feature point coordinates nor scanning coordinates, no data is received and the waiting continues;

[0017] When there are scanning coordinates but no feature point coordinates, whether to start data reception is selected according to the input buffer state, and invalid input data is deleted based on the scanning coordinates;

[0018] When there are feature point coordinates, whether to start data reception is selected according to the input buffer state, and valid data in the input data is extracted based on the feature point coordinates, and then descriptor calculation is performed based on the extracted valid data.

[0019] Further, the selecting whether to start data reception according to the input buffer state and deleting invalid input data based on the scanning coordinates includes:

[0020] S101 determines whether there is data stored in the input buffer;

[0021] S102: when no data is stored in the input buffer, data reception is started, and when receiving data, input data whose row number is less than the row value of the scanning coordinate is deleted, and input data whose row number is equal to the row value of the scanning coordinate and whose column number is less than the column value of the scanning coordinate is deleted, and this step is repeated until the row number of the input data is equal to the row value of the scanning coordinate and the column number is equal to the column value of the scanning coordinate;

[0022] S103 When data is stored in the input buffer:

[0023] If the row number of the current cache input data is smaller than the row value of the scanning coordinate, it is deleted and the process returns to step S102;

[0024] If the row number of the current cache input data is equal to the row value of the scanning coordinate, then all data in this row whose column number is less than the column value of the scanning coordinate is deleted and then data reception stops;

[0025] If the row number of the currently cached input data is greater than the row value of the scanning coordinate, it is retained and data reception is stopped.

[0026] Further, the selecting whether to start data reception according to the input buffer state and extracting valid data from the input data based on the feature point coordinates includes:

[0027] S201 determines whether there is data stored in the input buffer;

[0028] S202: Start data reception when no data is stored in the input buffer:

[0029] If the row number of the input data is less than the row value of the feature point coordinates, delete it;

[0030] If the row number of the input data is equal to the row value of the feature point coordinates, then delete all input data in the row whose column numbers are less than the column value of the feature point coordinates, retain the input data in the row whose column numbers are equal to the column value of the feature point coordinates, and cache the subsequent input data;

[0031] If the row number of the input data is equal to the row value of the feature point coordinates + 1, retain all the input data in the row whose column numbers are greater than or equal to the column value of the feature point coordinates, continue to cache the remaining data until the row of data is received, and stop receiving data;

[0032] Extracting the valid data from the buffered input data;

[0033] S203 When data is stored in the input buffer:

[0034] If the row number of the current cache input data is smaller than the row value of the feature point coordinates, it is deleted and the process returns to step S202;

[0035] If the row number of the currently cached input data is equal to the row value of the feature point coordinates, then delete all input data in this row whose column numbers are less than the column value of the feature point coordinates and then receive the next row of data. When receiving the next row of data, retain the input data whose column numbers are greater than or equal to the column value of the feature point coordinates, continue to cache the remaining data until the row of data is received, and then stop receiving data.

[0036] The valid data is extracted from the buffered input data.

[0037] Furthermore, the coordinates of the valid data are (x k ,y k )、(x k +1,y k )、(x k ,y k +1)、(x k +1,y k +1), where x k and k are the column value and row value of the feature point coordinates respectively.

[0038] Furthermore, before performing the descriptor calculation, the method also includes performing an interpolation calculation on the extracted valid data to perform data enhancement.

[0039] The present invention also provides a real-time SuperPoint network feature extraction and description device, comprising:

[0040] SuperPoint convolutional neural network module, used to extract features from input images and generate a first feature tensor for feature point extraction and a second feature tensor for descriptor calculation;

[0041] A feature point extraction module, used for extracting feature point coordinates based on the first feature tensor, and outputting the feature point coordinates and scanning coordinates in real time;

[0042] A descriptor calculation module is used to obtain the feature point coordinates and the scanning coordinates in real time, and select tensor data at corresponding positions from the second feature tensor according to the feature point coordinates and the scanning coordinates to perform descriptor calculation.

[0043] Beneficial Effects

[0044] Due to the adoption of the above technical scheme, the present invention has the following advantages and positive effects compared with the prior art: the present invention drives descriptor calculation through the scanning coordinates of feature point extraction and the feature point coordinates to achieve parallel processing, thereby achieving efficient real-time processing of image feature point extraction and descriptor calculation, and at the same time solves the problem that before the feature point is obtained, the descriptor calculation cannot extract the tensor data of the corresponding position due to the lack of feature point coordinates, which will cause the descriptor calculation to stop, thereby causing data accumulation in the descriptor decoder convolution layer 2, while affecting parallelism and increasing the output cache of the descriptor decoder convolution layer 2, increasing the consumption of storage resources; the present invention can independently operate to obtain the feature points and descriptors of the image by receiving grayscale image data through a stream interface, and can be completely separated from the processor to achieve acceleration. Due to its low computational complexity, it can be deployed and implemented in embedded systems including digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), and systems on chips (SOCs). BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a schematic diagram of the device structure of an embodiment of the present invention;

[0046] Figure 2 Schematic diagram of the convolutional layer structure of an embodiment of the present invention;

[0047] Figure 3 2 is a schematic diagram of a non-maximum suppression processing sequence according to an embodiment of the present invention;

[0048] Figure 4 is a flow chart of performing data processing according to scanning coordinates according to an embodiment of the present invention;

[0049] Figure 5 It is a flow chart of performing data processing according to feature point coordinates according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only intended to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art may make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0051] The first embodiment of the present invention relates to a real-time SuperPoint network feature extraction and description method, comprising:

[0052] Using the SuperPoint convolutional neural network module to extract features from the input image, and generate a first feature tensor for feature point extraction and a second feature tensor for descriptor calculation;

[0053] Constructing a feature point extraction module to extract feature point coordinates based on the first feature tensor, and outputting feature point coordinates and scanning coordinates in real time;

[0054] A descriptor calculation module is constructed to obtain feature point coordinates and scanning coordinates in real time, and tensor data at corresponding positions are selected from the second feature tensor according to the feature point coordinates and the scanning coordinates to perform descriptor calculation.

[0055] The constructed device includes the following functional structural modules: interface module, convolutional neural network module, feature point extraction module, and descriptor calculation module.

[0056] There are two types of interfaces in the present invention: one is the APB protocol interface, which is used for parameter configuration, weight data transmission and status information reporting, wherein the weight data only needs to be transmitted once and is stored in RAM internally; the other is the stream interface, which is used for the transmission of large amounts of data, and the stream interface is divided into input and output. The stream interface can be an interface similar to AXI4-Stream, or it can be other customized interfaces. Since the stream interface does not require an address, data transmission is simple and efficient, high-speed, and real-time, which improves the data processing speed and efficiency.

[0057] The shared encoder structure of the Superpoint convolutional neural network is similar to the convolutional structure of the VGG network, which is the shared encoder convolutional layers 1-8, which extract low-level geometric detection and description shared features to obtain the output tensor A∈R (H / 8)×(W / 8)×128 The feature point decoder maps tensor A to tensor B∈R through two feature point decoder convolutional layers 1 and 2. (H / 8)×(W / 8)×65 , perform feature point detection. The descriptor decoder maps the tensor A to a coarse descriptor tensor C∈R through two convolutional layers 1 and 2. (H / 8)×(W / 8)×256 , and generate descriptors.

[0058] Each convolutional layer of the convolutional neural network module adopts a pipeline design, and the stream interface is used for data transmission between each convolutional layer. In order to achieve high computing efficiency, the computing power of each convolutional layer should be matched, that is, the time for each convolutional layer to complete the calculation of a single frame of data should be as equal as possible, so as to give full play to the maximum performance of the pipeline design. Therefore, the computing power is allocated according to the number of multiplications of each convolutional layer. The ratio of the number of multipliers in the 12 convolutional layers from the shared encoder convolutional layer 1 to the descriptor decoder convolutional layer 2 is 1:64:16:16:8:16:4:4:8:0.45:8:1.78, among which the ratio value of the feature point decoder convolutional layer 2 and the descriptor decoder convolutional layer 2 is a decimal. When the number of multipliers is used, the final number of multipliers is rounded up. For example, when a 9-fold ratio is used, the number of multipliers used in each convolutional layer is 9, 576, 144, 144, 72, 144, 36, 36, 72, 5, 72, and 17. By allocating computing resources through this ratio, the computing power of each convolutional layer can be optimized and the processing delay can be minimized.

[0059] The convolutional layers use stream interfaces to transfer data, and are driven by data streams to complete data calculation and processing of each convolutional layer. Each convolutional layer consists of five functional parts: stream input interface, input buffer, convolution and pooling processing, output buffer, and stream output interface. The specific implementation steps of data processing driven by data streams are as follows:

[0060] Step 1: Check the status of the input buffer. If the input buffer is not full, continue to receive data. If the input buffer is full, stop receiving data.

[0061] Step 2: If the convolution and pooling processing is idle and there is data in the input cache, the input cache data is pushed to the convolution and pooling for processing to obtain the calculation result; during the convolution calculation process, the input cache data and weight data are convolved, and the weight data is also stored in the on-chip cache. The calculation process does not require any data interaction with the outside of the device, and no external memory is required.

[0062] Step 3: Check the status of the output buffer. If the output buffer is not full, the convolution calculation or maximum pooling calculation results are written to the output buffer. If the output buffer is full, the calculation results stop being written to the output buffer. After all the results of this calculation are written to the output buffer, the convolution and pooling processing returns to the idle state and prepares for the next calculation.

[0063] Step 4: Check the status of the output buffer. If the output buffer is not empty and the next-layer module requests to send data, push the output buffer data to the next layer; if the output buffer is empty or the next-layer module refuses to receive data, the output buffer data stops being output.

[0064] The data stream is used to drive input reception, convolution and pooling processing, and output push. There is no need for complex control. Through the streaming data drive, the calculation and processing of each convolution layer are adaptively completed. Then, through the optimal computing power allocated in proportion, ideally, it can achieve: always receiving data, always calculating data, and always outputting data, achieving high computing efficiency.

[0065] The feature point extraction module and the descriptor calculation module also use a similar method to interact with the convolutional neural network module. Taking the descriptor calculation module as an example, its internal input buffer is connected to the output buffer of the sub-decoder convolution layer 2 through a stream interface, and the interaction between them is carried out through the ready signal at the receiving end and the valid signal at the sending end. When the input buffer is not full, the ready signal is pulled high to prepare to receive data. If there is data in the output buffer of the sub-decoder convolution layer 2 at this time, the valid signal is pulled high. At this time, they are both high and the received data is valid. When the input buffer is full, the ready signal is pulled low, and the output buffer of the sub-decoder convolution layer 2 stops outputting.

[0066] The feature point extraction and descriptor calculation of the network post-processing are processed in parallel. The feature point decoder convolutional layer decodes the feature map and obtains the tensor B∈R (H / 8)×(W / 8)×65 The output of , where the number of channels 65 corresponds to the non-overlapping local 8×8 pixel block area in the input image, and 1 background channel representing no feature points. In the feature point extraction module, the tensor B is first processed by Softmax to obtain the initial filtered feature points. Then the output is converted from R to (H / 8)×(W / 8)×64 Convert tensor back to R H×W×1 Tensor, get the feature point probability map. Then use the non-maximum suppression (NMS) algorithm, each window only retains one maximum value, then the whole image is thresholded, and the points above the threshold are regarded as feature points, so as to find the best detection position, that is, the feature point coordinates (x k ,y k ).

[0067] While extracting feature points, the descriptor calculation is also performed in parallel. It receives the descriptor decoder and maps the tensor A to the coarse descriptor tensor C∈R through two convolutional layers. (H / 8)×(W / 8)×256 In order to reduce the amount of calculation, the tensor data at the corresponding position is selected according to the feature point coordinates for descriptor calculation, instead of calculating the entire tensor C∈R (H / 8)×(W / 8)×256 The tensor data at the corresponding position is linearly interpolated to obtain the complete descriptor, and finally L2 normalization is used to obtain a 256-dimensional descriptor vector of unit length.

[0068] Before the feature points are obtained, the descriptor module cannot extract the tensor data of the corresponding position because it does not have the feature point coordinates, causing the descriptor calculation to stop. This will cause the data of the descriptor decoder convolution layer 2 to accumulate, affecting parallelism. At the same time, it will also increase the output cache of the descriptor decoder convolution layer 2, increasing the consumption of storage resources.

[0069] According to the principle that target detection is processed from left to right and from top to bottom in the order of images, when NMS processes each 8×8 square window and scans the last data in the lower right corner, the coordinates (x c ,y c ), which is called scanning coordinates here. In this way, no matter whether there are feature points in the 8×8 square, the scanning coordinates can be output in order, so that the descriptor module can receive data synchronously, and does not have to wait until there are feature points to receive data. At the same time, since the scanning coordinates are output only when the last data in the lower right corner is output, if there are feature points at this time, they have been output before the scanning coordinates. Therefore, in an 8×8 square, the feature point coordinates are output before the scanning coordinates. By outputting the scanning coordinates, the data reading of the descriptor calculation is processed in a coordinated manner, the data accumulation of the convolution layer is reduced, and the calculation efficiency is improved.

[0070] The specific implementation steps of the description submodule for data processing based on the scanning coordinates and feature point coordinates are as follows:

[0071] Step 1: Check whether there are feature point coordinates and scanning coordinates. If there are none, the description submodule does not receive data and continues to wait.

[0072] Step 2: When there are scan coordinates but no feature point coordinates, delete the invalid input data.

[0073] Step 2.1: If there is no data in the input buffer, start data reception;

[0074] Step 2.1.1: If the row number of the input data is less than the row y of the scan coordinates c value, delete the input data of this row and continue to receive new data;

[0075] Step 2.1.2: If the row number of the input data is equal to the row y of the scan coordinates c value, and then determine the column number of the input row. If the column number is less than the column x of the scan coordinate c value, delete all values ​​in this row that are less than x c The column input data continues to receive new data; when the column number is equal to the column x of the scan coordinate c When the value is reached, stop receiving data;

[0076] Step 2.2: If there is data in the input buffer, delete the invalid input data;

[0077] Step 2.2.1: If the row number of the cached input data is less than the row y of the scan coordinate c If the value is not found, delete the input data of this row and return to step 2.1 to continue execution;

[0078] Step 2.2.2: If the row number of the cache input data is equal to the row y of the scan coordinate c value, delete all values ​​in this row that are less than x c After receiving the input data of the column, stop receiving data;

[0079] Step 2.2.3: If the row number of the cache input data is greater than the row y of the scan coordinate c value, the input data is retained, and receiving data stops;

[0080] Step 3: When feature point coordinates are available, extract valid data from the input data.

[0081] Step 3.1: If there is no data in the input buffer, start data reception;

[0082] Step 3.1.1: If the row number of the input data is less than the row y of the feature point coordinates k value, delete the input data of this row and continue to receive new data;

[0083] Step 3.1.2: If the row number of the input data is equal to the row y of the feature point coordinates k value, and then determine the column number of the input row. If the column number is less than the column x of the feature point coordinates k value, delete all values ​​in this row that are less than x k The column input data continues to receive new data; when the column number is equal to the column x of the feature point coordinates k When a value is reached, the input data at that time is saved and the remaining data of the row continues to be cached;

[0084] Step 3.1.3: If the row number of the input data is equal to the row y of the feature point coordinates k +1 value, then keep the column number greater than or equal to the feature point coordinate column x k The input data of the value is received, and the remaining data of the row is cached. After the data of the row is received, the receiving data is stopped; at this time, extract (x k ,y k ), (x k +1,y k ), (x k ,y k +1), (x k +1,y k +1) The data of these four points are interpolated;

[0085] Step 3.2: If there is data in the input buffer, delete the invalid data and keep the valid data;

[0086] Step 3.2.1: If the row number of the cache input data is less than the row y of the feature point coordinates k If the value is not found, delete the input data of this row and return to step 3.1 to continue execution;

[0087] Step 3.2.2: If the row number of the cache input data is equal to the row y of the feature point coordinates k value, delete all values ​​in this row that are less than x k After receiving the input data of the column, receive the data of the next row; when receiving the data of the next row, retain the column number greater than or equal to the feature point coordinate column x k The input data of the value is received, and the remaining data of the row is cached. After the data of the row is received, the receiving data is stopped; at this time, extract (x k ,y k ), (x k +1,y k ), (x k ,y k +1), (x k +1,y k +1) The data of these four points are interpolated;

[0088] The data reception of the descriptor module is driven by scanning coordinates and feature point coordinates, and valid data corresponding to the feature point coordinates are extracted and invalid data is deleted, thereby reducing data cache and calculation amount. The collaborative processing of the present invention enables feature point extraction and descriptor calculation to be performed in parallel, thereby improving the real-time performance of the network feature extraction and description device.

[0089] The present embodiment is further described below with reference to the accompanying drawings and specific devices.

[0090] like Figure 1 As shown, the functional structure of the constructed device includes: an interface module, a convolutional neural network module, a feature point extraction module, and a descriptor calculation module.

[0091] SuperPoint feature extraction and description consists of a common convolutional neural network encoder, a feature point detection decoder and a descriptor decoder, and transmits data to the outside through an interface. There are two types of interfaces for external communication between the embodiment of the present invention: one is the APB protocol interface, which is used for parameter configuration, weight data delivery and status information reporting, in which the weight data only needs to be delivered once and is stored in RAM internally; the other is the stream interface, which is used for the transmission of large amounts of data. The stream interface is divided into input and output. In the embodiment of the present invention, the definition of the stream interface signal involved is shown in Table 1:

[0092] Table 1 Stream interface signal definition

[0093]

[0094] The above stream interface can be an interface similar to AXI4-Stream, or other customized interface, as long as it meets the above signal function and timing requirements. Since the stream interface does not require an address, data transmission is simple and efficient, high-speed, and real-time, which improves data processing speed and efficiency.

[0095] The parameters of the Superpoint convolutional neural network are shown in Table 2. The first number of each convolutional layer is the number of input channels, the middle two numbers are the convolution kernel size, and the last number is the number of output channels. The pooling of the shared encoder convolutional layers 2, 4, and 6 refers to the maximum pooling operation with a step size of 2 after the convolution. The shared encoder structure is similar to the convolution structure of the VGG network. It extracts low-level geometric detection and describes shared features to obtain the output tensor A∈R (H / 8)×(W / 8)×128 The feature point decoder maps tensor A to tensor B∈R through two feature point decoder convolutional layers. (H / 8)×(W / 8)×65 , perform feature point detection. The descriptor decoder maps the tensor A to a coarse descriptor tensor C∈R through two convolutional layers. (H / 8)×(W / 8)×256 , and generate descriptors.

[0096] Table 2 Convolutional neural network parameters

[0097]

[0098]

[0099] *H refers to the vertical resolution of the image, W refers to the horizontal resolution of the image

[0100] The embodiment of the present invention performs pipeline design on each convolution layer, and uses a stream interface to transfer data between each convolution layer. In order to achieve high computing efficiency, the computing power of each convolution layer should be matched, that is, the time for each convolution layer to complete the calculation of a single frame of data should be as equal as possible, so as to exert the maximum performance of the pipeline design. Therefore, the computing power is allocated according to the number of multiplications of each convolution layer in Table 2. The ratio of the number of multipliers of the 12 convolution layers from the shared encoder convolution layer 1 to the descriptor decoder convolution layer 2 is 1:64:16:16:8:16:4:4:8:0.45:8:1.78, where the ratio value of the feature point decoder convolution layer 2 and the descriptor decoder convolution layer 2 is a decimal. When the number of multipliers is used, the final number of multipliers is rounded up. For example, when a 9-fold ratio is used, the number of multipliers used in each convolutional layer is 9, 576, 144, 144, 72, 144, 36, 36, 72, 5, 72, and 17. By allocating computing resources through this ratio, the computing power of each convolutional layer can be optimized and the processing delay can be minimized.

[0101] The embodiment of the present invention uses a stream interface to transfer data between convolutional layers, and drives the data stream to complete the data calculation and processing of each convolutional layer. Figure 2 shown.

[0102] Each convolution layer of the embodiment of the present invention is composed of five functional parts: stream input interface, input buffer, convolution and pooling processing, output buffer, and stream output interface. The specific implementation steps of the convolution layer data processing driven by the data stream are as follows:

[0103] Step 1: Check the status of the input buffer. If the input buffer is not full, continue to receive data. If the input buffer is full, stop receiving data.

[0104] Step 2: If the convolution and pooling processing is idle and there is data in the input cache, the input cache data is pushed to the convolution and pooling for processing to obtain the calculation result; during the convolution calculation process, the input cache data and weight data are convolved, and the weight data is also stored in the on-chip cache. The calculation process does not require any data interaction with the outside of the SuperPoint network feature extraction and description device, and no external memory is required.

[0105] Step 3: Check the status of the output buffer. If the output buffer is not full, the convolution calculation or maximum pooling calculation results are written to the output buffer. If the output buffer is full, the calculation results stop being written to the output buffer. After all the results of this calculation are written to the output buffer, the convolution and pooling processing returns to the idle state and prepares for the next calculation.

[0106] Step 4: Check the status of the output buffer. If the output buffer is not empty and the next-layer module requests to send data, push the output buffer data to the next layer; if the output buffer is empty or the next-layer module refuses to receive data, the output buffer data stops being output.

[0107] The embodiment of the present invention uses data stream to drive input reception, convolution and pooling processing, and output push, without complex control. It adaptively completes the calculation processing of each convolution layer through stream data drive. Then, through the optimal computing power allocated in proportion, it can ideally achieve: always receiving data, always calculating data, and always outputting data, achieving high computing efficiency.

[0108] The feature point extraction module and the descriptor calculation module also use a similar method to interact with the convolutional neural network module. Taking the descriptor calculation module as an example, its internal input buffer is connected to the output buffer of the sub-decoder convolution layer 2 through a stream interface, and the interaction between them is carried out through the ready signal at the receiving end and the valid signal at the sending end. When the input buffer is not full, the ready signal is pulled high to prepare to receive data. If there is data in the output buffer of the sub-decoder convolution layer 2 at this time, the valid signal is pulled high. At this time, they are both high and the received data is valid. When the input buffer is full, the ready signal is pulled low, and the output buffer of the sub-decoder convolution layer 2 stops outputting.

[0109] The embodiment of the present invention adopts parallel processing for feature point extraction and descriptor calculation in network post-processing. The feature point decoder convolutional layer decodes the feature map to obtain a tensor B∈R (H / 8)×(W / 8)×65 The output of , where the number of channels 65 corresponds to the non-overlapping local 8×8 pixel block area in the input image, and 1 background channel representing no feature points. In the feature point extraction module, the tensor B is first processed by Softmax to obtain the initial filtered feature points. Then the output is converted from R to (H / 8)×(W / 8)×64 Convert tensor back to R H×W×1 Tensor, get the feature point probability map. Then use the non-maximum suppression (NMS) algorithm, each window only retains one maximum value, then the whole image is thresholded, and the points above the threshold are regarded as feature points, so as to find the best detection position, that is, the feature point coordinates (x k ,y k ).

[0110] While extracting feature points, the descriptor calculation is also performed in parallel. It receives the descriptor decoder and maps the tensor A to the coarse descriptor tensor C∈R through two convolutional layers. (H / 8)×(W / 8)×256 In order to reduce the amount of calculation, the present invention selects the tensor data at the corresponding position according to the feature point coordinates to perform descriptor calculation, instead of calculating the entire tensor C∈R (H / 8)×(W / 8)×256The tensor data at the corresponding position is linearly interpolated to obtain the complete descriptor, and finally L2 normalization is used to obtain a 256-dimensional descriptor vector of unit length.

[0111] Before the feature points are obtained, the descriptor module cannot extract the tensor data of the corresponding position because it does not have the feature point coordinates, causing the descriptor calculation to stop. This will cause the data of the descriptor decoder convolution layer 2 to accumulate, affecting parallelism. At the same time, it will also increase the output cache of the descriptor decoder convolution layer 2, increasing the consumption of storage resources.

[0112] According to the principle that target detection is processed from left to right and from top to bottom in the order of images, when NMS processes each square window, the coordinates (x c ,y c ) output, which is called scanning coordinates here. By outputting scanning coordinates, the data reading of descriptor calculation is processed in a coordinated manner, which reduces the data accumulation of the convolution layer and improves the calculation efficiency. The processing process of scanning coordinates is shown in Figure 3 shown.

[0113] In the 8×8 grid, when the last data in the lower right corner is scanned, the coordinates (x c ,y c ). In this way, no matter whether there are feature points in the 8×8 square, the scanning coordinates can be output in order, so that the description submodule can receive data synchronously, and does not have to wait until there are feature points to receive data. At the same time, since the scanning coordinates are output only when the last data in the lower right corner is output, if there are feature points at this time, they have been output before the scanning coordinates. Therefore, in an 8×8 square, the feature point coordinates are output before the scanning coordinates.

[0114] like Figure 4 and Figure 5 As shown, the specific implementation steps of the description submodule for data processing according to the scanning coordinates and feature point coordinates are as follows:

[0115] Step 1: Check whether there are feature point coordinates and scanning coordinates. If there are none, the description submodule does not receive data and continues to wait.

[0116] Step 2: When there are scan coordinates but no feature point coordinates, delete the invalid input data.

[0117] Step 2.1: If there is no data in the input buffer, start data reception;

[0118] Step 2.1.1: If the row number of the input data (from 0 to H-1) is less than the row y of the scan coordinate c value, delete the input data of this row and continue to receive new data;

[0119] Step 2.1.2: If the row number of the input data is equal to the row y of the scan coordinates c value, and then determine the column number of the input row. If the column number is less than the column x of the scan coordinate c value, delete all values ​​in this row that are less than x c The column input data continues to receive new data; when the column number is equal to the column x of the scan coordinate c When the value is reached, stop receiving data;

[0120] Step 2.2: If there is data in the input buffer, delete the invalid input data;

[0121] Step 2.2.1: If the row number of the cache input data (from 0 to H-1) is less than the row y of the scan coordinate c If the value is not found, delete the input data of this row and return to step 2.1 to continue execution;

[0122] Step 2.2.2: If the row number of the cache input data is equal to the row y of the scan coordinate c value, delete all values ​​in this row that are less than x c After receiving the input data of the column, stop receiving data;

[0123] Step 2.2.3: If the row number of the cache input data is greater than the row y of the scan coordinate c value, the input data is retained, and receiving data stops;

[0124] Step 3: When feature point coordinates are available, extract valid data from the input data.

[0125] Step 3.1: If there is no data in the input buffer, start data reception;

[0126] Step 3.1.1: If the row number of the input data is less than the row y of the feature point coordinates k value, delete the input data of this row and continue to receive new data;

[0127] Step 3.1.2: If the row number of the input data is equal to the row y of the feature point coordinates k value, and then determine the column number of the input row. If the column number is less than the column x of the feature point coordinates k value, delete all values ​​in this row that are less than x k The column input data continues to receive new data; when the column number is equal to the column x of the feature point coordinates k When a value is reached, the input data at that time is saved and the remaining data of the row continues to be cached;

[0128] Step 3.1.3: If the row number of the input data is equal to the row y of the feature point coordinates k +1 value, then keep the column number greater than or equal to the feature point coordinate column x kThe input data of the value is received, and the remaining data of the row is cached. After the data of the row is received, the receiving data is stopped; at this time, extract (x k ,y k ), (x k +1,y k ), (x k ,y k +1), (x k +1,y k +1) The data of these four points are interpolated;

[0129] Step 3.2: If there is data in the input buffer, delete the invalid data and keep the valid data;

[0130] Step 3.2.1: If the row number of the cache input data is less than the row y of the feature point coordinates k If the value is not found, delete the input data of this row and return to step 3.1 to continue execution;

[0131] Step 3.2.2: If the row number of the cache input data is equal to the row y of the feature point coordinates k value, delete all values ​​in this row that are less than x k After receiving the input data of the column, receive the data of the next row; when receiving the data of the next row, retain the column number greater than or equal to the feature point coordinate column x k The input data of the value is received, and the remaining data of the row is cached. After the data of the row is received, the receiving data is stopped; at this time, extract (x k ,y k ), (x k +1,y k ), (x k ,y k +1), (x k +1,y k +1) The data of these four points are interpolated.

[0132] The data reception of the descriptor module is driven by scanning coordinates and feature point coordinates, and valid data corresponding to the feature point coordinates are extracted and invalid data is deleted, thereby reducing data cache and calculation amount. The collaborative processing of the present invention enables feature point extraction and descriptor calculation to be performed in parallel, thereby improving the real-time performance of the network feature extraction and description device.

[0133] In summary, the embodiment of the present invention proposes that each convolutional layer allocates the number of multipliers in proportion, so that the computing power of the convolutional neural network pipeline calculation is optimized and the processing delay is minimized. Each convolutional layer adaptively completes the data calculation and processing of the convolutional layer through the data flow driving method, achieves high computing efficiency, and achieves the goal of not requiring an external memory, reducing hardware costs. The scanning coordinates extracted by the feature point and the feature point coordinates drive the descriptor calculation to achieve parallel processing and improve real-time performance. The embodiment of the present invention can independently operate to obtain the feature points and descriptors of the image by receiving grayscale image data through the stream interface, and can be completely separated from the processor to achieve acceleration. The implementation method of the above-described device and technology in the embodiment of the present invention can be implemented in a digital electronic circuit system, an integrated circuit system, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), and a system on a chip (SOC).

[0134] Although the above discussion contains several specific implementation details, these should not be interpreted as limiting the scope of the present invention. Relevant personnel can make various changes and modifications without departing from the technical concept of the present invention. As long as they do not deviate from the spirit of the present invention, they are within the scope of protection of the present invention.

[0135] The second embodiment of the present invention relates to a real-time SuperPoint network feature extraction and description device, which includes the following functional structure modules: an interface module, a convolutional neural network module, a feature point extraction module, and a descriptor calculation module.

[0136] There are two types of interfaces in the present invention: one is the APB protocol interface, which is used for parameter configuration, weight data transmission and status information reporting, wherein the weight data only needs to be transmitted once and is stored in RAM internally; the other is the stream interface, which is used for the transmission of large amounts of data, and the stream interface is divided into input and output. The stream interface can be an interface similar to AXI4-Stream, or it can be other customized interfaces. Since the stream interface does not require an address, data transmission is simple and efficient, high-speed, and real-time, which improves the data processing speed and efficiency.

[0137] The shared encoder structure of the Superpoint convolutional neural network is similar to the convolutional structure of the VGG network, which is the shared encoder convolutional layers 1-8, which extract low-level geometric detection and description shared features to obtain the output tensor A∈R (H / 8)×(W / 8)×128 The feature point decoder maps tensor A to tensor B∈R through two feature point decoder convolutional layers 1 and 2. (H / 8)×(W / 8)×65 , perform feature point detection. The descriptor decoder maps the tensor A to a coarse descriptor tensor C∈R through two convolutional layers 1 and 2. (H / 8)×(W / 8)×256 , and generate descriptors.

[0138] Each convolutional layer of the convolutional neural network module adopts a pipeline design, and the stream interface is used for data transmission between each convolutional layer. In order to achieve high computing efficiency, the computing power of each convolutional layer should be matched, that is, the time for each convolutional layer to complete the calculation of a single frame of data should be as equal as possible, so as to give full play to the maximum performance of the pipeline design. Therefore, the computing power is allocated according to the number of multiplications of each convolutional layer. The ratio of the number of multipliers in the 12 convolutional layers from the shared encoder convolutional layer 1 to the descriptor decoder convolutional layer 2 is 1:64:16:16:8:16:4:4:8:0.45:8:1.78, among which the ratio value of the feature point decoder convolutional layer 2 and the descriptor decoder convolutional layer 2 is a decimal. When the number of multipliers is used, the final number of multipliers is rounded up. For example, when a 9-fold ratio is used, the number of multipliers used in each convolutional layer is 9, 576, 144, 144, 72, 144, 36, 36, 72, 5, 72, and 17. By allocating computing resources through this ratio, the computing power of each convolutional layer can be optimized and the processing delay can be minimized.

[0139] The convolutional layers use stream interfaces to transfer data, and are driven by data streams to complete data calculation and processing of each convolutional layer. Each convolutional layer consists of five functional parts: stream input interface, input buffer, convolution and pooling processing, output buffer, and stream output interface. The specific implementation steps of data processing driven by data streams are as follows:

[0140] Step 1: Check the status of the input buffer. If the input buffer is not full, continue to receive data. If the input buffer is full, stop receiving data.

[0141] Step 2: If the convolution and pooling processing is idle and there is data in the input cache, the input cache data is pushed to the convolution and pooling for processing to obtain the calculation result; during the convolution calculation process, the input cache data and weight data are convolved, and the weight data is also stored in the on-chip cache. The calculation process does not require any data interaction with the outside of the device, and no external memory is required.

[0142] Step 3: Check the status of the output buffer. If the output buffer is not full, the convolution calculation or maximum pooling calculation results are written to the output buffer. If the output buffer is full, the calculation results stop being written to the output buffer. After all the results of this calculation are written to the output buffer, the convolution and pooling processing returns to the idle state and prepares for the next calculation.

[0143] Step 4: Check the status of the output buffer. If the output buffer is not empty and the next-layer module requests to send data, push the output buffer data to the next layer; if the output buffer is empty or the next-layer module refuses to receive data, the output buffer data stops being output.

[0144] The data stream is used to drive input reception, convolution and pooling processing, and output push. There is no need for complex control. Through the streaming data drive, the calculation and processing of each convolution layer are adaptively completed. Then, through the optimal computing power allocated in proportion, ideally, it can achieve: always receiving data, always calculating data, and always outputting data, achieving high computing efficiency.

[0145] The feature point extraction and descriptor calculation of the network post-processing are processed in parallel. The feature point decoder convolutional layer decodes the feature map and obtains the tensor B∈R (H / 8)×(W / 8)×65 The output of , where the number of channels 65 corresponds to the non-overlapping local 8×8 pixel block area in the input image, and 1 background channel representing no feature points. In the feature point extraction module, the tensor B is first processed by Softmax to obtain the initial filtered feature points. Then the output is converted from R to (H / 8)×(W / 8)×64 Convert tensor back to R H×W×1 Tensor, get the feature point probability map. Then use the non-maximum suppression (NMS) algorithm, each window only retains one maximum value, then the whole image is thresholded, and the points above the threshold are regarded as feature points, so as to find the best detection position, that is, the feature point coordinates (x k ,y k ).

[0146] While extracting feature points, the descriptor calculation is also performed in parallel. It receives the descriptor decoder and maps the tensor A to the coarse descriptor tensor C∈R through two convolutional layers. (H / 8)×(W / 8)×256 In order to reduce the amount of calculation, the tensor data at the corresponding position is selected according to the feature point coordinates for descriptor calculation, instead of calculating the entire tensor C∈R (H / 8)×(W / 8)×256 The tensor data at the corresponding position is linearly interpolated to obtain the complete descriptor, and finally L2 normalization is used to obtain a 256-dimensional descriptor vector of unit length.

[0147] Before the feature points are obtained, the descriptor module cannot extract the tensor data of the corresponding position because it does not have the feature point coordinates, causing the descriptor calculation to stop. This will cause the data of the descriptor decoder convolution layer 2 to accumulate, affecting parallelism. At the same time, it will also increase the output cache of the descriptor decoder convolution layer 2, increasing the consumption of storage resources.

[0148] According to the principle that target detection is processed from left to right and from top to bottom in the order of images, when NMS processes each 8×8 square window and scans the last data in the lower right corner, the coordinates (x c ,y c), which is called scanning coordinates here. In this way, no matter whether there are feature points in the 8×8 square, the scanning coordinates can be output in order, so that the descriptor module can receive data synchronously, and does not have to wait until there are feature points to receive data. At the same time, since the scanning coordinates are output only when the last data in the lower right corner is output, if there are feature points at this time, they have been output before the scanning coordinates. Therefore, in an 8×8 square, the feature point coordinates are output before the scanning coordinates. By outputting the scanning coordinates, the data reading of the descriptor calculation is processed in a coordinated manner, the data accumulation of the convolution layer is reduced, and the calculation efficiency is improved.

[0149] The specific implementation steps of the description submodule for data processing based on the scanning coordinates and feature point coordinates are as follows:

[0150] Step 1: Check whether there are feature point coordinates and scanning coordinates. If there are none, the description submodule does not receive data and continues to wait.

[0151] Step 2: When there are scan coordinates but no feature point coordinates, delete the invalid input data.

[0152] Step 2.1: If there is no data in the input buffer, start data reception;

[0153] Step 2.1.1: If the row number of the input data is less than the row y of the scan coordinates c value, delete the input data of this row and continue to receive new data;

[0154] Step 2.1.2: If the row number of the input data is equal to the row y of the scan coordinates c value, and then determine the column number of the input row. If the column number is less than the column x of the scan coordinate c value, delete all values ​​in this row that are less than x c The column input data continues to receive new data; when the column number is equal to the column x of the scan coordinate c When the value is reached, stop receiving data;

[0155] Step 2.2: If there is data in the input buffer, delete the invalid input data;

[0156] Step 2.2.1: If the row number of the cached input data is less than the row y of the scan coordinate c If the value is not found, delete the input data of this row and return to step 2.1 to continue execution;

[0157] Step 2.2.2: If the row number of the cache input data is equal to the row y of the scan coordinate c value, delete all values ​​in this row that are less than x c After receiving the input data of the column, stop receiving data;

[0158] Step 2.2.3: If the row number of the cache input data is greater than the row y of the scan coordinate c value, the input data is retained, and receiving data stops;

[0159] Step 3: When feature point coordinates are available, extract valid data from the input data.

[0160] Step 3.1: If there is no data in the input buffer, start data reception;

[0161] Step 3.1.1: If the row number of the input data is less than the row y of the feature point coordinates k value, delete the input data of this row and continue to receive new data;

[0162] Step 3.1.2: If the row number of the input data is equal to the row y of the feature point coordinates k value, and then determine the column number of the input row. If the column number is less than the column x of the feature point coordinates k value, delete all values ​​in this row that are less than x k The column input data continues to receive new data; when the column number is equal to the column x of the feature point coordinates k When a value is reached, the input data at that time is saved and the remaining data of the row continues to be cached;

[0163] Step 3.1.3: If the row number of the input data is equal to the row y of the feature point coordinates k +1 value, then keep the column number greater than or equal to the feature point coordinate column x k The input data of the value is received, and the remaining data of the row is cached. After the data of the row is received, the receiving data is stopped; at this time, extract (x k ,y k ), (x k +1,y k ), (x k ,y k +1), (x k +1,y k +1) The data of these four points are interpolated;

[0164] Step 3.2: If there is data in the input buffer, delete the invalid data and keep the valid data;

[0165] Step 3.2.1: If the row number of the cache input data is less than the row y of the feature point coordinates k If the value is not found, delete the input data of this row and return to step 3.1 to continue execution;

[0166] Step 3.2.2: If the row number of the cache input data is equal to the row y of the feature point coordinates k value, delete all values ​​in this row that are less than x kAfter receiving the input data of the column, receive the data of the next row; when receiving the data of the next row, retain the column number greater than or equal to the feature point coordinate column x k The input data of the value is received, and the remaining data of the row is cached. After the data of the row is received, the receiving data is stopped; at this time, extract (x k ,y k ), (x k +1,y k ), (x k ,y k +1), (x k +1,y k +1) The data of these four points are interpolated;

[0167] The data reception of the descriptor module is driven by scanning coordinates and feature point coordinates, and valid data corresponding to the feature point coordinates are extracted and invalid data is deleted, thereby reducing data cache and calculation amount. The collaborative processing of the present invention enables feature point extraction and descriptor calculation to be performed in parallel, thereby improving the real-time performance of the network feature extraction and description device.

[0168] The embodiment of the present invention uses data stream to drive input reception, convolution and pooling processing, and output push, without complex control. It adaptively completes the calculation processing of each convolution layer through stream data drive. Then, through the optimal computing power allocated in proportion, it can ideally achieve: always receiving data, always calculating data, and always outputting data, achieving high computing efficiency.

Claims

1. A real-time SuperPoint network feature extraction and description method, characterized in that: include: Using the SuperPoint convolutional neural network module to extract features from the input image, and generate a first feature tensor for feature point extraction and a second feature tensor for descriptor calculation; Constructing a feature point extraction module to extract feature point coordinates based on the first feature tensor, and outputting the feature point coordinates and scanning coordinates in real time; A descriptor calculation module is constructed to obtain the feature point coordinates and the scanning coordinates in real time, and select tensor data at corresponding positions from the second feature tensor according to the feature point coordinates and the scanning coordinates to perform descriptor calculation.

2. The method according to claim 1, characterized in that The extracting feature point coordinates based on the first feature tensor and outputting the feature point coordinates and scanning coordinates in real time includes: Performing Softmax processing and Reshape operations on the first feature tensor in sequence to obtain a feature point probability map; The NMS algorithm is used to perform target detection on the feature point probability map, and the coordinates of the feature points in the current detection window scanned are output. When the last data in the current detection window is scanned, the coordinates at this resolution are output as the scanning coordinates.

3. The method according to claim 2, characterized in that Each convolutional layer of the convolutional neural network module is provided with an input buffer and an output buffer to realize parallel pipeline operations between the convolutional layers.

4. The method according to claim 3, characterized in that The descriptor calculation module is provided with an input cache and an output cache, and its input cache is connected to the output cache of the second convolutional layer of the descriptor decoder in the convolutional neural network module through a stream interface for obtaining the second feature tensor.

5. The method according to claim 4, characterized in that The selecting tensor data at a corresponding position from the second feature tensor according to the feature point coordinates and the scanning coordinates to perform descriptor calculation includes: Check whether the feature point coordinates and scanning coordinates are currently received; When there are neither feature point coordinates nor scanning coordinates, no data is received and the waiting continues; When there are scanning coordinates but no feature point coordinates, whether to start data reception is selected according to the input buffer state, and invalid input data is deleted based on the scanning coordinates; When there are feature point coordinates, whether to start data reception is selected according to the input buffer state, and valid data in the input data is extracted based on the feature point coordinates, and then descriptor calculation is performed based on the extracted valid data.

6. The method according to claim 5, characterized in that The step of selecting whether to start data reception according to the input buffer state and deleting invalid input data based on the scanning coordinates includes: S101 determines whether there is data stored in the input buffer; S102: when no data is stored in the input buffer, data reception is started, and when receiving data, input data whose row number is less than the row value of the scanning coordinate is deleted, and input data whose row number is equal to the row value of the scanning coordinate and whose column number is less than the column value of the scanning coordinate is deleted, and this step is repeated until the row number of the input data is equal to the row value of the scanning coordinate and the column number is equal to the column value of the scanning coordinate; S103 When data is stored in the input buffer: If the row number of the current cache input data is less than the row value of the scanning coordinate, delete it and return to step S102; if the row number of the current cache input data is equal to the row value of the scanning coordinate, delete all data in this row whose column number is less than the column value of the scanning coordinate and stop receiving data; If the row number of the currently cached input data is greater than the row value of the scanning coordinate, it is retained and data reception is stopped.

7. The method according to claim 5, characterized in that The step of selecting whether to start data reception according to the input buffer state and extracting valid data from the input data based on the feature point coordinates includes: S201 determines whether there is data stored in the input buffer; S202: Start data reception when no data is stored in the input buffer: If the row number of the input data is less than the row value of the feature point coordinates, delete it; If the row number of the input data is equal to the row value of the feature point coordinates, then delete all input data in the row whose column numbers are less than the column value of the feature point coordinates, retain the input data in the row whose column numbers are equal to the column value of the feature point coordinates, and cache the subsequent input data; If the row number of the input data is equal to the row value of the feature point coordinate + 1, then retain all the input data in the row whose column numbers are greater than or equal to the column value of the feature point coordinate, and continue to cache the remaining data until the row of data is received. Stop receiving data; Extracting the valid data from the buffered input data; S203 When data is stored in the input buffer: If the row number of the current cache input data is less than the row value of the feature point coordinate, delete it and return to step S202； If the row number of the currently cached input data is equal to the row value of the feature point coordinates, then delete all input data in this row whose column numbers are less than the column value of the feature point coordinates and then receive the next row of data. When receiving the next row of data, retain the input data whose column numbers are greater than or equal to the column value of the feature point coordinates, continue to cache the remaining data until the row of data is received, and then stop receiving data. The valid data is extracted from the buffered input data.

8. The method according to claim 7, characterized in that The coordinates of the valid data are (x k ,y k )、(x k +1,y k )、(x k ,y k +1)、(x k +1,y k +1), where x k and k are the column value and row value of the feature point coordinates respectively.

9. The method according to claim 5, characterized in that Before the descriptor calculation is performed, the method also includes a step of performing interpolation calculation on the extracted valid data to perform data enhancement.

10. A real-time SuperPoint network feature extraction and description device, characterized in that: include: SuperPoint convolutional neural network module, used to extract features from input images and generate a first feature tensor for feature point extraction and a second feature tensor for descriptor calculation; A feature point extraction module, used for extracting feature point coordinates based on the first feature tensor, and outputting the feature point coordinates and scanning coordinates in real time; A descriptor calculation module is used to obtain the feature point coordinates and the scanning coordinates in real time, and select tensor data at corresponding positions from the second feature tensor according to the feature point coordinates and the scanning coordinates to perform descriptor calculation.

Images