On-chip convolutional neural network computing system based on data flow driving
By designing an on-chip convolutional neural network computing system driven by data flow, the problem of limited application in the prior art in environments with strict power and resource constraints is solved, and efficient and real-time convolutional neural network computing is realized, which is suitable for the deployment of embedded systems.
Patent Information
- Application Number
- CN202411933694.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-26
- Publication Date
- 2025-05-02
AI Technical Summary
The existing convolutional neural network computing systems are limited in environments such as mobile robots with strict power and resource constraints. Traditional practices require off-chip memory, which increases hardware cost and power consumption, making it difficult to achieve the operation of real-time embedded systems.
A on-chip convolutional neural network computing system based on data flow driver is designed. Through the pipeline design of shared encoder, feature point decoder and descriptor sub-decoder, the stream interface and cache mechanism are used to achieve high parallelism and efficient computing, avoiding dependence on external memory.
It realizes efficient convolutional neural network computing, reduces hardware costs, improves computing efficiency and real-time performance, and is suitable for the deployment of embedded systems.
Smart Images
Figure CN119918598A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to an on-chip convolutional neural network computing system driven by data streams. Background Art
[0002] Feature points refer to locations with obvious changes in an image, usually local areas with good distinguishing ability, and can exist stably under different scales and rotation transformations. Descriptors are methods of numerically describing feature points, usually represented by vectors or matrices, to describe the local area around feature points.
[0003] Feature point extraction and descriptor calculation are important components of SLAM. In recent years, convolutional neural networks have been used for feature point extraction to improve the accuracy of feature point extraction. As one of the most advanced feature point extraction methods, SuperPoint has a self-supervised learning framework, isomorphic adaptation, to enhance rich point detection and robust descriptors. It includes the feature point detection step and the descriptor generation step in a single forward pass, surpassing the traditional manual method in accuracy.
[0004] However, the SuperPoint algorithm is usually deployed on a CPU or GPU platform, and its power consumption is too high, which limits its application in mobile robots, which have strict power and resource constraints. In particular, for convolutional neural network calculations, the traditional approach (such as Xilinx's deep learning processing unit DPU) is to calculate each convolution layer separately, and then calculate the next layer after completing the calculation of one layer, until the last convolution calculation is completed. This requires off-chip memory such as DDR, which not only increases hardware costs, but also has a huge demand for bandwidth. The complexity of calculation and storage makes it difficult for convolutional neural networks to run on real-time embedded systems, and frequent reading and writing of DDR not only increases power consumption, but also increases processing delays. Feature point extraction and descriptor calculation generally extract feature points first, and then perform descriptor calculations based on feature points. The order of processing these two parts makes parallel processing difficult, which in turn increases processing delays and reduces real-time performance.
[0005] Therefore, there is an urgent need to develop a real-time device that does not require off-chip memory for convolutional neural network calculations and can process feature point extraction and descriptor calculations with high parallelism. Summary of the invention
[0006] The technical problem to be solved by the present invention is to provide an on-chip convolutional neural network computing system driven by data flow, which can effectively realize and accelerate on-chip convolutional neural network computing.
[0007] The technical solution adopted by the present invention to solve the technical problem is: to provide an on-chip convolutional neural network computing system based on data flow drive, comprising:
[0008] A convolutional neural network module is used to extract features from an input image and generate a first feature tensor for feature point extraction and a second feature tensor for descriptor calculation; the convolutional neural network module includes a shared encoder, a feature point decoder and a descriptor decoder composed of a plurality of convolutional layers, a stream interface is used for data transmission between the convolutional layers, and convolutional layer data processing driven by data stream is implemented based on a cache mechanism;
[0009] A feature point extraction module, used for extracting feature point coordinates based on the first feature tensor, and outputting the feature point coordinates and scanning coordinates in real time;
[0010] A descriptor calculation module is used to obtain the feature point coordinates and the scanning coordinates in real time, and select tensor data at corresponding positions from the second feature tensor according to the feature point coordinates and the scanning coordinates to perform descriptor calculation.
[0011] Furthermore, each convolutional layer includes a stream input interface, an input buffer, a convolution and pooling processing module, an output buffer and a stream output interface.
[0012] Furthermore, the convolutional layer implements convolutional layer data processing driven by data flow by the following method:
[0013] Monitor the status of the input buffer and the convolution and pooling processing module: if the input buffer is not full, continue to receive data, otherwise stop receiving data; if the convolution and pooling processing module is idle and there is data in the input buffer, push the input buffer data to the convolution and pooling processing module for processing to obtain the calculation result;
[0014] Monitor the status of the output cache: If the output cache is not full, write the calculation results of the convolution and pooling processing modules into the output cache until all the calculation results are written into the output cache, and then the convolution and pooling processing returns to the idle state and prepares for the next calculation. If the output cache is full, stop writing until it is detected that the output cache is not full. If the output cache is not empty and the next layer module requests to send data, push the output cache data to the next layer. If the output cache is empty or the next layer module refuses to receive data, stop outputting the output cache data.
[0015] Furthermore, when performing convolution calculation, the convolution and pooling processing module obtains weight data from the on-chip cache and performs convolution calculation with the received input cache data.
[0016] Furthermore, the input cache of the descriptor calculation module is connected to the output cache of the second convolutional layer of the descriptor decoder through a stream interface. If the input cache of the descriptor calculation module is not full and there is data in the output cache of the second convolutional layer of the descriptor decoder, the second feature tensor is received; otherwise, if the input cache of the descriptor calculation module is full, the output cache of the second convolutional layer of the descriptor decoder stops pushing data.
[0017] Furthermore, the feature point extraction module obtains the first feature tensor from the feature point decoder through a stream interface.
[0018] Furthermore, the computing power of each convolutional layer in the convolutional neural network module is matched.
[0019] Furthermore, the convolutional neural network module includes a total of 12 convolutional layers, among which the 1st to 8th convolutional layers constitute a shared encoder, the 9th to 10th convolutional layers constitute a feature point decoder, and the 11th to 12th convolutional layers constitute a descriptor decoder, and the ratio of the number of multipliers of the 12 convolutional layers is 1:64:16:16:8:16:4:4:8:0.45:8:1.78.
[0020] Furthermore, the convolutional neural network module includes a total of 9 convolutional layers, among which the 1st to 5th convolutional layers constitute a shared encoder, the 6th to 7th convolutional layers constitute a feature point decoder, the 8th to 9th convolutional layers constitute a descriptor decoder, and the ratio of the number of multipliers of the 9 convolutional layers is 1:16:16:8:8:0.45:16:4:8.
[0021] Beneficial Effects
[0022] Due to the adoption of the above technical scheme, the present invention has the following advantages and positive effects compared with the prior art: the present invention optimizes the computing power of the convolutional neural network pipeline calculation and minimizes the processing delay by allocating the number of multipliers to each convolutional layer in proportion. Each convolutional layer can adaptively complete the data calculation and processing of the convolutional layer through the data flow driving method, thereby achieving high computing efficiency. In addition, no data interaction with the outside of the device is required during the convolution calculation process, so no external memory such as DDR is required, thereby reducing hardware costs. The present invention can independently operate to obtain the feature points and descriptors of the image by receiving grayscale image data through a stream interface, and send them to the outside of the device through a stream output interface for use by other modules, which can completely separate from the processor to achieve acceleration. At the same time, due to its low computing complexity, it provides the possibility of its deployment in an embedded system. The present invention realizes parallel processing of feature point extraction and descriptor calculation by utilizing the scanning coordinates of feature point extraction and the feature point coordinates to drive the descriptor calculation, combining data flow driving and caching mechanism, thereby improving real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 is a schematic diagram of the device structure of an embodiment of the present invention;
[0024] Figure 2 Schematic diagram of the structure of the convolutional layer used in the embodiment of the present invention;
[0025] Figure 3 Schematic diagram of the non-maximum suppression processing sequence in an embodiment of the present invention. DETAILED DESCRIPTION
[0026] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.
[0027] An embodiment of the present invention relates to an on-chip convolutional neural network computing system driven by data flow, which includes the following functional structure modules: an interface module, a convolutional neural network module, a feature point extraction module, and a descriptor computing module.
[0028] The interface of the present invention has two categories: one is the APB protocol interface, which is used for parameter configuration, weight data transmission and status information reporting, wherein the weight data only needs to be transmitted once and is stored in RAM internally; the other is the stream interface, which is used for the transmission of large amounts of data, and the stream interface is divided into input and output. The stream interface can be an interface similar to AXI4-Stream, or it can be other customized interfaces. Since the stream interface does not require an address, data transmission is simple and efficient, high-speed, and real-time, which improves the data processing speed and efficiency.
[0029] The convolutional neural network module can use the Superpoint convolutional neural network. The shared encoder structure of the Superpoint convolutional neural network is similar to the convolution structure of the VGG network, which is the shared encoder convolution layer 1-8, which extracts low-level geometric detection and describes shared features to obtain the output tensor A∈R (H / 8)×(W / 8)×128 The feature point decoder maps tensor A to tensor B∈R through two feature point decoder convolutional layers 1 and 2. (H / 8)×(W / 8)×65 , perform feature point detection. The descriptor decoder maps the tensor A to a coarse descriptor tensor C∈R through two convolutional layers 1 and 2. (H / 8)×(W / 8)×256 , and generate descriptors.
[0030] In some embodiments, the convolutional neural network module can also use a lightweight convolutional neural network, whose number of layers is reduced from 12 layers of SuperPoint to 9 layers, reducing the computational complexity while maintaining the accuracy similar to the traditional extraction scheme. The lightweight convolutional neural network uses a single shared encoder to process and reduce the dimension of the input image. After the encoder, it is divided into two decoders, one for feature point detection and the other for feature point description. The shared encoder structure is similar to the convolution structure of the VGG network, which are shared encoder convolution layers 1-4, which extract low-level geometric detection and description shared features to obtain the output tensor A∈R (H / 4)×(W / 4)×128 The feature point decoder maps tensor A to tensor B∈R through two feature point decoder convolutional layers 1 and 2. (H / 8)×(W / 8)×65 , perform feature point detection. The descriptor decoder maps the tensor A to a coarse descriptor tensor C∈R through three convolutional layers 1-3 (H / 8)×(W / 8)×256 , and generate descriptors.
[0031] Each convolutional layer of the convolutional neural network module adopts a pipeline design, and the stream interface is used to transfer data between each convolutional layer. In order to achieve high computing efficiency, the computing power of each convolutional layer should be matched, that is, the time for each convolutional layer to complete the calculation of a single frame of data should be as equal as possible, so as to maximize the performance of the pipeline design. Therefore, the computing power is allocated according to the number of multiplications of each convolutional layer:
[0032] For the Superpoint convolutional neural network, the ratio of the number of multipliers in the 12 convolutional layers from the shared encoder convolutional layer 1 to the descriptor decoder convolutional layer 2 is 1:64:16:16:8:16:4:4:8:0.45:8:1.78, where the ratio of the feature point decoder convolutional layer 2 and the descriptor decoder convolutional layer 2 is a decimal. When the number of multipliers is used, the final number of multipliers is rounded up. For example, when using a 9-fold ratio, the number of multipliers used in each convolutional layer is 9, 576, 144, 144, 72, 144, 36, 36, 72, 5, 72, 17, respectively. By allocating computing resources through this ratio, the computing power of each convolutional layer can be optimized and the processing delay can be minimized.
[0033] For lightweight convolutional neural networks, the ratio of the number of multipliers in the 9 convolutional layers from the shared encoder convolutional layer 1 to the descriptor decoder convolutional layer 2 is 1:16:16:8:8:0.45:16:4:8, where the ratio value of the feature point decoder convolutional layer 2 is a decimal. When the number of multipliers is used, the final number of multipliers is rounded up. For example, when using a 9-fold ratio, the number of multipliers used in each convolutional layer is 9, 144, 144, 72, 72, 5, 144, 36, 72, respectively. By allocating computing resources through this ratio, the computing power of each convolutional layer can be optimized and the processing delay can be minimized.
[0034] The convolutional layers use stream interfaces to transfer data, and are driven by data streams to complete data calculation and processing of each convolutional layer. Each convolutional layer consists of five functional parts: stream input interface, input buffer, convolution and pooling processing, output buffer, and stream output interface. The specific implementation steps of data processing driven by data streams are as follows:
[0035] Step 1: Check the status of the input buffer. If the input buffer is not full, continue to receive data. If the input buffer is full, stop receiving data.
[0036] Step 2: If the convolution and pooling processing is idle and there is data in the input cache, the input cache data is pushed to the convolution and pooling for processing to obtain the calculation result; during the convolution calculation process, the input cache data and weight data are convolved, and the weight data is also stored in the on-chip cache. The calculation process does not require any data interaction with the outside of the network feature extraction and description device, and no external memory is required.
[0037] Step 3: Check the status of the output buffer. If the output buffer is not full, the convolution calculation or maximum pooling calculation results are written to the output buffer. If the output buffer is full, the calculation results stop being written to the output buffer. After all the results of this calculation are written to the output buffer, the convolution and pooling processing returns to the idle state and prepares for the next calculation.
[0038] Step 4: Check the status of the output buffer. If the output buffer is not empty and the next-layer module requests to send data, push the output buffer data to the next layer; if the output buffer is empty or the next-layer module refuses to receive data, the output buffer data stops being output.
[0039] The data stream is used to drive input reception, convolution and pooling processing, and output push. There is no need for complex control. Through the streaming data drive, the calculation and processing of each convolution layer are adaptively completed. Then, through the optimal computing power allocated in proportion, ideally, it can achieve: always receiving data, always calculating data, and always outputting data, achieving high computing efficiency.
[0040] The feature point extraction module and the descriptor calculation module also use a similar method to interact with the convolutional neural network module. Taking the descriptor calculation module as an example, its internal input buffer is connected to the output buffer of the sub-decoder convolution layer 2 through a stream interface, and the interaction between them is carried out through the ready signal at the receiving end and the valid signal at the sending end. When the input buffer is not full, the ready signal is pulled high to prepare to receive data. If there is data in the output buffer of the sub-decoder convolution layer 2 at this time, the valid signal is pulled high. At this time, they are both high and the received data is valid. When the input buffer is full, the ready signal is pulled low, and the output buffer of the sub-decoder convolution layer 2 stops outputting.
[0041] The feature point extraction and descriptor calculation of the network post-processing are processed in parallel. The feature point decoder convolutional layer decodes the feature map and obtains the tensor B∈R (H / 8)×(W / 8)×65 The output of , where the number of channels 65 corresponds to the non-overlapping local 8×8 pixel block area in the input image, and 1 background channel representing no feature points. In the feature point extraction module, the tensor B is first processed by Softmax to obtain the initial filtered feature points. Then the output is converted from R to (H / 8)×(W / 8)×64 Convert tensor back to R H×W×1 Tensor, get the feature point probability map. Then use the non-maximum suppression (NMS) algorithm, each window only retains one maximum value, then the whole image is thresholded, and the points above the threshold are regarded as feature points, so as to find the best detection position, that is, the feature point coordinates (x k ,y k ).
[0042] While extracting feature points, the descriptor calculation is also performed in parallel. It receives the descriptor decoder and maps the tensor A to the coarse descriptor tensor C∈R through two convolutional layers. (H / 8)×(W / 8)×256 In order to reduce the amount of calculation, the tensor data at the corresponding position is selected according to the feature point coordinates for descriptor calculation, instead of calculating the entire tensor C∈R (H / 8)×(W / 8)×256 The tensor data at the corresponding position is linearly interpolated to obtain the complete descriptor, and finally L2 normalization is used to obtain a 256-dimensional descriptor vector of unit length.
[0043] Before the feature points are obtained, the descriptor module cannot extract the tensor data of the corresponding position because it does not have the feature point coordinates, causing the descriptor calculation to stop. This will cause the data of the descriptor decoder convolution layer 2 to accumulate, affecting parallelism. At the same time, it will also increase the output cache of the descriptor decoder convolution layer 2, increasing the consumption of storage resources.
[0044] According to the principle that target detection is processed from left to right and from top to bottom in the order of images, when NMS processes each 8×8 square window and scans the last data in the lower right corner, the coordinates (x c ,y c ), which is called scanning coordinates here. In this way, no matter whether there are feature points in the 8×8 square, the scanning coordinates can be output in order, so that the descriptor module can receive data synchronously, and does not have to wait until there are feature points to receive data. At the same time, since the scanning coordinates are output only when the last data in the lower right corner is output, if there are feature points at this time, they have been output before the scanning coordinates. Therefore, in an 8×8 square, the feature point coordinates are output before the scanning coordinates. By outputting the scanning coordinates, the data reading of the descriptor calculation is processed in a coordinated manner, the data accumulation of the convolution layer is reduced, and the calculation efficiency is improved.
[0045] The specific implementation steps of the description submodule for data processing based on the scanning coordinates and feature point coordinates are as follows:
[0046] Step 1: Check whether there are feature point coordinates and scanning coordinates. If there are none, the description submodule does not receive data and continues to wait.
[0047] Step 2: When there are scan coordinates but no feature point coordinates, delete the invalid input data.
[0048] Step 2.1: If there is no data in the input buffer, start data reception;
[0049] Step 2.1.1: If the row number of the input data is less than the row y of the scan coordinates c value, delete the input data of this row and continue to receive new data;
[0050] Step 2.1.2: If the row number of the input data is equal to the row y of the scan coordinates c value, and then determine the column number of the input row. If the column number is less than the column x of the scan coordinate c value, delete all the values in this row that are less than x c The column input data continues to receive new data; when the column number is equal to the column x of the scan coordinate c When the value is reached, stop receiving data;
[0051] Step 2.2: If there is data in the input buffer, delete the invalid input data;
[0052] Step 2.2.1: If the row number of the cached input data is less than the row y of the scan coordinate c If the value is not found, delete the input data of this row and return to step 2.1 to continue execution;
[0053] Step 2.2.2: If the row number of the cache input data is equal to the row y of the scan coordinatec value, delete all the values in this row that are less than x c After receiving the input data of the column, stop receiving data;
[0054] Step 2.2.3: If the row number of the cache input data is greater than the row y of the scan coordinate c value, the input data is retained, and receiving data stops;
[0055] Step 3: When feature point coordinates are available, extract valid data from the input data.
[0056] Step 3.1: If there is no data in the input buffer, start data reception;
[0057] Step 3.1.1: If the row number of the input data is less than the row y of the feature point coordinates k value, delete the input data of this row and continue to receive new data;
[0058] Step 3.1.2: If the row number of the input data is equal to the row y of the feature point coordinates k value, and then determine the column number of the input row. If the column number is less than the column x of the feature point coordinates k value, delete all the values in this row that are less than x k The column input data continues to receive new data; when the column number is equal to the column x of the feature point coordinates k When a value is reached, the input data at that time is saved and the remaining data of the row continues to be cached;
[0059] Step 3.1.3: If the row number of the input data is equal to the row y of the feature point coordinates k +1 value, then keep the column number greater than or equal to the feature point coordinate column x k The input data of the value is received, and the remaining data of the row is cached. After the data of the row is received, the receiving data is stopped; at this time, extract (x k ,y k ), (x k +1,y k ), (x k ,y k +1), (x k +1,y k +1) The data of these four points are interpolated;
[0060] Step 3.2: If there is data in the input buffer, delete the invalid data and keep the valid data;
[0061] Step 3.2.1: If the row number of the cache input data is less than the row y of the feature point coordinates k If the value is not found, delete the input data of this row and return to step 3.1 to continue execution;
[0062] Step 3.2.2: If the row number of the cache input data is equal to the row y of the feature point coordinates k value, delete all the values in this row that are less than x k After receiving the input data of the column, receive the data of the next row; when receiving the data of the next row, retain the column number greater than or equal to the feature point coordinate column x k The input data of the value is received, and the remaining data of the row is cached. After the data of the row is received, the receiving data is stopped; at this time, extract (x k ,y k ), (x k +1,y k ), (x k ,y k +1), (x k +1,y k +1) The data of these four points are interpolated.
[0063] The data reception of the descriptor module is driven by scanning coordinates and feature point coordinates, and valid data corresponding to the feature point coordinates are extracted and invalid data is deleted, thereby reducing data cache and calculation amount. The collaborative processing of the present invention enables feature point extraction and descriptor calculation to be performed in parallel, thereby improving the real-time performance of the network feature extraction and description device.
[0064] The present implementation is further described below in conjunction with the accompanying drawings and two specific embodiments.
[0065] Example 1: SuperPoint network feature extraction and description.
[0066] SuperPoint network feature extraction and description device Figure 1 As shown, the functional structure includes: interface module, convolutional neural network module, feature point extraction module, and descriptor calculation module.
[0067] SuperPoint feature extraction and description consists of a common convolutional neural network encoder, feature point detection decoder and descriptor decoder, and transmits data to the outside through an interface. There are two types of interfaces for external communication between the embodiment of the present invention: one is the APB protocol interface, which is used for parameter configuration, weight data delivery and status information reporting, where the weight data only needs to be delivered once and is stored in RAM internally; the other is the stream interface, which is used for the transmission of large amounts of data. The stream interface is divided into input and output. In the embodiment of the present invention, the stream interface signal definitions involved are shown in Table 1:
[0068] Table 1 Stream interface signal definition
[0069]
[0070] The above stream interface can be an interface similar to AXI4-Stream, or other customized interface, as long as it meets the above signal function and timing requirements. Since the stream interface does not require an address, data transmission is simple and efficient, high-speed, and real-time, which improves data processing speed and efficiency.
[0071] The parameters of the Superpoint convolutional neural network are shown in Table 2. The first number of each convolutional layer is the number of input channels, the middle two numbers are the convolution kernel size, and the last number is the number of output channels. The pooling of the shared encoder convolutional layers 2, 4, and 6 refers to the maximum pooling operation with a step size of 2 after the convolution. The shared encoder structure is similar to the convolution structure of the VGG network. It extracts low-level geometric detection and describes shared features to obtain the output tensor A∈R (H / 8)×(W / 8)×128 The feature point decoder maps tensor A to tensor B∈R through two feature point decoder convolutional layers. (H / 8)×(W / 8)×65 , perform feature point detection. The descriptor decoder maps the tensor A to a coarse descriptor tensor C∈R through two convolutional layers. (H / 8)×(W / 8)×256 , and generate descriptors.
[0072] Table 2 Convolutional neural network parameters
[0073]
[0074] *H refers to the vertical resolution of the image, W refers to the horizontal resolution of the image
[0075] The embodiment of the present invention performs pipeline design on each convolution layer, and uses a stream interface to transfer data between each convolution layer. In order to achieve high computing efficiency, the computing power of each convolution layer should be matched, that is, the time for each convolution layer to complete the calculation of a single frame of data should be as equal as possible, so as to exert the maximum performance of the pipeline design. Therefore, the computing power is allocated according to the number of multiplications of each convolution layer in Table 2. The ratio of the number of multipliers of the 12 convolution layers from the shared encoder convolution layer 1 to the descriptor decoder convolution layer 2 is 1:64:16:16:8:16:4:4:8:0.45:8:1.78, where the ratio value of the feature point decoder convolution layer 2 and the descriptor decoder convolution layer 2 is a decimal. When the number of multipliers is used, the final number of multipliers is rounded up. For example, when a 9-fold ratio is used, the number of multipliers used in each convolutional layer is 9, 576, 144, 144, 72, 144, 36, 36, 72, 5, 72, and 17. By allocating computing resources through this ratio, the computing power of each convolutional layer can be optimized and the processing delay can be minimized.
[0076] The embodiment of the present invention uses a stream interface to transfer data between convolutional layers, and drives the data stream to complete the data calculation and processing of each convolutional layer. Figure 2 shown.
[0077] Each convolution layer of the embodiment of the present invention is composed of five functional parts: stream input interface, input buffer, convolution and pooling processing, output buffer, and stream output interface. The specific implementation steps of the convolution layer data processing driven by the data stream are as follows:
[0078] Step 1: Check the status of the input buffer. If the input buffer is not full, continue to receive data. If the input buffer is full, stop receiving data.
[0079] Step 2: If the convolution and pooling processing is idle and there is data in the input cache, the input cache data is pushed to the convolution and pooling for processing to obtain the calculation result; during the convolution calculation process, the input cache data and weight data are convolved, and the weight data is also stored in the on-chip cache. The calculation process does not require any data interaction with the outside of the SuperPoint network feature extraction and description device, and no external memory is required.
[0080] Step 3: Check the status of the output buffer. If the output buffer is not full, the convolution calculation or maximum pooling calculation results are written to the output buffer. If the output buffer is full, the calculation results stop being written to the output buffer. After all the results of this calculation are written to the output buffer, the convolution and pooling processing returns to the idle state and prepares for the next calculation.
[0081] Step 4: Check the status of the output buffer. If the output buffer is not empty and the next-layer module requests to send data, push the output buffer data to the next layer; if the output buffer is empty or the next-layer module refuses to receive data, the output buffer data stops being output.
[0082] The embodiment of the present invention uses data stream to drive input reception, convolution and pooling processing, and output push, without complex control. It adaptively completes the calculation processing of each convolution layer through stream data drive. Then, through the optimal computing power allocated in proportion, it can ideally achieve: always receiving data, always calculating data, and always outputting data, achieving high computing efficiency.
[0083] The embodiment of the present invention adopts parallel processing for feature point extraction and descriptor calculation in network post-processing. The feature point decoder convolutional layer decodes the feature map to obtain a tensor B∈R (H / 8)×(W / 8)×65 The output of , where the number of channels 65 corresponds to the non-overlapping local 8×8 pixel block area in the input image, and 1 background channel representing no feature points. In the feature point extraction module, the tensor B is first processed by Softmax to obtain the initial filtered feature points. Then the output is converted from R to (H / 8)×(W / 8)×64 Convert tensor back to R H×W×1Tensor, get the feature point probability map. Then use the non-maximum suppression (NMS) algorithm, each window only retains one maximum value, then the whole image is thresholded, and the points above the threshold are regarded as feature points, so as to find the best detection position, that is, the feature point coordinates (x k ,y k ).
[0084] While extracting feature points, the descriptor calculation is also performed in parallel. It receives the descriptor decoder and maps the tensor A to the coarse descriptor tensor C∈R through two convolutional layers. (H / 8)×(W / 8)×256 In order to reduce the amount of calculation, the present invention selects the tensor data at the corresponding position according to the feature point coordinates to perform descriptor calculation, instead of calculating the entire tensor C∈R (H / 8)×(W / 8)×256 The tensor data at the corresponding position is linearly interpolated to obtain the complete descriptor, and finally L2 normalization is used to obtain a 256-dimensional descriptor vector of unit length.
[0085] Before the feature points are obtained, the descriptor module cannot extract the tensor data of the corresponding position because it does not have the feature point coordinates, causing the descriptor calculation to stop. This will cause the data of the descriptor decoder convolution layer 2 to accumulate, affecting parallelism. At the same time, it will also increase the output cache of the descriptor decoder convolution layer 2, increasing the consumption of storage resources.
[0086] According to the principle that target detection is processed from left to right and from top to bottom in the order of images, when NMS processes each square window, the coordinates (x c ,y c ) output, which is called scanning coordinates here. By outputting scanning coordinates, the data reading of descriptor calculation is processed in a coordinated manner, which reduces the data accumulation of the convolution layer and improves the calculation efficiency. The processing process of scanning coordinates is shown in Figure 3 shown.
[0087] In the 8×8 grid, when the last data in the lower right corner is scanned, the coordinates (x c ,y c ). In this way, no matter whether there are feature points in the 8×8 square, the scanning coordinates can be output in order, so that the description submodule can receive data synchronously, and does not have to wait until there are feature points to receive data. At the same time, since the scanning coordinates are output only when the last data in the lower right corner is output, if there are feature points at this time, they have been output before the scanning coordinates. Therefore, in an 8×8 square, the feature point coordinates are output before the scanning coordinates.
[0088] The specific implementation steps of the description submodule for data processing based on the scanning coordinates and feature point coordinates are as follows:
[0089] Step 1: Check whether there are feature point coordinates and scanning coordinates. If there are none, the description submodule does not receive data and continues to wait.
[0090] Step 2: When there are scan coordinates but no feature point coordinates, delete the invalid input data.
[0091] Step 2.1: If there is no data in the input buffer, start data reception;
[0092] Step 2.1.1: If the row number of the input data (from 0 to H-1) is less than the row y of the scan coordinate c value, delete the input data of this row and continue to receive new data;
[0093] Step 2.1.2: If the row number of the input data is equal to the row y of the scan coordinates c value, and then determine the column number of the input row. If the column number is less than the column x of the scan coordinate c value, delete all the values in this row that are less than x c The column input data continues to receive new data; when the column number is equal to the column x of the scan coordinate c When the value is reached, stop receiving data;
[0094] Step 2.2: If there is data in the input buffer, delete the invalid input data;
[0095] Step 2.2.1: If the row number of the cache input data (from 0 to H-1) is less than the row y of the scan coordinate c If the value is not found, delete the input data of this row and return to step 2.1 to continue execution;
[0096] Step 2.2.2: If the row number of the cache input data is equal to the row y of the scan coordinate c value, delete all the values in this row that are less than x c After receiving the input data of the column, stop receiving data;
[0097] Step 2.2.3: If the row number of the cache input data is greater than the row y of the scan coordinate c value, the input data is retained, and receiving data stops;
[0098] Step 3: When feature point coordinates are available, extract valid data from the input data.
[0099] Step 3.1: If there is no data in the input buffer, start data reception;
[0100] Step 3.1.1: If the row number of the input data is less than the row y of the feature point coordinates k value, delete the input data of this row and continue to receive new data;
[0101] Step 3.1.2: If the row number of the input data is equal to the row y of the feature point coordinates k value, and then determine the column number of the input row. If the column number is less than the column x of the feature point coordinates k value, delete all the values in this row that are less than x k The column input data continues to receive new data; when the column number is equal to the column x of the feature point coordinates k When a value is reached, the input data at that time is saved and the remaining data of the row continues to be cached;
[0102] Step 3.1.3: If the row number of the input data is equal to the row y of the feature point coordinates k +1 value, then keep the column number greater than or equal to the feature point coordinate column x k The input data of the value is received, and the remaining data of the row is cached. After the data of the row is received, the receiving data is stopped; at this time, extract (x k ,y k ), (x k +1,y k ), (x k ,y k +1), (x k +1,y k +1) The data of these four points are interpolated;
[0103] Step 3.2: If there is data in the input buffer, delete the invalid data and keep the valid data;
[0104] Step 3.2.1: If the row number of the cache input data is less than the row y of the feature point coordinates k If the value is not found, delete the input data of this row and return to step 3.1 to continue execution;
[0105] Step 3.2.2: If the row number of the cache input data is equal to the row y of the feature point coordinates k value, delete all the values in this row that are less than x k After receiving the input data of the column, receive the data of the next row; when receiving the data of the next row, retain the column number greater than or equal to the feature point coordinate column x k The input data of the value is received, and the remaining data of the row is cached. After the data of the row is received, the receiving data is stopped; at this time, extract (x k ,y k ), (x k +1,y k ), (x k ,y k +1), (x k +1,y k +1) The data of these four points are interpolated.
[0106] Example 2: Lightweight network feature extraction and description.
[0107] Lightweight network feature extraction and description device such as Figure 1 As shown, the functional structure includes: interface module, convolutional neural network module, feature point extraction module, and descriptor calculation module.
[0108] There are two types of interfaces for the device of the present invention to communicate with the outside: one is the APB protocol interface, which is used for parameter configuration, weight data delivery and status information reporting, wherein the weight data only needs to be delivered once and is stored in RAM internally; the other is the stream interface, which is used for the transmission of large amounts of data. The stream interface is divided into input and output. In the embodiment of the present invention, the stream interface signal definitions involved are shown in Table 3:
[0109] Table 3 Stream interface signal definition
[0110]
[0111]
[0112] The above stream interface can be an interface similar to AXI4-Stream, or other customized interface, as long as it meets the above signal function and timing requirements. Since the stream interface does not require an address, data transmission is simple and efficient, high-speed, and real-time, which improves data processing speed and efficiency.
[0113] The parameters of the lightweight convolutional neural network are shown in Table 4. A single shared encoder is used to process and reduce the dimension of the input image. After the encoder, it is divided into two decoders, one for feature point detection and the other for feature point description. The first number of each convolution layer is the number of input channels, the middle two numbers are the convolution kernel size, and the last number is the number of output channels. The shared encoder convolution layers 2 and 4 are convolutions with a stride of 2 to complete downsampling. The shared encoder structure is similar to the convolution structure of the VGG network. It extracts low-level geometric detection and description shared features to obtain the output tensor A∈R (H / 4)×(W / 4)×128 The feature point decoder maps tensor A to tensor B∈R through two feature point decoder convolutional layers. (H / 8)×(W / 8)×65 , perform feature point detection, where the stride of the feature point decoder convolution layer 1 is 2. The descriptor decoder maps the tensor A to a coarse descriptor tensor C∈R through three convolutional layers (H / 8)×(W / 8)×256 , descriptor generation is performed, where the stride of the descriptor decoder convolution layer 2 is 2. The number of lightweight convolutional neural network layers is reduced from 12 layers of SuperPoint to 9 layers, which reduces the computational complexity while maintaining the accuracy similar to the traditional extraction scheme.
[0114] Table 4 Convolutional neural network parameters
[0115]
[0116] *H refers to the vertical resolution of the image, W refers to the horizontal resolution of the image
[0117] The embodiment of the present invention performs pipeline design on each convolution layer, and uses a stream interface to transfer data between each convolution layer. In order to achieve high computing efficiency, the computing power of each convolution layer should be matched, that is, the time for each convolution layer to complete the calculation of a single frame of data should be as equal as possible, so as to exert the maximum performance of the pipeline design. Therefore, the computing power is allocated according to the number of multiplications of each convolution layer in Table 2. The ratio of the number of multipliers of the 9 convolution layers from the shared encoder convolution layer 1 to the descriptor decoder convolution layer 2 is 1:16:16:8:8:0.45:16:4:8, where the ratio value of the feature point decoder convolution layer 2 is a decimal. When the number of multipliers is used, the final number of multipliers is obtained by rounding up. For example, when using a 9-fold ratio, the number of multipliers used in each convolution layer is 9, 144, 144, 72, 72, 5, 144, 36, 72, respectively. By allocating computing resources through this proportional relationship, the computing power of each convolutional layer can be optimized and the processing delay can be minimized.
[0118] The embodiment of the present invention uses a stream interface to transfer data between convolutional layers, and drives the data stream to complete the data calculation and processing of each convolutional layer. Figure 2 shown.
[0119] Each convolution layer of the embodiment of the present invention is composed of five functional parts: stream input interface, input buffer, convolution and pooling processing, output buffer, and stream output interface. The specific implementation steps of the convolution layer data processing driven by the data stream are as follows:
[0120] Step 1: Check the status of the input buffer. If the input buffer is not full, continue to receive data. If the input buffer is full, stop receiving data.
[0121] Step 2: If the convolution and pooling processing is idle and there is data in the input cache, the input cache data is pushed to the convolution and pooling for processing to obtain the calculation result; during the convolution calculation process, the input cache data and weight data are convolved, and the weight data is also stored in the on-chip cache. The calculation process does not require any data interaction with the outside of the network feature extraction and description device, and no external memory is required.
[0122] Step 3: Check the status of the output buffer. If the output buffer is not full, the convolution calculation or maximum pooling calculation results are written to the output buffer. If the output buffer is full, the calculation results stop being written to the output buffer. After all the results of this calculation are written to the output buffer, the convolution and pooling processing returns to the idle state and prepares for the next calculation.
[0123] Step 4: Check the status of the output buffer. If the output buffer is not empty and the next-layer module requests to send data, push the output buffer data to the next layer; if the output buffer is empty or the next-layer module refuses to receive data, the output buffer data stops being output.
[0124] The embodiment of the present invention uses data stream to drive input reception, convolution and pooling processing, and output push, without complex control. It adaptively completes the calculation processing of each convolution layer through stream data drive. Then, through the optimal computing power allocated in proportion, it can ideally achieve: always receiving data, always calculating data, and always outputting data, achieving high computing efficiency.
[0125] The embodiment of the present invention adopts parallel processing for feature point extraction and descriptor calculation in network post-processing. The feature point decoder convolutional layer decodes the feature map to obtain a tensor B∈R (H / 8)×(W / 8)×65 The output of , where the number of channels 65 corresponds to the non-overlapping local 8×8 pixel block area in the input image, and 1 background channel representing no feature points. In the feature point extraction module, the tensor B is first processed by Softmax to obtain the initial filtered feature points. Then the output is converted from R to (H / 8)×(W / 8)×64 Convert tensor back to R H×W×1 Tensor, get the feature point probability map. Then use the non-maximum suppression (NMS) algorithm, each window only retains one maximum value, then the whole image is thresholded, and the points above the threshold are regarded as feature points, so as to find the best detection position, that is, the feature point coordinates (x k ,y k ).
[0126] While extracting feature points, the descriptor calculation is also performed in parallel. It receives the descriptor decoder and maps the tensor A to the coarse descriptor tensor C∈R through two convolutional layers. (H / 8)×(W / 8)×256 In order to reduce the amount of calculation, the present invention selects the tensor data at the corresponding position according to the feature point coordinates to perform descriptor calculation, instead of calculating the entire tensor C∈R (H / 8)×(W / 8)×256 The tensor data at the corresponding position is linearly interpolated to obtain the complete descriptor, and finally L2 normalization is used to obtain a 256-dimensional descriptor vector of unit length.
[0127] Before the feature points are obtained, the descriptor module cannot extract the tensor data of the corresponding position because it does not have the feature point coordinates, causing the descriptor calculation to stop. This will cause the data of the descriptor decoder convolution layer 2 to accumulate, affecting parallelism. At the same time, it will also increase the output cache of the descriptor decoder convolution layer 2, increasing the consumption of storage resources.
[0128] According to the principle that target detection is processed from left to right and from top to bottom in the order of images, when NMS processes each square window, the coordinates (x c ,y c ) output, which is called scanning coordinates here. By outputting scanning coordinates, the data reading of descriptor calculation is processed in a coordinated manner, which reduces the data accumulation of the convolution layer and improves the calculation efficiency. The processing process of scanning coordinates is shown in Figure 3 shown.
[0129] In the 8×8 grid, when the last data in the lower right corner is scanned, the coordinates (x c ,y c ). In this way, no matter whether there are feature points in the 8×8 square, the scanning coordinates can be output in order, so that the description submodule can receive data synchronously, and does not have to wait until there are feature points to receive data. At the same time, since the scanning coordinates are output only when the last data in the lower right corner is output, if there are feature points at this time, they have been output before the scanning coordinates. Therefore, in an 8×8 square, the feature point coordinates are output before the scanning coordinates.
[0130] The specific implementation steps of the description submodule for data processing based on the scanning coordinates and feature point coordinates are as follows:
[0131] Step 1: Check whether there are feature point coordinates and scanning coordinates. If there are none, the description submodule does not receive data and continues to wait.
[0132] Step 2: When there are scan coordinates but no feature point coordinates, delete the invalid input data.
[0133] Step 2.1: If there is no data in the input buffer, start data reception;
[0134] Step 2.1.1: If the row number of the input data (from 0 to H-1) is less than the row y of the scan coordinate c value, delete the input data of this row and continue to receive new data;
[0135] Step 2.1.2: If the row number of the input data is equal to the row y of the scan coordinates c value, and then determine the column number of the input row. If the column number is less than the column x of the scan coordinate c value, delete all the values in this row that are less than x cThe column input data continues to receive new data; when the column number is equal to the column x of the scan coordinate c When the value is reached, stop receiving data;
[0136] Step 2.2: If there is data in the input buffer, delete the invalid input data;
[0137] Step 2.2.1: If the row number of the cache input data (from 0 to H-1) is less than the row y of the scan coordinate c If the value is not found, delete the input data of this row and return to step 2.1 to continue execution;
[0138] Step 2.2.2: If the row number of the cache input data is equal to the row y of the scan coordinate c value, delete all the values in this row that are less than x c After receiving the input data of the column, stop receiving data;
[0139] Step 2.2.3: If the row number of the cache input data is greater than the row y of the scan coordinate c value, the input data is retained, and receiving data stops;
[0140] Step 3: When feature point coordinates are available, extract valid data from the input data.
[0141] Step 3.1: If there is no data in the input buffer, start data reception;
[0142] Step 3.1.1: If the row number of the input data is less than the row y of the feature point coordinates k value, delete the input data of this row and continue to receive new data;
[0143] Step 3.1.2: If the row number of the input data is equal to the row y of the feature point coordinates k value, and then determine the column number of the input row. If the column number is less than the column x of the feature point coordinates k value, delete all the values in this row that are less than x k The column input data continues to receive new data; when the column number is equal to the column x of the feature point coordinates k When a value is reached, the input data at that time is saved and the remaining data of the row continues to be cached;
[0144] Step 3.1.3: If the row number of the input data is equal to the row y of the feature point coordinates k +1 value, then keep the column number greater than or equal to the feature point coordinate column x k The input data of the value is received, and the remaining data of the row is cached. After the data of the row is received, the receiving data is stopped; at this time, extract (x k ,y k ), (x k +1,y k), (x k ,y k +1), (x k +1,y k +1) The data of these four points are interpolated;
[0145] Step 3.2: If there is data in the input buffer, delete the invalid data and keep the valid data;
[0146] Step 3.2.1: If the row number of the cache input data is less than the row y of the feature point coordinates k If the value is not found, delete the input data of this row and return to step 3.1 to continue execution;
[0147] Step 3.2.2: If the row number of the cache input data is equal to the row y of the feature point coordinates k value, delete all the values in this row that are less than x k After receiving the input data of the column, receive the data of the next row; when receiving the data of the next row, retain the column number greater than or equal to the feature point coordinate column x k The input data of the value is received, and the remaining data of the row is cached. After the data of the row is received, the receiving data is stopped; at this time, extract (x k ,y k ), (x k +1,y k ), (x k ,y k +1), (x k +1,y k +1) The data of these four points are interpolated.
[0148] In summary, the two embodiments of the present invention propose that each convolutional layer allocates the number of multipliers in proportion, so that the computing power of the convolutional neural network pipeline calculation is optimized and the processing delay is minimized. Each convolutional layer adaptively completes the data calculation and processing of the convolutional layer through the data flow driving method, achieves high computing efficiency, and achieves the goal of not requiring an external memory, reducing hardware costs. The scanning coordinates extracted by the feature point and the feature point coordinate drive descriptor calculation are used to achieve parallel processing and improve real-time performance. The embodiment of the present invention can independently operate to obtain the feature points and descriptors of the image by receiving grayscale image data through the stream interface, and can be completely separated from the processor to achieve acceleration. Due to its low computational complexity, it can be deployed in an embedded system. The implementation method of the above-described device and technology in the embodiment of the present invention can be implemented in a digital electronic circuit system, an integrated circuit system, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), and a system on a chip (SOC).
[0149] Although the above discussion includes several specific implementation details, these should not be interpreted as limiting the scope of the present invention. Relevant personnel can make various changes and modifications without departing from the technical idea of the present invention. As long as they do not deviate from the spirit of the present invention, they are all within the protection scope of the present invention.
Claims
1. A data flow driven on-chip convolutional neural network computing system, characterized in that: include: A convolutional neural network module is used to extract features from an input image and generate a first feature tensor for feature point extraction and a second feature tensor for descriptor calculation; the convolutional neural network module includes a shared encoder, a feature point decoder and a descriptor decoder composed of a plurality of convolutional layers, a stream interface is used for data transmission between the convolutional layers, and convolutional layer data processing driven by data stream is implemented based on a cache mechanism; A feature point extraction module, used for extracting feature point coordinates based on the first feature tensor, and outputting the feature point coordinates and scanning coordinates in real time; A descriptor calculation module is used to obtain the feature point coordinates and the scanning coordinates in real time, and select tensor data at corresponding positions from the second feature tensor according to the feature point coordinates and the scanning coordinates to perform descriptor calculation.
2. The system according to claim 1, characterized in that Each convolutional layer includes a stream input interface, an input buffer, a convolution and pooling processing module, an output buffer and a stream output interface.
3. The system according to claim 2, characterized in that The convolutional layer implements convolutional layer data processing driven by data flow through the following method: Monitor the status of the input buffer and the convolution and pooling processing module: if the input buffer is not full, continue to receive data, otherwise stop receiving data; if the convolution and pooling processing module is idle and there is data in the input buffer, push the input buffer data to the convolution and pooling processing module for processing to obtain the calculation result; Monitor the status of the output cache: If the output cache is not full, write the calculation results of the convolution and pooling processing modules to the output cache until all the calculation results are written to the output cache, and then the convolution and pooling processing returns to the idle state and prepares for the next calculation. If the output cache is full, stop writing until the output cache is not full. If the output buffer is not empty and the next layer module requests to send data, the output buffer data is pushed to the next layer; if the output buffer is empty or the next layer module refuses to receive data, the output of the output buffer data is stopped.
4. The system according to claim 3, characterized in that When performing convolution calculation, the convolution and pooling processing module obtains weight data from the on-chip cache and performs convolution calculation with the received input cache data.
5. The system according to claim 1, characterized in that The input cache of the descriptor calculation module is connected to the output cache of the second convolutional layer of the descriptor decoder through a stream interface. If the input cache of the descriptor calculation module is not full and there is data in the output cache of the second convolutional layer of the descriptor decoder, the second feature tensor is received; otherwise, if the input cache of the descriptor calculation module is full, the output cache of the second convolutional layer of the descriptor decoder stops pushing data.
6. The system according to claim 1, characterized in that The feature point extraction module obtains the first feature tensor from the feature point decoder through a stream interface.
7. The system according to claim 1, characterized in that The computing power of each convolutional layer in the convolutional neural network module is matched.
8. The system according to claim 1, characterized in that The convolutional neural network module includes a total of 12 convolutional layers, among which the 1st to 8th convolutional layers constitute a shared encoder, the 9th to 10th convolutional layers constitute a feature point decoder, the 11th to 12th convolutional layers constitute a descriptor decoder, and the ratio of the number of multipliers in the 12 convolutional layers is 1:64:16:16:8:16:4:4:8:0.45:8:1.
78.
9. The system according to claim 1, characterized in that The convolutional neural network module includes a total of 9 convolutional layers, among which the 1st to 5th convolutional layers constitute a shared encoder, the 6th to 7th convolutional layers constitute a feature point decoder, the 8th to 9th convolutional layers constitute a descriptor decoder, and the ratio of the number of multipliers in the 9 convolutional layers is 1:16:16:8:8:0.45:16:4:8.