Hardware acceleration method and system based on neural network feature extraction

By designing a hardware acceleration system based on neural networks and using a pipelined hardware module, the CPU can avoid processing convolutional instructions and post-processing operations, and solve the problems of high energy consumption and redundant calculation of SuperPoint hardware acceleration system, achieving efficient Image feature extraction.

CN118940812AActive Publication Date: 2024-11-12HUAZHONG UNIV OF SCI & TECH

Patent Information

Application Number
CN202410967413.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-12
Estimated Expiration
2044-07-18

AI Technical Summary

Technical Problem

The existing SuperPoint hardware acceleration system has high energy consumption and high resource overhead when processing image feature extraction, and there are redundant post-processing operations and data comparisons, which limits the improvement of energy efficiency.

Method used

A hardware acceleration system based on neural network feature extraction was designed, including a coded convolution processor, feature point convolution processor, SoftMax&NMS module, feature image pixel selector and selective descriptor convolution processor, using pipeline method Run, avoid the CPU processing convolution instructions and post-processing operations, selectively generate descriptors through feature point positions, and reduce redundant calculations.

Benefits of technology

It significantly improves energy efficiency, reduces off-chip memory access and redundant descriptor convolution, reduces hardware overhead, and improves computing energy efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940812A_ABST
    Figure CN118940812A_ABST
Patent Text Reader

Abstract

The invention belongs to the related technical field of image processing, and discloses a hardware acceleration method and system based on neural network feature extraction, and the system comprises a coding convolution processor, a shared feature map cache, a feature point detection structure, a feature image pixel point selector and a descriptor generation structure which are operated according to an assembly line mode. The coding convolution processor carries out convolution on the input image and outputs and stores the input image to the shared feature map cache; the feature point detection structure generates feature points through convolution and post-processing; the feature image pixel point selector generates a pixel region according to the feature point information; the descriptor generation structure only reads feature point related pixel region data from the shared feature map and calculates a descriptor. The end-to-end full hardware acceleration design avoids external memory access, the pipeline design improves the operation efficiency, a feature point pixel region can be obtained in advance, and only the pixel region is processed through a feature image pixel selector and a descriptor generation structure, so that the resource overhead can be reduced, and the system energy efficiency can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field related to image processing, and more specifically, relates to a hardware acceleration method and system based on neural network feature extraction. Background Art

[0002] The SuperPoint algorithm is a neural network-based feature extraction algorithm that can solve the matching problem between two images caused by translation, rotation, affine transformation, etc. The SuperPoint algorithm can convert a picture into a set of a large number of local feature vectors (descriptors). Compared with existing feature extraction algorithms such as LIFT, SIFT, and ORB, SuperPoint has the highest nearest neighbor average precision, the highest matching score, and the best overall homography index, showing a powerful image feature extraction effect. The algorithm can still obtain relatively reliable matching results under conditions such as affine changes, 3D perspective changes, scaling, exposure changes, and noise in the image. Therefore, the SuperPoint algorithm is widely used in scenes that require image recognition and matching, such as mapping and positioning, matching and reconstruction, motion segmentation and tracking, and image panoramic fusion of intelligent autonomous mobile robots and autonomous driving.

[0003] The existing hardware acceleration systems that implement the SuperPoint algorithm have low energy efficiency and high resource overhead. In 2020, Zhilin Xu and others from Tsinghua University used the Xilinx general AI acceleration IP core DPU (Deep learning Processing Unit) to implement the convolution operation of the SuperPoint network, and then focused on reducing the amount of computation for the post-convolution processing operations. In 2022, Ye Liu and others from the University of Electronic Science and Technology of China also used DPU to implement convolution operations, but reduced and adjusted the convolution network so that the feature point detection module in the two parallel modules completed the operation before the descriptor generation module. The above two hardware acceleration system designs that implement the SuperPoint feature extraction algorithm have both accelerated the processing speed compared to the software implementation of the original algorithm, but the above existing hardware acceleration system designs still have the following shortcomings:

[0004] The existing SuperPoint hardware acceleration system not only needs to embed a CPU to handle the frequent off-chip memory accesses caused by frequent DPU IP convolution instructions and inter-layer convolution operations, but also needs to perform post-processing operations through the CPU, which results in high energy consumption and low energy efficiency of the hardware acceleration system.

[0005] The existing SuperPoint hardware acceleration system has redundant post-processing operations, and the data comparison between adjacent pixels in the NMS module has redundant repeated comparisons, which further limits the improvement of energy efficiency. Summary of the invention

[0006] In view of the above defects or improvement needs of the prior art, the present invention provides a hardware acceleration method and system based on neural network feature extraction, which aims to improve the energy efficiency of feature extraction.

[0007] To achieve the above object, according to one aspect of the present invention, a hardware acceleration system based on neural network feature extraction is provided, which includes an encoding convolution processor, a feature point convolution processor, a SoftMax&NMS module, a feature map pixel selector, a selective descriptor convolution processor, and an interpolation and normalization module running in a pipeline manner:

[0008] The coded convolution processor is used to extract features from the input image to obtain a shared feature map;

[0009] The feature point convolution processor is used to convolve the shared feature map to obtain a confidence feature map;

[0010] The SoftMax&NMS module is used to extract candidate feature points from the confidence feature map;

[0011] The feature map pixel point selector is used to selectively generate a pixel area where a feature point is located according to currently valid feature point position information;

[0012] The selective descriptor convolution processor is used to read the feature data located in the pixel area from the shared feature map and perform descriptor convolution to generate the descriptor of the effective feature point;

[0013] The interpolation and normalization structure is used to interpolate and generally normalize the descriptors of the effective feature points to obtain a descriptor feature map;

[0014] Among them, the encoding convolution processor, the feature point convolution processor and the SoftMax&NMS module all complete the corresponding pipeline tasks within M cycles, and the selective descriptor convolution processor completes the generation of a valid feature point descriptor within N*M cycles, N>1.

[0015] In some embodiments, the time required for the coded convolution processor to calculate an output pixel feature is M cycles, and the pipeline method includes:

[0016] The coded convolution processor sequentially obtains output pixel features in every M cycles;

[0017] The feature point convolution processor completes calculation of an output pixel feature in every M cycles to obtain a first calculation result;

[0018] The SoftMax&NMS module completes the calculation of a first calculation result in every M cycles to obtain a second calculation result;

[0019] The selective descriptor convolution processor completes the calculation of a second calculation result in every N*M cycles to obtain a descriptor of the corresponding feature point;

[0020] Among them, N is equal to or approximately the ratio of the sum of the areas of all feature points to the total area of ​​the overall shared feature map.

[0021] In some embodiments, the coded convolution processor comprises an intermediate cache memory, a convolution module and a maximum pooling module running in a pipeline, each of the convolution modules comprising a data selector, a ping-pong window register, a weight memory and a bias register, a multiplier, a product register, an adder, a demultiplexer and an accumulation register;

[0022] The weight memory is used to store the weights of the multiplication operations performed by each multiplier;

[0023] The bias register is used to store the bias accumulated by the addition tree;

[0024] The data selector is used to obtain the feature points to be calculated in the shared feature map and store them in the ping-pong window register;

[0025] The multiplexer is used to extract the feature points to be calculated from the ping-pong window register and input them into multiple multipliers in parallel;

[0026] Each multiplier is used to obtain a weight from the weight memory and perform a product operation with the feature point to be calculated, and the product result is stored in the product register;

[0027] The addition tree is used to obtain the product result from the product register and obtain the bias from the bias register and accumulate;

[0028] The demultiplexer is used to store part of the accumulated results into the accumulation register and send part of the accumulated results back to the addition tree for the next round of addition operation;

[0029] The result of the accumulation register is stored in a downstream intermediate buffer memory.

[0030] In some embodiments, the process of generating the pixel area where the feature point is located by the feature map pixel point selector includes:

[0031] Determine the coordinates (i, j) of the current valid feature point in the confidence feature map;

[0032] Divide the coordinates (i, j) by k to get the coordinates (x, y). k is the ratio of the side length of the confidence feature map to the side length of the shared feature map;

[0033] Round the division result to get the center pixel (x1, y1), and determine the three interpolation pixels based on the remainder of the division result:

[0034] If i÷k has a remainder and j÷k has a remainder, the three interpolated pixels are (x1,y1-1), (x1-1,y1) and (x1-1,y1-1);

[0035] If i÷k has a remainder and j÷k is an integer, the three interpolated pixel points are (x1,y1+1), (x1-1,y1) and (x1-1,y1+1);

[0036] If i÷k is an integer and j÷k has a remainder, the three interpolated pixels are (x1,y1-1), (x1+1,y1) and (x1+1,y1-1);

[0037] If i÷k is an integer and j÷k is an integer, the three interpolated pixels are (x1,y1+1), (x1+1,y1) and (x1+1,y1+1);

[0038] A pixel region of a current valid feature point is determined, wherein the central pixel point and three interpolation pixel points are located in the central region of the pixel region.

[0039] In some of these embodiments, the hardware acceleration system further comprises a shared feature map cache for storing the shared feature map;

[0040] The feature map pixel point selector includes a FIFO module, a reordering module, an interpolation module, a first address generation module, an interval generation module and a feature point address synthesis module;

[0041] The FIFO module is used to cache the coordinates of valid feature points;

[0042] The reorderer module is used to obtain the row coordinates of the current valid feature point from the FIFO module, and determine the row index of the valid feature point in the shared feature map cache based on the row coordinates to obtain the reordering number;

[0043] The interpolation module is used to obtain the coordinates of the current valid feature point from the FIFO module and determine the pixel area of ​​the current valid feature point;

[0044] The first address generating module is used to generate the first address of the pixel area in the shared feature map cache according to the rearrangement number and the position of the first pixel point of the pixel area;

[0045] The interval generating module is used to determine the address difference of each pixel according to the relative position of each pixel point in the pixel area;

[0046] The feature point address synthesis module is used to determine the address of each pixel point in the pixel area in the shared feature map cache according to the first address and the address difference.

[0047] In some embodiments, the SoftMax&NMS module includes:

[0048] The SoftMax submodule includes a subregion division module, an approximate calculation module, an accumulation register, a first comparison and controller, a position information register and a divider; the subregion division module is used to divide the confidence feature map into multiple subregions, and the approximate calculation module is used to perform exponential operation on the feature data α of each pixel point in the same subregion to obtain the operation result 2 α The accumulation register is used to accumulate all the operation results of the same sub-region to obtain the accumulation result ∑2 α The first comparison and controller is used to determine the feature data α with the largest exponential operation result in the same sub-region max , the position information register is used to store α max Location information max ; The divider is used to perform the division operation α max ÷∑2 α , get the corresponding sub-region location information E max The effective value of

[0049] The NMS submodule is used to sequentially obtain sub-areas whose valid values ​​are not set to 0 and use them as the central area, compare the current central area and the adjacent valid values ​​that are within the set distance range from the current central area valid value and have not been compared with the current central area valid value. If the central area valid values ​​are all greater than the adjacent valid values, the pixel point at the location of the current central area valid value is identified as a feature point and the adjacent valid value is set to 0. Otherwise, there is no feature point in the current central area and the adjacent valid value is maintained unchanged.

[0050] In some embodiments, the SoftMax submodule further includes a maximum index register and a maximum register, wherein the maximum index register is used to store the index of the pixel point with the largest exponential operation result, and the maximum register is used to store the largest exponential operation result;

[0051] The first comparison and controller is used for comparing the current exponential operation result with the exponential operation result in the maximum value register whenever the approximate calculation module outputs an exponential operation result. If the former is greater than the latter, the current exponential operation result is stored in the maximum value register as the updated maximum exponential operation result, and the index of the current pixel point is stored in the maximum value index register as the index of the pixel point with the maximum exponential operation result after the update;

[0052] After the comparison of all the pixels in the current sub-region is completed, the position information of the corresponding pixel is stored in the position information register according to the index of the pixel in the maximum value index register.

[0053] In some of these embodiments, the NMS submodule includes a first selector, a second selector, and a second comparison and controller;

[0054] The first selector is used to obtain the position information of the current central area effective value and the position information of the adjacent sub-area effective value that has not been compared with the current central area effective value and send them to the second comparison and controller;

[0055] The second comparison and controller is used to first calculate the position distance between the current central area effective value and the adjacent sub-area effective value and determine whether the position distance between the two effective values ​​is within the set range. If so, the adjacent sub-area is considered to be a valid sub-area;

[0056] The second selector is used to obtain the current effective value of the central area and the effective value of the effective sub-area and send them to the second comparison and controller;

[0057] The second comparison and controller is also used to compare the size of the effective value of the current central area and the effective values ​​of each effective sub-area. If the effective value of the current central area is greater than the effective values ​​of each effective sub-area, the pixel point at the position of the effective value of the current central area is output as a feature point and the adjacent effective values ​​are set to 0 and the effective value of the effective sub-area is set to 0. Otherwise, it is considered that there is no feature point in the current central area and the effective values ​​are maintained unchanged.

[0058] In some embodiments, the set range is the area covered by a rectangular window centered on the position of the current central area valid value, and the length and width of the rectangular window are both larger than the length and width of the sub-area and less than twice the length and width of the sub-area.

[0059] According to another aspect of the present invention, a hardware acceleration method based on the above hardware acceleration system is provided, which comprises:

[0060] Step S1: At the first stage of the pipeline, the input image enters the coded convolution processor for feature extraction, and a shared feature map is obtained and stored in the shared feature map cache;

[0061] Step S2: At the second stage of the pipeline, the feature data in the shared feature map cache is sent to the feature point convolution processor for convolution to obtain the confidence feature map and store it in the pipeline register. At this time, before the descriptor convolution is performed, the system has obtained the feature point information;

[0062] Step S3: At the third stage of the pipeline, the confidence feature map in the pipeline register is sent to the SoftMax&NMS module to calculate the feature confidence of each pixel and perform non-maximum suppression to obtain feature points;

[0063] Step S4: In the first half of the fourth stage of the pipeline, the pixel area where the feature point is located is selectively generated according to the currently valid feature point position information by the feature map pixel point selector;

[0064] Step S5: in the second half of the fourth stage of the pipeline, the feature data located in the pixel area is read from the shared feature map by a selective descriptor convolution processor and descriptor convolution is performed to generate a descriptor of the valid feature point;

[0065] Step S6: at the fifth stage of the pipeline, the descriptors of the valid feature points are interpolated and generally normalized by an interpolation and normalization structure to obtain a descriptor feature map;

[0066] Among them, the encoding convolution processor, the feature point convolution processor and the SoftMax&NMS module all complete the corresponding pipeline tasks within M cycles, and the selective descriptor convolution processor completes the generation of a valid feature point descriptor within N*M cycles, N>1.

[0067] In general, compared with the prior art, the above technical solutions conceived by the present invention mainly have the following beneficial effects:

[0068] 1. The hardware acceleration system based on neural network feature extraction provided by the present invention implements the SuperPoint algorithm by designing a dedicated end-to-end hardware acceleration system, which can avoid frequent off-chip memory access caused by embedding a CPU to process frequent DPU IP convolution instructions and inter-layer convolution operations, and also avoid post-processing operations through the CPU. Through a dedicated acceleration design, off-chip memory access consumption is avoided, and energy efficiency is significantly improved.

[0069] 2. The hardware acceleration system based on neural network feature extraction provided by the present invention, the encoding convolution processor, feature point convolution processor, SoftMax&NMS module, selective descriptor convolution processor, interpolation and normalization module are operated in a pipeline manner, which can further improve the operating efficiency of the overall structure.

[0070] 3. The hardware acceleration system based on neural network feature extraction provided by the present invention can determine the effective feature points in advance due to the pipeline operation mode. After the effective feature points are determined, the feature map pixel point selector can select the pixel area where the feature points are located according to the positions of the feature points. When the fourth-level pipeline generates descriptors, only the feature data in the pixel area needs to be convolved, thus avoiding the redundant descriptor convolution of all non-feature points and further improving the computing energy efficiency.

[0071] 4. The hardware acceleration system based on neural network feature extraction provided by the present invention takes into account that the distribution of feature points in the shared feature map is sparse. If the feature point convolution processor and the descriptor convolution processor are allocated computing resources of the same specification relative to the network, the computing resources of the descriptor convolution processor will be idle for a long time, and the ratio of its idle time to working time is close to the ratio of the sum of the feature point area to the total area of ​​the overall shared feature map. In the present invention, the processing time of the selective descriptor convolution processor is longer, which can reduce the computing hardware resources such as the multiplier of the descriptor convolution processor. Specifically, N can be equal to the ratio of the sum of the areas of the shared feature map where all feature points are located to the total area of ​​the shared feature map, which reduces the hardware overhead. Therefore, the computing resource overhead of the descriptor convolution module can be reduced without reducing the system operation speed.

[0072] 5. The fully pipelined end-to-end accelerator designed by the present invention supports convolution of only a part of the feature map, while the current work mainly calls the DPU for convolution acceleration processing. The general calculation result of the DPU cannot be convolved on a certain section of the shared feature map, and can only convolve the shared feature map as a whole; moreover, in the pipeline process of the present invention, the pipeline level where the feature point convolution is located is higher than the pipeline level where the descriptor convolution is located, so the position information of the feature point can be obtained before the descriptor convolution, so that a part of the area centered on the feature point is selected for descriptor convolution. Therefore, thanks to the pipeline design, the feature point position can be obtained before calculating the descriptor, and the descriptor convolution processing does not need to convolve the entire shared feature map, but only needs to convolve the feature point and the pixel area near it to obtain the descriptor, avoiding all redundant descriptor convolution processes, thereby reducing computing resource overhead.

[0073] 6. In a specific embodiment, in the design of the Softmax submodule, multiple sub-regions are divided. Based on the characteristic that the maximum value output by the Softmax function is most likely to be retained after the NMS operation, only the maximum value of the sub-region is retained for operation, thereby simplifying the operation process, reducing the number of Softmax submodule outputs, and reducing the number of data comparisons in the NMS submodule. Moreover, considering that the absolute value of the Softmax operation result is not critical, while ensuring that the relative size of the Softmax operation result remains unchanged, the present invention innovatively uses an exponential operation with a base of 2 as an approximate operation, which can further simplify the operation. When designing the NMS submodule, the comparison is avoided by setting 0 during comparison, so that the comparison can be repeated. By improving the SoftMax&NMS module, the Softmax&NMS operation process can be simplified, and the circuit resource overhead and system energy consumption can be greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 is a structural block diagram of a hardware acceleration system in one embodiment of the present invention;

[0075] Figure 2 is a schematic diagram of the pipeline operation of the hardware acceleration system in one embodiment of the present invention;

[0076] Figure 3 is a schematic diagram of the structure of a coding convolution processor in one embodiment of the present invention;

[0077] Figure 4 is a structural diagram of a Conv2a module in one embodiment of the present invention;

[0078] Figure 5 is a schematic diagram of determining an interpolation value according to a remainder in an embodiment of the present invention;

[0079] Figure 6 is a structural block diagram of a feature map pixel selector in one embodiment of the present invention;

[0080] Figure 7 It is a structural diagram of the SoftMax&NMS module in one embodiment of the present application;

[0081] Figure 8 is a traversal direction diagram of the central area in one embodiment of the present invention;

[0082] Fig. 9 3 is a schematic diagram of comparison between a central area and adjacent sub-areas in one embodiment of the present invention. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0084] like Figure 1 The structure block diagram of the hardware acceleration system in one embodiment of the present invention is shown, which mainly includes a pipeline-operated coding convolution processor, a feature point detection structure, a feature map pixel point selector and a descriptor generation structure, wherein the coding convolution processor is used to extract features from the input image to obtain a shared feature map, the feature point detection structure is used to calculate and output feature points, the feature map pixel point selector will selectively generate the pixel area where the feature point is located according to the currently valid feature point information, and the descriptor generation structure reads the feature data located in the pixel area from the shared feature map and performs descriptor convolution and post-processing to generate the corresponding descriptor. Based on the above end-to-end full hardware acceleration system, the SuperPoint algorithm can be implemented.

[0085] Among them, the feature point detection structure includes a feature point convolution processor and a SoftMax&NMS module. The feature point convolution processor is used to convolve the shared feature map to obtain a confidence feature map, and the SoftMax&NMS module is used to extract feature points from the confidence feature map. The feature map pixel selector will selectively generate the pixel area where the feature point is located based on the current valid feature point position information.

[0086] The descriptor generation structure is used to read feature data located in the pixel area from the shared feature map and perform descriptor convolution to generate descriptors of valid feature points.

[0087] It can be understood that the hardware acceleration system also has a shared feature map cache for storing the shared feature maps generated by the encoding convolution processor.

[0088] In one embodiment, the descriptor generation structure includes a selective descriptor convolution processor and an interpolation and normalization module. The selective descriptor convolution processor is used to obtain the feature value of the pixel area where the feature point provided by the feature map pixel point selector is located from the shared feature map and perform convolution processing; the interpolation and normalization module is used to perform interpolation and normalization processing on the convolution result of the selective descriptor convolution processor so that the size of the feature map matches the original image, thereby calculating the descriptor of the corresponding feature point.

[0089] In the present invention, the coded convolution processor, the feature point detection structure and the descriptor generation structure are operated in a pipeline manner, which can further improve the operation efficiency of the overall structure. At the same time, due to the use of the pipeline operation mode, the effective feature points can be determined in advance. After the effective feature points are determined, the pixel area where the effective feature points are located can be determined. When generating the descriptor, only the feature data in the pixel area needs to be convoluted, which further improves the computing energy efficiency.

[0090] The following first introduces the pipeline operation mode.

[0091] Specifically, the encoding convolution processor, the feature point convolution processor, the SoftMax&NMS module, the selective descriptor convolution processor, and the interpolation and normalization module operate in a pipeline manner. The time required for the encoding convolution processor to calculate an output pixel feature is M cycles. The encoding convolution processor obtains the output pixel features in turn in each M cycles. The feature point convolution processor completes the calculation of an output pixel feature in each M cycles to obtain a first calculation result. The SoftMax&NMS module completes the calculation of a first calculation result in each M cycles to obtain a second calculation result. The selective descriptor convolution processor completes the calculation of a second calculation result in each N*M cycles to obtain a third calculation result. The interpolation and normalization module calculates the third calculation result in turn to obtain the descriptor of the corresponding feature point.

[0092] For the pipeline processing of the main computing modules in the hardware acceleration system, the calculation time of each output feature pixel of the encoding convolution processor is designed to be M cycles. In order to achieve matching pipeline calculation at the module level, the feature point convolution processor and the SoftMax&NMS module are designed to calculate a point in the feature map in a time of less than or equal to M cycles. Considering the redundancy of descriptor calculation, that is, non-feature points do not need to calculate descriptors, the calculation speed of the selective descriptor convolution processor is designed to be one-Nth of the speed of the feature point convolution processor. Therefore, the time for the selective descriptor convolution processor to calculate one pixel is designed to be M×N cycles. The time for the interpolation and normalization module to calculate one pixel in the descriptor map is less than or equal to M×N cycles.

[0093] like Figure 2The figure shows a pipeline operation schematic diagram of the hardware acceleration system in one embodiment of the present invention. For example, in order to achieve a processing speed of 20 frames per second for VGA images (640×480) at a clock frequency of 200MHz, the corresponding convolution calculation of each pixel in the input image must be completed within 32 cycles. Since there are three 2×2 maximum pooling layers in the encoder network, that is, the downsampling rate is 4, the output pixel duration after each maximum pooling layer is four times that of the input pixel. Therefore, the calculation time of each output feature pixel of the encoder is designed to be 2048 cycles, that is, 32×4×4×4. In order to achieve matching pipeline processing at the module level, the feature point convolution processor and the SoftMax&NMS post-processing module are designed to calculate a point in the feature map in 2048 cycles or less. Considering the data information mismatch characteristics discussed above, the calculation speed of the descriptor decoder is designed to be one-fourth of the feature point decoder speed. Therefore, the time for the selective descriptor convolution processor to calculate one pixel is designed to be 8192 cycles. The interpolation and normalization modules are designed to take 786 cycles to compute one pixel in the descriptor map. The total latency of the proposed pipeline architecture is approximately 5780 cycles and 14758 cycles for feature point and descriptor computation, respectively.

[0094] In one embodiment, the encoding convolution processor includes an intermediate cache memory, a convolution module, and a maximum pooling module that operate in a pipeline. Figure 3 The figure shows a schematic diagram of the structure of a coding convolution processor in one embodiment of the present invention, which is composed of an intermediate buffer, a pipeline convolution module and a maximum pooling module.

[0095] Each convolution module completes the convolution operation of the input feature pixels within a specified clock cycle (i.e., 32-2048 cycles). Taking the first two layers as an example, the multipliers of the Conv1a module perform eight 3×3 convolutions in sequence, with 3 multiplications per cycle, requiring a total of 24 cycles. Subsequently, Buffcv1b stores 3 rows of feature maps with 8 channels. The Conv1b module uses 18 multipliers to perform eight 3×3×8 filter convolutions in sequence, calculating 3×3×2 elements per cycle, corresponding to 1 / 4 filters, and requires 32 cycles to complete. It is worth noting that the convolution layer pipeline architecture of the feature point convolution processor and the descriptor convolution processor is similar to that of the encoding convolution processor, except for the number of multipliers and the lack of a maximum pooling module.

[0096] In one embodiment, each convolution module includes a data selector, a ping-pong window register, a weight memory and a bias register, a multiplier, a product register, an adder, a demultiplexer, and an accumulation register;

[0097] The weight memory is used to store the weights of the multiplication operations performed by each multiplier;

[0098] The bias register is used to store the bias for accumulation by the addition tree;

[0099] The data selector is used to obtain the feature points to be calculated in the shared feature map and store them in the ping-pong window register;

[0100] The multiplexer is used for extracting feature points to be calculated from the ping-pong window register and inputting them into multiple multipliers in parallel;

[0101] Each multiplier is used to obtain a weight from a weight memory and perform a product operation with a feature point to be calculated, and the product result is stored in a product register;

[0102] The addition tree is used to obtain the product result from the product register and the bias from the bias register and accumulate;

[0103] The demultiplexer is used to store part of the accumulated results into the accumulation register and send part of the accumulated results back to the addition tree for the next round of addition operation;

[0104] The result of the accumulation register is stored in the intermediate buffer memory connected later.

[0105] like Figure 4The structure diagram of one of the Conv2a modules in the coding convolution processor in one embodiment of the present invention is shown. The data selector provides a read address and a read enable, and writes the input feature map data of a certain channel read into the ping-pong window register in the correct order to ensure that each input feature map data can be correctly multiplied by the corresponding convolution weight value. The weight data and bias data in the neural network are stored in the weight memory and the bias register respectively. The data in each ping-pong window register is reused for 16 clock cycles to complete the multiplication and accumulation operations corresponding to 16 filters. In order to realize uninterrupted convolution operations, the convolution layer submodule adopts a ping-pong structure, and the activation of ping-pong window register 1 or 2 is realized by the subsequent multiplexer. When the data in ping-pong window register 1 is used up, the data in ping-pong window register 2 can be used immediately, and at the same time, ping-pong window register 1 can read the input feature map data of the next channel. In a convolution process of the convolutional layer submodule, due to the limited number of multipliers, it is impossible to calculate all channels of a pixel at once. Therefore, each calculation process is recorded as a partial and intermediate result. When the 3×3 multiplication and addition operations corresponding to all channels are completed, the convolution operation of the pixel is completed. The circuit of the accumulation operation is implemented by the demultiplexer, the accumulation register and the multiplexer. The demultiplexer is responsible for allocating the partial sum calculated by the addition tree to the corresponding accumulation register. For example, the partial sum of the nth channel will be stored in Reg_n. After the partial sum stored in the accumulation register selects the corresponding channel through the multiplexer, it will be looped back to the addition tree in the next round of addition operation, and enter the addition tree together with the bias weight for accumulation, thereby realizing the accumulation of partial sums.

[0106] The key to the pipeline design of the convolution module is to utilize the fixed data stream of the input. This is because the previous module continues to perform calculations and passes the results to the intermediate buffer Buffcv2a, which stores feature maps with 8 channels and 3 rows. It is required to complete the convolution operation in units of channels as early as possible. In the calculation stage, 9 multipliers are used in parallel in each cycle to multiply the 9 pixels stored in the ping-pong window register with the 9 weights of one channel of the 3×3×8 filter. In the next cycle, the same 9 input feature map pixels will be reused and multiplied with the 9 weights of the corresponding channel of the next filter. This process is repeated to complete the multiplication operation of the same channel of the current input feature map in different filters, thereby efficiently realizing the fixed data stream of the input. Through the pipeline accumulator, the results of the 16 channels corresponding to a data point in the output feature map are finally obtained in sequence within 16 cycles.

[0107] In the present invention, the operation efficiency of the overall structure can be further improved by designing an assembly line operation mode.

[0108] The specific design of the feature map pixel selector is introduced below.

[0109] The process of determining the pixel area includes:

[0110] Determine the coordinates (i, j) of the current valid feature point in the confidence feature map;

[0111] Divide the coordinates (i, j) by k to get the coordinates (x, y). k is the side length of the confidence feature map / the side length of the shared feature map, and k is a positive integer;

[0112] Round the division result to get the center pixel (x1, y1), and determine the three interpolation pixels based on the remainder of the division result:

[0113] If i÷k has a remainder and j÷k has a remainder, the three interpolated pixels are (x1,y1-1), (x1-1,y1) and (x1-1,y1-1);

[0114] If i÷k has a remainder and j÷k is an integer, the three interpolated pixel points are (x1,y1+1), (x1-1,y1) and (x1-1,y1+1);

[0115] If i÷k is an integer and j÷k has a remainder, the three interpolated pixels are (x1,y1-1), (x1+1,y1) and (x1+1,y1-1);

[0116] If i÷k is an integer and j÷k is an integer, the three interpolated pixels are (x1,y1+1), (x1+1,y1) and (x1+1,y1+1);

[0117] The pixel area of ​​the current valid feature point is determined, and the central pixel point and three interpolation pixel points are located in the central area of ​​the pixel area.

[0118] Specifically, take the shared feature map of 80×60×256 and the confidence feature map of 640×480 as an example. The length of the shared feature map is 80 and the width is 60, and the length of the confidence feature map is 640 and the width is 480. Therefore, the side length of the confidence feature map is 8 times the side length of the shared feature map, that is, the pixel points of the confidence feature map are the pixels obtained by expanding the length and width of the shared feature map by 8 times, k = 8. Since the feature points determined from the 640×480 confidence feature map need to extract the feature data of the pixel area where the feature points are located from the 80×60×256 shared feature map, it is necessary to reduce the coordinates (i, j) of the feature points in the 640×480 confidence feature map to obtain the coordinates (x, y). Then, the pixel region in the 80×60×256 shared feature map is determined based on the coordinates (x, y).

[0119] Since there is an error in coordinate scaling, the coordinate accuracy is improved through interpolation. In this embodiment, the integer part of the coordinate (x, y) is used as the central pixel point (x1, y1), and the interpolation strategy is determined according to the remainder. For details, refer to Figure 5 As shown, the interpolation method can improve the accuracy of the pixel area. After determining the central pixel point and the interpolation pixel point, appropriate area extension is performed with the four pixel points as the center to obtain the pixel area corresponding to the feature point. For example, the size of the pixel area in this embodiment is set to 8×8.

[0120] After determining the pixel area of ​​the feature point, the descriptor generation structure is used to process the feature data located in the pixel area in the shared feature map to generate a descriptor for the feature point. In this way, on the one hand, the descriptor generation structure only needs to process the feature points to generate descriptors corresponding to the feature points, avoiding redundant calculations of non-feature points, reducing hardware overhead and improving computational energy efficiency. On the other hand, the positioning accuracy of the pixel area is improved through the above-mentioned interpolation processing method. When generating the descriptor, only the feature data in the pixel area needs to be convolved, further improving computational energy efficiency.

[0121] like Figure 6 The structure block diagram of the feature map pixel point selector in one embodiment of the present invention is shown, which includes a FIFO module, a reordering module, an interpolation module, a first address generating module, an interval generating module and a feature point address synthesizing module, wherein the FIFO module is used to cache the coordinates of valid feature points; the reordering module is used to obtain the row coordinates of the current valid feature point from the FIFO module, and determine the row index of the valid feature point in the shared feature map cache based on the row coordinates to obtain the rearrangement number; the interpolation module is used to obtain the coordinates of the current valid feature point from the FIFO module and determine the pixel area of ​​the current valid feature point; the first address generating module is used to generate the first address of the pixel area in the shared feature map cache according to the rearrangement number and the position of the first pixel point in the pixel area; the interval generating module is used to determine the address difference of each pixel according to the relative position of each pixel point in the pixel area; the feature point address synthesizing module is used to determine the address of each pixel point in the pixel area in the shared feature map cache according to the first address and the address difference.

[0122] In this embodiment, redundant convolution operations are avoided by selecting only the feature map data of the feature point and its neighborhood points. Specifically, the synchronous FIFO module caches the row and column coordinates of the feature point, and the row and column coordinates are output to the reorderer and interpolator for subsequent data processing. The reorderer is responsible for reading the number of rows of the feature point and determining the read row index of the data in the shared feature map cache. Since the shared feature map cache is limited by the memory size, only five rows of feature map data can be stored at a time. When a new row of data needs to be stored, the row in the current cache after dividing the number of rows by five and finding the remainder will be replaced. For example, after the first five rows of data are read, the sixth row of data will be stored in the first row. Therefore, after determining the number of rows of the feature map, it is necessary to perform a modulo 5 operation on the number of rows to find the corresponding row in the cache memory. This remainder is called the reordering number, which is used to identify the storage location of the data, thereby ensuring that the data in the shared feature map cache can be read in the correct order. The interpolator derives the selection method of the feature point based on the row and column information of the feature point. Specifically, the coordinate indexes of the pixels with coordinates (i, j) in the 640×480 image where the feature points are located are divided by 8 to obtain the corresponding coordinates (x, y) before interpolation in the output feature map of the selective descriptor convolution processor. The present invention records the pixel point of the feature map corresponding to the rounded coordinates (x, y) as the center pixel point. In addition to the center pixel point, the remaining three pixels required for interpolation are determined by the remainder after the coordinates (x, y) are rounded, that is, the remainder after the coordinates (i, j) are divided by 8 to determine the data selection method, and the first address generator generates the first address of the feature point based on the information of the rearrangement number and the interpolation number. The interval generator is the address difference of each adjacent element in the descriptor convolution process. The 4*4 feature area address is obtained by adding the first address and the interval address.

[0123] The descriptor generation structure uses the above address to read the feature points from the shared data cache memory, and the selection of the feature points calculated by the descriptor generator is completed at this time.

[0124] Based on the above design, the SoftMax&NMS module can also be improved to further reduce the amount of calculation and improve computing energy efficiency.

[0125] like Figure 7 The figure shows a schematic diagram of the structure of the SoftMax&NMS module in one embodiment of the present application, which mainly includes a SoftMax submodule and an NMS submodule.

[0126] The SoftMax submodule includes a sub-region division module (not shown in the figure), an approximate calculation module (i.e., appro_2 αmodule), an accumulator register, a first comparison and controller, a position information register and a divider; a sub-region division module is used to divide the confidence feature map into multiple sub-regions, and an approximate calculation module is used to perform an exponential operation on the feature data α of each pixel point in the same sub-region to obtain an operation result 2 α The accumulator register is used to accumulate all the operation results of the same sub-region to obtain the accumulated result ∑2 α ; The comparison and controller are used to determine the characteristic data α with the largest operation result in the same sub-area max , the position information register is used to store α max Location information max ; The divider is used to perform the division operation α max ÷∑2 α , get the corresponding sub-region location information E max Valid values.

[0127] Specifically, the size of the confidence feature map is the same as the size of the input image. For example, if the input image is a 640×480 VGA image, the down-sampling convolution processing by the coded convolution processor will generate a shared feature map of 80 (length) × 60 (width) × 256 (channels). The feature point convolution processor convolves the shared feature map to obtain a feature map of 80 (length) × 60 (width) × 64 (channels), which is then flattened through the channels to restore the confidence feature map of 640×480.

[0128] In the SoftMax submodule, the 640×480 confidence feature map is first divided into multiple sub-regions through the sub-region division module. For example, each sub-region is 8×8 in size and can be divided into 4800 sub-regions. Each sub-region can obtain the effective value of the corresponding sub-region through the approximate calculation module, the accumulation register, the first comparison and controller, and the divider. The expression of the effective value is Therefore, it is a number less than 1.

[0129] In the specific design, a maximum value index register and a maximum value register can be added. The maximum value index register is used to store the index of the pixel point with the largest exponential operation result, and the maximum value register is used to store the largest exponential operation result. The comparison and controller is used to compare the current exponential operation result with the exponential operation result in the maximum value register every time the approximate calculation module outputs an exponential operation result. If the former is greater than the latter, the current exponential operation result is stored in the maximum value register as the updated maximum exponential operation result, and the index of the current pixel point is stored in the maximum value index register as the index of the pixel point with the largest exponential operation result after the update; after the comparison of all the pixels in the current sub-area is completed, the position information of the corresponding pixel point is stored in the position information register according to the index of the pixel point in the maximum value index register.

[0130] Continuing with the example of a sub-region size of 8×8, the 64 elements contained in each 8×8 sub-region are received one by one, and each element is processed 2 α This process is approximated by appro_2 α Approximate calculation module implementation, which uses shift to perform exponential calculation on feature map data. Accumulation register pair appro_2 α The calculation results of the module are stored and accumulated in each cycle, and the final accumulated value is used as the divisor of the CORDIC division operation. The Softmax submodule compares each time, that is, each input feature map data 2 α The exponential calculation result will be the same as the current 2 α The maximum value is compared. If the current input is 2 α The exponential calculation result is greater than the maximum value recorded, then the current input 2 α The exponential calculation result is the current new maximum value, and 2 α The index of the maximum value index register is stored in the 2 register. α The maximum value in the approximate calculation result is selected and used as the dividend of the CORDIC division operation. The output position information register stores the position index of the maximum feature point data. The result of the CORDIC division operation is the output of the Softmax function.

[0131] In one embodiment, a Buffer controller and a cache Buffer may be set behind the SoftMax submodule, and the effective value output by the SoftMax submodule is saved in the cache Buffer through the Buffer controller for reading by the NMS submodule.

[0132] The NMS submodule is used to sequentially obtain the sub-areas that have not been set to 0 and use them as the central area, compare the current central area and the adjacent valid values ​​that are within the set distance range from the current central area valid value and have not been compared with the current central area valid value. If the central area valid values ​​are all greater than the adjacent valid values, the pixel point at the location of the current central area valid value is identified as a feature point and the adjacent valid value is set to 0. Otherwise, there is no feature point in the current central area and the adjacent valid value remains unchanged.

[0133] like Figure 8The figure shows a traversal direction diagram of the central area in an embodiment of the present invention. Each square in the figure represents a sub-area. First, the first sub-area is obtained as the central area. Its adjacent areas are a sub-area on the right and two sub-areas on the bottom. The valid values ​​of the three adjacent sub-areas are the adjacent valid values. After judgment, if the second sub-area is not set to 0, the second sub-area continues to be the central area and is compared with its adjacent valid values. If the second sub-area is set to 0, the second sub-area is skipped and the third sub-area is used as the next central area, and so on. This can reduce the amount of data calculation, save hardware resources, and improve energy efficiency.

[0134] It is understandable that the setting range can be determined based on experience or experimental comparison. For example, the setting range is the area covered by a rectangular window centered at the location of the current central area valid value, and the length and width of the rectangular window are both larger than the length and width of the sub-area and both smaller than 2 times the length and width of the sub-area. Fig. 9 The figure shows a schematic diagram of the comparison between the central area and the adjacent sub-areas in one embodiment of the present invention. Sub-area (5) is taken as the central area. At this time, (1), (2), (3), (4), (6), (7), (8), and (9) are all adjacent sub-areas. Since (1), (2), (3), and (4) have been compared with (5) when they are taken as the central area, when sub-area (5) is taken as the central area, it is not necessary to compare with (1), (2), (3), and (4). It only needs to be compared with sub-areas (6), (7), (8), and (9). In this embodiment, the setting range is a 9×9 window centered on the effective value position of sub-region (5). After comparison, only sub-region (6) is within the setting range. Therefore, it is only necessary to compare the effective values ​​of sub-regions (5) and (6). If the effective value of sub-region (5) is larger, the pixel point at the effective value of sub-region (5) is output as the feature point, and the effective value of sub-region (6) is set to 0. Otherwise, sub-region (6) is continued to be used as the center area to determine whether there is a feature point.

[0135] In the specific design, the NMS submodule includes a first selector, a second selector, and a second comparison and controller; the first selector is used to obtain the position information of the current central area effective value and the position information of the adjacent sub-area effective value that has not been compared with the current central area effective value and send it to the second comparison and controller; the second comparison and controller is used to first calculate the position distance between the current central area effective value and the adjacent sub-area effective value and determine whether the position distance between the two effective values ​​is within a set range. If so, the adjacent sub-area is considered to be a valid sub-area; the second selector is used to obtain the current central area effective value and the effective value of the effective sub-area and send them to the second comparison and controller; the second comparison and controller is also used to compare the current central area effective value and the effective value of each effective sub-area. If the current central area effective value is greater than the effective value of each effective sub-area, the pixel point at the position of the current central area effective value is output as a feature point and the adjacent effective value is set to 0 and the effective value of the effective sub-area is set to 0. Otherwise, it is considered that there is no feature point in the current central area and each effective value is maintained unchanged.

[0136] Specifically, the first selector reads the position information corresponding to the valid values ​​of the central sub-region and its adjacent sub-region from the position information of the cache Buffer. The adjacent sub-regions read here filter out the adjacent sub-regions that have been compared with the current central sub-region, and send them to the second comparison and controller for comparison. The second comparison and controller first determines the distance between the position of the valid value of the adjacent sub-region and the valid value position of the central sub-region. If the distance between the valid value position of the central sub-region and the valid value position of the adjacent sub-region is within the 9×9 window range, the adjacent sub-region is considered to be a valid sub-region, and the effective value size can be compared. Otherwise, it is an invalid sub-region and there is no need to compare the effective value size. The second selector selects the valid values ​​of the central sub-region and the valid sub-region, takes out these data for size comparison, and if the valid values ​​of the central sub-region are greater than the valid values ​​of the adjacent sub-regions, the pixel corresponding to the valid value of the central sub-region is the feature point. The NMS submodule outputs the coordinates of the feature point and sets the valid value of the valid sub-region to 0.

[0137] In this embodiment, the SoftMax&NMS module is improved, wherein, in the design of the Softmax submodule, multiple sub-regions are divided, and based on the characteristic that the maximum value of the Softmax function output is most likely to be retained after the NMS operation, only the maximum value of the sub-region is retained for operation, thereby simplifying the operation process, reducing the number of Softmax submodule outputs, and reducing the number of data comparisons of the NMS submodule. Moreover, considering that the absolute value of the Softmax operation result is not critical, while ensuring that the relative size of the Softmax operation result remains unchanged, the present invention innovatively uses the exponential operation with a base of 2 as an approximate operation, which can further simplify the operation. When designing the NMS submodule, the comparison is avoided by setting 0 during comparison, so that the comparison can be repeated. By improving the SoftMax&NMS module, the Softmax&NMS operation process can be simplified, and the circuit resource overhead and system energy consumption can be greatly reduced.

[0138] Accordingly, the present invention also relates to a hardware acceleration method based on the above hardware acceleration system, which comprises:

[0139] Step S1: At the first stage of the pipeline, the input image enters the coded convolution processor for feature extraction, and a shared feature map is obtained and stored in the shared feature map cache;

[0140] Step S2: At the second stage of the pipeline, the feature data in the shared feature map cache is sent to the feature point convolution processor for convolution to obtain the confidence feature map and store it in the pipeline register. At this time, before the descriptor convolution is performed, the system has obtained the feature point information;

[0141] Step S3: At the third stage of the pipeline, the confidence feature map in the pipeline register is sent to the SoftMax&NMS module to calculate the feature confidence of each pixel and perform non-maximum suppression to obtain feature points;

[0142] Step S4: In the first half of the fourth stage of the pipeline, the pixel area where the feature point is located is selectively generated according to the currently valid feature point position information by the feature map pixel point selector;

[0143] Step S5: In the second half of the fourth stage of the pipeline, the feature data located in the pixel area is read from the shared feature map by a selective descriptor convolution processor and descriptor convolution is performed to generate a descriptor of a valid feature point;

[0144] Step S6: at the fifth stage of the pipeline, the descriptors of the valid feature points are interpolated and generally normalized through the interpolation and normalization structure to obtain a descriptor feature map;

[0145] Among them, the encoding convolution processor, feature point convolution processor and SoftMax&NMS module all complete the corresponding pipeline tasks within M cycles, and the selective descriptor convolution processor completes the generation of a valid feature point descriptor within N*M cycles, N>1.

[0146] Specifically, the design of each structural unit in the hardware acceleration system can refer to the above introduction, which will not be repeated here.

[0147] In the present invention, by designing a dedicated end-to-end hardware acceleration system to implement the SuperPoint algorithm, it is possible to avoid embedding a CPU to process frequent DPU IP convolution instructions and frequent off-chip memory accesses caused by inter-layer convolution operations, and also avoid post-processing operations by the CPU, significantly improving energy efficiency. Moreover, by improving each hardware module, the calculation process can be simplified, greatly reducing circuit resource overhead and system energy consumption.

[0148] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. It should be noted that "in one embodiment", "for example", "for example", etc. of the present invention are intended to illustrate the present invention, rather than to limit the present invention.

[0149] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention.

Claims

1. A hardware acceleration system based on neural network feature extraction, It is characterized in that It includes the pipelined encoding convolution processor, feature point convolution processor, SoftMax&NMS module, feature map pixel selector, selective descriptor convolution processor, interpolation and normalization module: The coded convolution processor is used to extract features from the input image to obtain a shared feature map; The feature point convolution processor is used to convolve the shared feature map to obtain a confidence feature map; The SoftMax&NMS module is used to extract candidate feature points from the confidence feature map; The feature map pixel point selector is used to selectively generate a pixel area where a feature point is located according to currently valid feature point position information; The selective descriptor convolution processor is used to read the feature data located in the pixel area from the shared feature map and perform descriptor convolution to generate the descriptor of the valid feature point; The interpolation and normalization structure is used to interpolate and generally normalize the descriptors of the effective feature points to obtain a descriptor feature map; Among them, the encoding convolution processor, the feature point convolution processor and the SoftMax&NMS module all complete the corresponding pipeline tasks within M cycles, and the selective descriptor convolution processor completes the generation of a valid feature point descriptor within N*M cycles, N>1.

2. The hardware acceleration system according to claim 1, It is characterized in that The time required for the coded convolution processor to calculate an output pixel feature is M cycles, and the pipeline method includes: The coded convolution processor sequentially obtains output pixel features in every M cycles; The feature point convolution processor completes calculation of an output pixel feature in every M cycles to obtain a first calculation result; The SoftMax&NMS module completes the calculation of a first calculation result in every M cycles to obtain a second calculation result; The selective descriptor convolution processor completes the calculation of a second calculation result in every N*M cycles to obtain a descriptor of the corresponding feature point; Where N is equal to or approximately the ratio of the sum of the areas of all feature points to the total area of ​​the overall shared feature map.

3. The hardware acceleration system according to claim 1, It is characterized in that The coded convolution processor comprises an intermediate cache memory, a convolution module and a maximum pooling module running in a pipeline, each of the convolution modules comprises a data selector, a ping-pong window register, a weight memory and a bias register, a multiplier, a product register, an adder, a demultiplexer and an accumulation register; The weight memory is used to store the weights of the multiplication operations performed by each multiplier; The bias register is used to store the bias accumulated by the addition tree; The data selector is used to obtain the feature points to be calculated in the shared feature map and store them in the ping-pong window register; The multiplexer is used to extract the feature points to be calculated from the ping-pong window register and input them into multiple multipliers in parallel; Each multiplier is used to obtain a weight from the weight memory and perform a product operation with the feature point to be calculated, and the product result is stored in the product register; The addition tree is used to obtain the product result from the product register and obtain the bias from the bias register and accumulate; The demultiplexer is used to store part of the accumulated results into the accumulation register and send part of the accumulated results back to the addition tree for the next round of addition operation; The result of the accumulation register is stored in a downstream intermediate buffer memory.

4. The hardware acceleration system according to claim 1, It is characterized in that The process of generating the pixel area where the feature point is located by the feature map pixel point selector includes: Determine the coordinates (i, j) of the current valid feature point in the confidence feature map; Divide the coordinates (i, j) by k to get the coordinates (x, y). k is the ratio of the side length of the confidence feature map to the side length of the shared feature map; Round the division result to get the center pixel (x1, y1), and determine the three interpolation pixels based on the remainder of the division result: If i÷k has a remainder and j÷k has a remainder, the three interpolated pixels are (x1,y1-1), (x1-1,y1) and (x1-1,y1-1); If i÷k has a remainder and j÷k is an integer, the three interpolated pixel points are (x1,y1+1), (x1-1,y1) and (x1-1,y1+1); If i÷k is an integer and j÷k has a remainder, the three interpolated pixels are (x1,y1-1), (x1+1,y1) and (x1+1,y1-1); If i÷k is an integer and j÷k is an integer, the three interpolated pixels are (x1,y1+1), (x1+1,y1) and (x1+1,y1+1); A pixel region of a current valid feature point is determined, wherein the central pixel point and three interpolation pixel points are located in the central region of the pixel region.

5. The hardware acceleration system according to claim 4, It is characterized in that The hardware acceleration system further comprises a shared feature map cache for storing the shared feature map; The feature map pixel point selector includes a FIFO module, a reordering module, an interpolation module, a first address generation module, an interval generation module and a feature point address synthesis module; The FIFO module is used to cache the coordinates of valid feature points; The reorderer module is used to obtain the row coordinates of the current valid feature point from the FIFO module, and determine the row index of the valid feature point in the shared feature map cache based on the row coordinates to obtain the reordering number; The interpolation module is used to obtain the coordinates of the current valid feature point from the FIFO module and determine the pixel area of ​​the current valid feature point; The first address generating module is used to generate the first address of the pixel area in the shared feature map cache according to the rearrangement number and the position of the first pixel point of the pixel area; The interval generating module is used to determine the address difference of each pixel according to the relative position of each pixel point in the pixel area; The feature point address synthesis module is used to determine the address of each pixel point in the pixel area in the shared feature map cache according to the first address and the address difference.

6. The hardware acceleration system according to claim 1, It is characterized in that The SoftMax&NMS module includes: The SoftMax submodule includes a subregion division module, an approximate calculation module, an accumulation register, a first comparison and controller, a position information register and a divider; the subregion division module is used to divide the confidence feature map into multiple subregions, and the approximate calculation module is used to perform exponential operation on the feature data α of each pixel point in the same subregion to obtain the operation result 2 α The accumulation register is used to accumulate all the operation results of the same sub-region to obtain the accumulation result ∑2 α The first comparison and controller is used to determine the feature data α with the largest exponential operation result in the same sub-region max , the position information register is used to store α max Location information max ; The divider is used to perform the division operation α max ÷∑2 α , get the corresponding sub-region location information E max The effective value of The NMS submodule is used to sequentially obtain sub-areas whose valid values ​​are not set to 0 and use them as the central area, compare the current central area and the adjacent valid values ​​that are within the set distance range from the current central area valid value and have not been compared with the current central area valid value. If the central area valid values ​​are all greater than the adjacent valid values, the pixel point at the location of the current central area valid value is identified as a feature point and the adjacent valid value is set to 0. Otherwise, there is no feature point in the current central area and the adjacent valid value is maintained unchanged.

7. The hardware acceleration system according to claim 6, It is characterized in that The SoftMax submodule also includes a maximum index register and a maximum register, wherein the maximum index register is used to store the index of the pixel point with the largest exponential operation result, and the maximum register is used to store the largest exponential operation result; The first comparison and controller is used for comparing the current exponential operation result with the exponential operation result in the maximum value register whenever the approximate calculation module outputs an exponential operation result. If the former is greater than the latter, the current exponential operation result is stored in the maximum value register as the updated maximum exponential operation result, and the index of the current pixel point is stored in the maximum value index register as the index of the pixel point with the maximum exponential operation result after the update; After the comparison of all the pixels in the current sub-region is completed, the position information of the corresponding pixel is stored in the position information register according to the index of the pixel in the maximum value index register.

8. The hardware acceleration system according to claim 6, It is characterized in that The NMS submodule includes a first selector, a second selector, and a second comparison and controller; The first selector is used to obtain the position information of the current central area effective value and the position information of the adjacent sub-area effective value that has not been compared with the current central area effective value and send them to the second comparison and controller; The second comparison and controller is used to first calculate the position distance between the current central area effective value and the adjacent sub-area effective value and determine whether the position distance between the two effective values ​​is within the set range. If so, the adjacent sub-area is considered to be a valid sub-area; The second selector is used to obtain the effective value of the current central area and the effective value of the effective sub-area and send them to the second comparison and controller; The second comparison and controller is also used to compare the size of the effective value of the current central area and the effective values ​​of each effective sub-area. If the effective value of the current central area is greater than the effective values ​​of each effective sub-area, the pixel point at the position of the effective value of the current central area is output as a feature point and the adjacent effective values ​​are set to 0 and the effective value of the effective sub-area is set to 0. Otherwise, it is considered that there is no feature point in the current central area and the effective values ​​are maintained unchanged.

9. The hardware acceleration system according to any one of claims 6 to 8, It is characterized in that The set range is the area covered by a rectangular window centered at the location of the current central area valid value, and the length and width of the rectangular window are both larger than the length and width of the sub-area and smaller than twice the length and width of the sub-area.

10. A hardware acceleration method based on the hardware acceleration system according to any one of claims 1 to 9, It is characterized in that include: Step S1: At the first stage of the pipeline, the input image enters the coded convolution processor for feature extraction, and a shared feature map is obtained and stored in the shared feature map cache; Step S2: At the second stage of the pipeline, the feature data in the shared feature map cache is sent to the feature point convolution processor for convolution to obtain the confidence feature map and store it in the pipeline register. At this time, before the descriptor convolution is performed, the system has obtained the feature point information; Step S3: At the third stage of the pipeline, the confidence feature map in the pipeline register is sent to the SoftMax&NMS module to calculate the feature confidence of each pixel and perform non-maximum suppression to obtain feature points; Step S4: In the first half of the fourth stage of the pipeline, the pixel area where the feature point is located is selectively generated according to the currently valid feature point position information by the feature map pixel point selector; Step S5: in the second half of the fourth stage of the pipeline, the feature data located in the pixel area is read from the shared feature map by a selective descriptor convolution processor and descriptor convolution is performed to generate a descriptor of the valid feature point; Step S6: at the fifth stage of the pipeline, the descriptors of the valid feature points are interpolated and generally normalized by an interpolation and normalization structure to obtain a descriptor feature map; Among them, the encoding convolution processor, the feature point convolution processor and the SoftMax&NMS module all complete the corresponding pipeline tasks within M cycles, and the selective descriptor convolution processor completes the generation of a valid feature point descriptor within N*M cycles, N>1.

Citation Information

Patent Citations

  • FPGA-based reconfigurable convolutional neural network accelerator

    CN113222130A

  • Method and system for convolution model multi-mode hardware accelerator

    CN113853616A

  • Compiler and hardware abstraction layer architecture for a neural network accelerator

    US10789402B1

  • Deep learning object detection processing device

    WO2022107925A1

Cited By

  • Voice wake-up method, system, device and equipment

    CN116386643A