A hardware acceleration method and system based on neural network feature extraction

Abstract

The application belongs to the technical field of image processing, and discloses a hardware acceleration method and system based on neural network feature extraction, which comprises an encoding convolution processor running in a pipeline mode, a shared feature map cache, a feature point detection structure, a feature image pixel point selector and a descriptor generation structure; the encoding convolution processor performs convolution on an input image and stores the output to the shared feature map cache; the feature point detection structure generates feature points through convolution and post-processing; the feature image pixel point selector generates a pixel region according to feature point information; and the descriptor generation structure reads only pixel region data related to the feature points from the shared feature map and calculates descriptors. The end-to-end full hardware acceleration design avoids external storage access, the pipeline design improves operation efficiency, the pixel region of the feature points can be obtained in advance, the descriptor generation structure only processes the pixel region through the feature image pixel point selector, and therefore resource consumption can be reduced and system energy efficiency can be improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and more specifically, relates to a hardware acceleration method and system based on neural network feature extraction. Background Technology

[0002] The SuperPoint algorithm is a neural network-based feature extraction algorithm that can solve matching problems between two images caused by translation, rotation, affine transformations, etc. SuperPoint converts an image into a set of numerous local feature vectors (descriptors). Compared to existing feature extraction algorithms such as LIFT, SIFT, and ORB, SuperPoint boasts the highest average nearest neighbor accuracy, the highest matching score, and the best overall homography metric, demonstrating powerful image feature extraction capabilities. This algorithm can still achieve relatively reliable matching results even under conditions of affine transformations, 3D viewpoint changes, scaling, exposure variations, and noise. Therefore, the SuperPoint algorithm is widely used in intelligent autonomous mobile robots and autonomous driving applications, including mapping and localization, matching and reconstruction, motion segmentation and tracking, and panoramic image fusion—scenarios requiring image recognition and matching.

[0003] Existing hardware acceleration systems for implementing the SuperPoint algorithm suffer from low energy efficiency and high resource consumption. In 2020, Zhilin Xu et al. from Tsinghua University used Xilinx's general-purpose AI acceleration IP core DPU (Deep Learning Processing Unit) to implement convolution operations in the SuperPoint network, and subsequently focused on reducing the computational load of post-convolution processing. In 2022, Ye Liu et al. from the University of Electronic Science and Technology of China also used a DPU to implement convolution operations, but they reduced and adjusted the convolution network, ensuring that the feature point detection module in the two parallel modules completed the operation before the descriptor generation module. While the above two hardware acceleration system designs for implementing the SuperPoint feature extraction algorithm have increased processing speed compared to the original software implementation, these existing hardware acceleration system designs still have the following shortcomings:

[0004] Existing SuperPoint hardware acceleration systems not only require embedded CPUs to handle frequent off-chip memory accesses caused by frequent DPU IP convolution instructions and inter-layer convolution operations, but also require post-processing operations via the CPU, resulting in high power consumption and low energy efficiency in hardware acceleration systems.

[0005] The existing SuperPoint hardware acceleration system has redundant post-processing operations, and the data comparison between adjacent pixels in the NMS module has redundant and repeated comparisons, which further limits the improvement of energy efficiency. Summary of the Invention

[0006] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a hardware acceleration method and system based on neural network feature extraction, the purpose of which is to improve the energy efficiency of feature extraction.

[0007] To achieve the above objectives, according to one aspect of the present invention, a hardware acceleration system based on neural network feature extraction is provided, comprising a pipelined encoding convolution processor, a feature point convolution processor, a SoftMax&NMS module, a feature map pixel selector, a selective descriptor convolution processor, and an interpolation and normalization module.

[0008] The coded convolutional processor is used to extract features from the input image to obtain a shared feature map;

[0009] The feature point convolution processor is used to convolve the shared feature map to obtain a confidence feature map;

[0010] The SoftMax&NMS module is used to extract candidate feature points from the confidence feature map;

[0011] The feature map pixel selector is used to selectively generate the pixel region where the feature point is located based on the currently valid feature point location information;

[0012] The selective descriptor convolution processor is used to read feature data located in the pixel region from the shared feature map and perform descriptor convolution to generate descriptors for the effective feature points;

[0013] The interpolation and normalization structure is used to interpolate and normalize the descriptors of the effective feature points to obtain a descriptor feature map.

[0014] The encoding convolution processor, the feature point convolution processor, and the SoftMax&NMS module all complete their respective pipeline tasks within M cycles, and the selective descriptor convolution processor generates a valid feature point descriptor within N*M cycles, where N>1.

[0015] In some embodiments, the time required for the coded convolutional processor to compute an output pixel feature is M cycles, and the pipelined approach includes:

[0016] The coded convolutional processor sequentially obtains the output pixel features every M cycles;

[0017] The feature point convolution processor completes the calculation of the feature of one output pixel every M cycles to obtain the first calculation result;

[0018] The SoftMax&NMS module completes the calculation of a first calculation result every M cycles to obtain a second calculation result;

[0019] The selective descriptor convolution processor completes the calculation of a second calculation result every N*M cycles to obtain the descriptor of the corresponding feature point;

[0020] Where N is equal to or approximately equal to the ratio of the sum of the areas of all feature point regions to the total area of ​​the overall shared feature map.

[0021] In some embodiments, the encoded convolution processor includes a pipelined intermediate cache memory, convolution modules, and max pooling modules, each of the convolution modules including a data selector, a ping-pong window register, a weight memory and a bias register, a multiplier, a product register, an adder, a demultiplexer, and an accumulation register.

[0022] The weight memory is used to store the weights of each multiplier for multiplication operations;

[0023] The bias register is used to store the bias for accumulation in the addition tree;

[0024] The data selector is used to acquire the feature points to be calculated in the shared feature map and store them in the ping-pong window register;

[0025] The multiplexer is used to extract the feature points to be calculated from the ping-pong window register and input them into multiple multipliers in parallel.

[0026] Each multiplier is used to retrieve weights from the weight memory and perform a product operation with the feature points to be calculated, and the product result is stored in the product register.

[0027] The addition tree is used to obtain the product result from the product register and to obtain the bias from the bias register and perform accumulation;

[0028] The demultiplexer is used to store part of the accumulated result into the accumulated register and send part of the accumulated result back to the addition tree for the next round of addition operation;

[0029] The result of the accumulation register is stored in the intermediate cache memory connected later.

[0030] In some embodiments, the process by which the feature map pixel selector generates the pixel region where the feature point is located includes:

[0031] Determine the coordinates (i,j) of the current valid feature point in the confidence feature map;

[0032] Divide the coordinates (i,j) by k to obtain the coordinates (x,y). k is the ratio of the side length of the confidence feature map to the side length of the shared feature map;

[0033] Round the division result to obtain the center pixel (x1, y1). Determine three interpolation pixels based on the remainder of the division result:

[0034] If i ÷ k has a remainder and j ÷ k has a remainder, then the three interpolated pixels are (x1, y1-1), (x1-1, y1) and (x1-1, y1-1) respectively.

[0035] If i ÷ k has a remainder and j ÷ k is an integer, then the three interpolated pixels are (x1, y1+1), (x1-1, y1) and (x1-1, y1+1) respectively.

[0036] If i÷k is an integer and j÷k has a remainder, then the three interpolated pixels are (x1,y1-1), (x1+1,y1) and (x1+1,y1-1) respectively.

[0037] If i÷k is an integer and j÷k is an integer, then the three interpolated pixels are (x1, y1+1), (x1+1, y1) and (x1+1, y1+1) respectively.

[0038] Determine the pixel region of the current valid feature point, wherein the center pixel and three interpolated pixels are located in the central region of the pixel region.

[0039] In some embodiments, the hardware acceleration system further includes a shared feature map cache for storing the shared feature maps;

[0040] The feature map pixel selector includes a FIFO module, a reordering module, an interpolation module, a first address generation module, an interval generation module, and a feature point address synthesis module.

[0041] The FIFO module is used to cache the coordinates of valid feature points;

[0042] The reorderer module is used to obtain the row coordinates of the current valid feature points from the FIFO module, and determine the row index of the valid feature points in the shared feature map cache based on the row coordinates to obtain the reorder number;

[0043] The interpolation module is used to obtain the coordinates of the current valid feature point from the FIFO module and determine the pixel region of the current valid feature point;

[0044] The first address generation module is used to generate the first address of the pixel region in the shared feature map cache based on the rearrangement number and the position of the first pixel of the pixel region;

[0045] The interval generation module is used to determine the address difference of each pixel based on the relative position of each pixel within the pixel region;

[0046] The feature point address synthesis module is used to determine the address of each pixel in the pixel region in the shared feature map cache based on the starting address and the address difference.

[0047] In some embodiments, the SoftMax&NMS module includes:

[0048] The SoftMax submodule includes a sub-region partitioning module, an approximation calculation module, an accumulation register, a first comparison and controller, a position information register, and a divider. The sub-region partitioning module divides the confidence feature map into multiple sub-regions, and the approximation calculation module performs exponential operations on the feature data α of each pixel within the same sub-region to obtain the calculation result 2. α The accumulator register is used to sum all the operation results of the same sub-region to obtain the accumulated result ∑2. α The first comparison and controller is used to determine the feature data α with the largest exponential operation result in the same sub-region. max The position information register is used to store α max Location information E max The divider is used to perform division operation α. max ÷∑2 α The location information E of the corresponding sub-region is obtained. max The effective value;

[0049] The NMS submodule is used to sequentially obtain the sub-regions whose valid values ​​are not set to 0 and use them as the center region. It compares the current center region with the adjacent valid values ​​within a set distance from the current center region's valid value that have not been compared with the current center region's valid value. If the valid values ​​of the center region are all greater than the adjacent valid values, then the pixel at the location of the current center region's valid value is identified as a feature point and the adjacent valid values ​​are set to 0. Otherwise, there are no feature points in the current center region and the adjacent valid values ​​remain unchanged.

[0050] In some embodiments, the SoftMax submodule further includes a maximum value index register and a maximum value register, wherein the maximum value index register is used to store the index of the pixel with the largest exponential operation result, and the maximum value register is used to store the largest exponential operation result;

[0051] The first comparison and controller is used to compare the current exponential operation result with the exponential operation result in the maximum value register whenever the approximate calculation module outputs an exponential operation result. If the current one is greater than the latter, the current exponential operation result is stored in the maximum value register as the updated maximum exponential operation result, and the index of the current pixel is stored in the maximum value index register as the index of the pixel with the largest updated exponential operation result.

[0052] After comparing all pixels in the current sub-region, the position information of the corresponding pixel is stored in the position information register according to the index of the pixel in the maximum value index register.

[0053] In some embodiments, the NMS submodule includes a first selector, a second selector, and a second comparer and controller;

[0054] The first selector is used to obtain the location information of the current central region's valid value and the location information of the valid values ​​of adjacent sub-regions that have not been compared with the current central region's valid value, and send them to the second comparison and controller;

[0055] The second comparison and controller is used to first calculate the positional distance between the current effective value of the central region and the effective value of the adjacent sub-region and determine whether the positional distance between the two effective values ​​is within the set range. If so, the adjacent sub-region is considered to be a valid sub-region.

[0056] The second selector is used to obtain the current valid value of the central region and the valid values ​​of the valid sub-regions and send them to the second comparison and controller;

[0057] The second comparison and controller is also used to compare the effective value of the current central region with the effective values ​​of each effective sub-region. If the effective value of the current central region is greater than the effective values ​​of each effective sub-region, the pixel at the location of the effective value of the current central region is output as the feature point, the adjacent effective values ​​are set to 0, and the effective values ​​of the effective sub-regions are set to 0. Otherwise, it is considered that there are no feature points in the current central region and the effective values ​​are kept unchanged.

[0058] In some embodiments, the set range is the area covered by a rectangular window centered on the location of the current valid value of the central region, wherein the length and width of the rectangular window are both greater than the length and width of the sub-region and both are less than twice the length and width of the sub-region.

[0059] According to another aspect of the present invention, a hardware acceleration method based on the above-described hardware acceleration system is provided, comprising:

[0060] Step S1: In the first stage of the pipeline, the input image enters the coding convolution processor for feature extraction, obtains a shared feature map, and stores it in the shared feature map cache;

[0061] Step S2: In the second stage of the pipeline, the feature data in the shared feature map buffer is sent to the feature point convolution processor for convolution to obtain the confidence feature map and store it in the pipeline register. At this time, the system has already obtained the feature point information before performing descriptor convolution.

[0062] Step S3: In the third stage of the pipeline, the confidence feature map in the pipeline register is sent to the SoftMax&NMS module to calculate the feature confidence of each pixel and perform non-maximum suppression to obtain the feature points;

[0063] Step S4: In the first half of the fourth stage of the pipeline, the pixel region where the feature point is located is selectively generated by the feature map pixel selector based on the currently valid feature point position information.

[0064] Step S5: In the latter half of the fourth stage of the pipeline, the feature data located in the pixel region is read from the shared feature map by the selective descriptor convolution processor and descriptor convolution is performed to generate the descriptor of the effective feature point;

[0065] Step S6: In the fifth stage of the pipeline, the descriptors of the effective feature points are interpolated and normalized by interpolation and normalization structure to obtain the descriptor feature map;

[0066] The encoding convolution processor, the feature point convolution processor, and the SoftMax&NMS module all complete their respective pipeline tasks within M cycles, and the selective descriptor convolution processor generates a valid feature point descriptor within N*M cycles, where N>1.

[0067] In summary, compared with the prior art, the technical solutions conceived in this invention have the following main advantages:

[0068] 1. The hardware acceleration system based on neural network feature extraction provided by this invention implements the SuperPoint algorithm by designing a dedicated end-to-end hardware acceleration system. This avoids the frequent off-chip memory accesses caused by embedding the CPU to handle frequent DPU IP convolution instructions and inter-layer convolution operations. It also avoids post-processing operations through the CPU. Through dedicated acceleration design, it avoids off-chip memory access consumption and significantly improves energy efficiency.

[0069] 2. The hardware acceleration system based on neural network feature extraction provided by the present invention, wherein the encoding convolution processor, feature point convolution processor, SoftMax&NMS module, selective descriptor convolution processor, interpolation and normalization module operate in a pipeline manner, which can further improve the overall operating efficiency of the structure.

[0070] 3. The hardware acceleration system based on neural network feature extraction provided by this invention, due to its pipeline operation mode, can predetermine effective feature points. After determining the effective feature points, the feature map pixel selector can select the pixel region where the feature points are located. When generating descriptors in the fourth-level pipeline, only the feature data in the pixel region needs to be convolved, avoiding redundant descriptor convolution of all non-feature points, and further improving computational efficiency.

[0071] 4. The hardware acceleration system based on neural network feature extraction provided by this invention considers the sparse distribution of feature points in the shared feature map. If the feature point convolution processor and the descriptor convolution processor are allocated computing resources of equal scale to the network, the computing resources of the descriptor convolution processor will be idle for a long time. The ratio of its idle time to its working time is close to the ratio of the sum of the areas of the feature point regions to the total area of ​​the overall shared feature map. In this invention, the processing time of the selective descriptor convolution processor is longer, which can reduce the computational hardware resources of the descriptor convolution processor, such as multipliers. Specifically, N can be made equal to the ratio of the sum of the areas of the shared feature map occupied by all feature point regions to the total area of ​​the shared feature map, thus reducing hardware overhead. Therefore, the computational resource overhead of the descriptor convolution module can be reduced without reducing the system's running speed.

[0072] 5. The end-to-end accelerator designed in this invention supports convolution on only a portion of the feature map. Current methods primarily utilize the DPU for convolution acceleration, but the DPU's general computation results cannot convolve a specific segment of the shared feature map; it can only perform overall convolution on the shared feature map. Furthermore, in this invention, the pipeline level for feature point convolution is higher than that for descriptor convolution. Therefore, the location information of the feature points can be obtained before descriptor convolution, allowing for the selection of a portion of the region centered on the feature points for descriptor convolution. Thus, thanks to the pipeline design, the feature point positions can be obtained before descriptor computation. Descriptor convolution processing does not require convolution on the entire shared feature map; convolution on the feature points and their nearby pixel regions is sufficient to obtain the descriptor, avoiding all redundant descriptor convolution processes and reducing computational resource overhead.

[0073] 6. In a specific embodiment, the Softmax submodule design divides the system into multiple sub-regions. Based on the characteristic that the maximum value output by the Softmax function is most likely to be retained after the NMS operation, only the maximum value of the sub-region is retained for calculation. This simplifies the calculation process, reduces the number of outputs from the Softmax submodule, and decreases the number of data comparisons in the NMS submodule. Furthermore, considering that the absolute value of the Softmax result is not critical, this invention innovatively uses a base-2 exponential operation as an approximation operation while ensuring that the relative size of the Softmax result remains unchanged, which further simplifies the calculation. In the NMS submodule design, setting the value to 0 during comparison avoids repeated comparisons, thus allowing for repeated comparisons. By improving the SoftMax & NMS module, the Softmax & NMS operation process can be simplified, significantly reducing circuit resource overhead and system power consumption. Attached Figure Description

[0074] Figure 1 This is a structural block diagram of a hardware acceleration system according to an embodiment of the present invention;

[0075] Figure 2 This is a schematic diagram of the pipeline operation of a hardware acceleration system in one embodiment of the present invention;

[0076] Figure 3 This is a schematic diagram of the structure of an coded convolutional processor according to an embodiment of the present invention;

[0077] Figure 4 This is a structural diagram of the Conv2a module in one embodiment of the present invention;

[0078] Figure 5 This is a schematic diagram illustrating the determination of interpolation based on the remainder in one embodiment of the present invention;

[0079] Figure 6 This is a structural block diagram of a feature map pixel selector according to an embodiment of the present invention;

[0080] Figure 7 This is a schematic diagram of the SoftMax&NMS module in one embodiment of this application;

[0081] Figure 8 This is a traversal direction map of the central region in one embodiment of the present invention;

[0082] Figure 9 This is a schematic diagram comparing the central region and adjacent sub-regions in one embodiment of the present invention. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0084] like Figure 1 The diagram shown illustrates the structural block of a hardware acceleration system according to an embodiment of the present invention. It mainly includes a pipelined coded convolutional processor, a feature point detection structure, a feature map pixel selector, and a descriptor generation structure. The coded convolutional processor extracts features from the input image to obtain a shared feature map. The feature point detection structure calculates and outputs feature points. The feature map pixel selector selectively generates pixel regions containing feature points based on currently valid feature point information. The descriptor generation structure reads feature data located in pixel regions from the shared feature map and performs descriptor convolution and post-processing to generate corresponding descriptors. Based on this end-to-end hardware acceleration system, the SuperPoint algorithm can be implemented.

[0085] The feature point detection structure includes a feature point convolution processor and a SoftMax&NMS module. The feature point convolution processor is used to convolve the shared feature map to obtain a confidence feature map, and the SoftMax&NMS module is used to extract feature points from the confidence feature map. The feature map pixel selector selectively generates the pixel region where the feature point is located based on the currently valid feature point location information.

[0086] The descriptor generation structure is used to read feature data located in pixel regions from the shared feature map and perform descriptor convolution to generate descriptors of valid feature points.

[0087] Understandably, the hardware acceleration system also features a shared feature map cache for storing shared feature maps generated by the coded convolution processor.

[0088] In one embodiment, the descriptor generation structure includes a selective descriptor convolution processor and an interpolation and normalization module. The selective descriptor convolution processor is used to obtain feature values ​​of the pixel regions where feature points are located from the shared feature map provided by the feature map pixel selector and perform convolution processing; the interpolation and normalization module is used to interpolate and normalize the convolution result of the selective descriptor convolution processor so that the size of the feature map matches the original image, thereby calculating the descriptor of the corresponding feature point.

[0089] In this invention, the encoding convolution processor, feature point detection structure, and descriptor generation structure operate in a pipelined manner, which can further improve the overall operational efficiency of the structure. Simultaneously, due to the pipelined operation, effective feature points can be determined in advance. After determining the effective feature points, the pixel regions where these points are located can be identified. When generating descriptors, only the feature data within the pixel regions needs to be convolved, further improving computational efficiency.

[0090] The following section will first introduce the assembly line operation mode.

[0091] Specifically, the encoding convolution processor, feature point convolution processor, SoftMax&NMS module, selective descriptor convolution processor, and interpolation and normalization module operate in a pipeline manner. The encoding convolution processor takes M cycles to calculate the feature of an output pixel. The encoding convolution processor obtains the output pixel feature sequentially within each M cycles. The feature point convolution processor completes the calculation of the feature of an output pixel within each M cycles, obtaining the first calculation result. The SoftMax&NMS module completes the calculation of the first calculation result within each M cycles, obtaining the second calculation result. The selective descriptor convolution processor completes the calculation of the second calculation result within each N*M cycles, obtaining the third calculation result. The interpolation and normalization module sequentially calculates the third calculation result to obtain the descriptor of the corresponding feature point.

[0092] For the pipelined processing of the main computation modules in the hardware-accelerated system, the computation time for each output feature pixel of the encoding convolution processor is designed to be M cycles. To achieve matched pipelined computation at the module level, the feature point convolution processor and the SoftMax&NMS module are designed to compute a point in the feature map in less than or equal to M cycles. Considering the redundancy of descriptor computation—that is, non-feature points do not require descriptor computation—the computation speed of the selective descriptor convolution processor is designed to be 1 / N of the speed of the feature point convolution processor. Therefore, the computation time for the selective descriptor convolution processor to compute one pixel is designed to be M×N cycles. The interpolation and normalization modules compute one pixel in the descriptor map in less than or equal to M×N cycles.

[0093] like Figure 2The diagram illustrates the pipeline operation of a hardware acceleration system according to an embodiment of the present invention. For example, to achieve a processing speed of 20 frames per second for a VGA image (640×480) at a clock frequency of 200MHz, the corresponding convolution calculation for each pixel in the input image must be completed within 32 cycles. Since the encoder network contains three 2×2 max-pooling layers (i.e., a downsampling rate of 4), the duration of the output pixel after each max-pooling layer is four times that of the input pixel. Therefore, the computation time for each output feature pixel of the encoder is designed to be 2048 cycles, i.e., 32×4×4×4. To achieve matched pipeline processing at the module level, the feature point convolution processor and the SoftMax&NMS post-processing module are designed to compute a point in the feature map within 2048 cycles or less. Considering the data mismatch characteristics discussed earlier, the computation speed of the descriptor decoder is designed to be one-quarter of the feature point decoder speed. Therefore, the computation time for the selective descriptor convolution processor to compute one pixel is designed to be 8192 cycles. The interpolation and normalization modules are designed to compute one pixel in the descriptor map in 786 cycles. The total latency of the proposed pipeline architecture is approximately 5780 cycles and 14758 cycles, respectively, for feature point and descriptor computation.

[0094] In one embodiment, the encoding convolution processor includes a pipelined intermediate cache memory, a convolution module, and a max-pooling module. For example... Figure 3 The diagram shown is a schematic of the structure of an encoding convolution processor in one embodiment of the present invention, which consists of an intermediate buffer, a pipelined convolution module, and a max pooling module.

[0095] Each convolutional module completes the convolution operation on the input feature pixels within a specified clock cycle (i.e., 32-2048 cycles). Taking the first two layers as an example, the multipliers of the Conv1a module sequentially execute eight 3×3 convolutions, performing 3 multiplications per cycle, requiring a total of 24 cycles. Subsequently, Buffcv1b stores three rows of feature maps with 8 channels. The Conv1b module uses 18 multipliers to sequentially execute eight 3×3×8 filter convolutions, calculating 3×3×2 elements per cycle, corresponding to 1 / 4 of a filter, requiring 32 cycles to complete. It is worth noting that the convolutional layer pipeline architecture of the feature point convolution processor and the descriptor convolution processor is similar to that of the encoding convolution processor, the difference being the number of multipliers and the absence of a max-pooling module.

[0096] In one embodiment, each convolutional module includes a data selector, a ping-pong window register, a weight memory and a bias register, a multiplier, a product register, an adder, a demultiplexer, and an accumulation register;

[0097] The weight memory is used to store the weights of each multiplier in the multiplication operation;

[0098] The bias register is used to store the bias for accumulation in the addition tree;

[0099] The data selector is used to acquire the feature points to be calculated in the shared feature map and store them in the ping-pong window register;

[0100] Multiplexers are used to extract feature points to be computed from ping-pong window registers and input them into multiple multipliers in parallel;

[0101] Each multiplier is used to retrieve weights from the weight memory and perform a product operation with the feature points to be calculated. The product result is stored in the product register.

[0102] The addition tree is used to obtain the product result from the product register and to obtain the bias from the bias register and perform accumulation;

[0103] The demultiplexer is used to store part of the accumulated result into the accumulation register and send part of the accumulated result back to the addition tree for the next round of addition operation;

[0104] The result of the accumulator register is stored in the intermediate cache memory connected later.

[0105] like Figure 4The diagram shows the structure of one of the Conv2a modules in the coded convolutional processor according to an embodiment of the present invention. The data selector provides read address and read enable, writing the read input feature map data of a certain channel into the ping-pong window register in the correct order to ensure that each input feature map data can be correctly multiplied by its corresponding convolution weight value. The weight data and bias data in the neural network are stored in the weight memory and bias register, respectively. The data in each ping-pong window register is multiplexed for 16 clock cycles to complete the multiply-accumulate operations corresponding to the 16 filters. To achieve uninterrupted convolution operations, the convolutional layer submodule adopts a ping-pong structure, with the activation of ping-pong window register 1 or 2 implemented by a subsequent multiplexer. When the data in ping-pong window register 1 is used up, the data in ping-pong window register 2 can be used immediately, while ping-pong window register 1 can read the input feature map data of the next channel. During a convolution process in a convolutional layer submodule, due to the limited number of multipliers, it is impossible to calculate all channels of a pixel at once. Therefore, each calculation process is recorded as a partial sum and intermediate result. The convolution operation of that pixel is considered complete only when all the 3×3 multiply-add operations corresponding to all channels are completed. The accumulation operation circuit is implemented by a demultiplexer, an accumulation register, and a multiplexer. The demultiplexer is responsible for allocating the partial sums calculated by the addition tree to the corresponding accumulation registers. For example, the partial sum of the nth channel will be stored in Reg_n. After the partial sums stored in the accumulation register are selected by the multiplexer to the corresponding channel, they will be looped back into the addition tree in the next round of addition operations, and entered into the addition tree together with the bias weights for accumulation, thus realizing the accumulation of partial sums.

[0106] The key to the pipelined design of the convolution module lies in utilizing a fixed data flow of input. This is because the previous module continuously performs computations and passes the results to the intermediate buffer Buffcv2a, which stores feature maps with 8 channels and 3 rows. The convolution operation must be completed as early as possible, channel by channel. During the computation phase, in each cycle, 9 multipliers are used in parallel, multiplying the 9 pixels stored in the ping-pong window register with the 9 weights of one channel of a 3×3×8 filter. In the next cycle, the same 9 input feature map pixels are reused, multiplied with the 9 weights of the corresponding channel of the next filter. This process is repeated to complete the multiplication operation of the same channel of the current input feature map in different filters, thus efficiently achieving a fixed data flow of input. Through the pipelined accumulator, the results of 16 channels corresponding to a data point in the output feature map are finally obtained sequentially over 16 cycles.

[0107] In this invention, by designing a production line operation mode, the overall operating efficiency of the structure can be further improved.

[0108] The following section will introduce the specific design of the feature map pixel selector.

[0109] The process of determining a pixel region includes:

[0110] Determine the coordinates (i,j) of the current valid feature point in the confidence feature map;

[0111] Divide the coordinates (i,j) by k to obtain the coordinates (x,y). k is the side length of the confidence feature map / the side length of the shared feature map, and k is a positive integer;

[0112] Round the division result to obtain the center pixel (x1, y1). Determine three interpolation pixels based on the remainder of the division result:

[0113] If i ÷ k has a remainder and j ÷ k has a remainder, then the three interpolated pixels are (x1, y1-1), (x1-1, y1) and (x1-1, y1-1) respectively.

[0114] If i ÷ k has a remainder and j ÷ k is an integer, then the three interpolated pixels are (x1, y1+1), (x1-1, y1) and (x1-1, y1+1) respectively.

[0115] If i÷k is an integer and j÷k has a remainder, then the three interpolated pixels are (x1,y1-1), (x1+1,y1) and (x1+1,y1-1) respectively.

[0116] If i÷k is an integer and j÷k is an integer, then the three interpolated pixels are (x1, y1+1), (x1+1, y1) and (x1+1, y1+1) respectively.

[0117] Determine the pixel region of the current valid feature point, with the center pixel and three interpolated pixels located in the center of the pixel region.

[0118] Specifically, taking a shared feature map of 80×60×256 and a confidence feature map of 640×480 as an example, the shared feature map has a length of 80 and a width of 60, while the confidence feature map has a length of 640 and a width of 480. Therefore, the side length of the confidence feature map is 8 times that of the shared feature map. This means that the pixels in the confidence feature map are the pixels obtained by expanding both the length and width of the shared feature map by 8 times, where k = 8. Since the feature points determined from the 640×480 confidence feature map require extracting the feature data of the pixel region where the feature points are located from the 80×60×256 shared feature map, the coordinates (i,j) of the feature points in the 640×480 confidence feature map need to be reduced to obtain coordinates (x,y). Then, the pixel region in the 80×60×256 shared feature map is determined based on the coordinates (x,y).

[0119] Because coordinate scaling introduces errors, interpolation is used to improve coordinate accuracy. In this embodiment, the integer part of the coordinates (x, y) is used as the center pixel (x1, y1), and the interpolation strategy is determined based on the remainder. See below for details. Figure 5 As shown, this interpolation method can improve the accuracy of pixel regions. After determining the center pixel and the interpolation pixel, the pixel region corresponding to the feature point can be obtained by extending the region appropriately with the four pixel points as the center. For example, in this embodiment, the size of the pixel region is set to 8×8.

[0120] After determining the pixel region of the feature points, the descriptor generation structure processes the feature data located within the pixel region in the shared feature map to generate descriptors for the feature points. In this way, the descriptor generation structure only needs to process the feature points to generate the corresponding descriptors, avoiding redundant calculations for non-feature points, thus reducing hardware overhead and improving computational efficiency. Furthermore, by improving the localization accuracy of the pixel region through the aforementioned interpolation processing, and by only needing to perform convolution processing on the feature data within the pixel region when generating the descriptor, computational efficiency is further improved.

[0121] like Figure 6 The diagram shows a structural block diagram of a feature map pixel selector according to an embodiment of the present invention. It includes a FIFO module, a reordering module, an interpolation module, a first address generation module, an interval generation module, and a feature point address synthesis module. The FIFO module caches the coordinates of valid feature points. The reordering module obtains the row coordinates of the current valid feature point from the FIFO module and determines the row index of the valid feature point in the shared feature map cache based on the row coordinates, thus obtaining the reordering number. The interpolation module obtains the coordinates of the current valid feature point from the FIFO module and determines the pixel region of the current valid feature point. The first address generation module generates the first address of the pixel region in the shared feature map cache based on the reordering number and the position of the first pixel in the pixel region. The interval generation module determines the address difference of each pixel based on the relative position of each pixel within the pixel region. The feature point address synthesis module determines the address of each pixel within the pixel region in the shared feature map cache based on the first address and the address difference.

[0122] In this embodiment, redundant convolution operations are avoided by selecting only feature map data of feature points and their neighboring points. Specifically, the synchronous FIFO module caches the row and column coordinates of feature points, and outputs these coordinates to the reorderer and interpolator for subsequent data processing. The reorderer is responsible for reading the row number of the feature points and determining the row index to read in the shared feature map cache. Due to memory size limitations, the shared feature map cache can only store five rows of feature map data at a time. When a new row of data needs to be stored, it replaces the row in the current cache that is the remainder after dividing the row number by five. For example, after the first five rows of data have been read, the sixth row of data will be stored in the first row. Therefore, after determining the row number of the feature map, a modulo 5 operation needs to be performed on the row number to find the corresponding row in the cache memory. This remainder is called the reorder number, which is used to identify the storage location of the data, thereby ensuring that the data in the shared feature map cache can be read in the correct order. The interpolator determines the selection method of the feature points based on the row and column information of the feature points. Specifically, dividing the coordinate indices of the pixel at coordinate (i,j) in the 640×480 image containing the feature point by 8 yields the pre-interpolation coordinates (x,y) in the output feature map of the selective descriptor convolution processor. This invention designates the feature map pixel corresponding to the rounded coordinate (x,y) as the center pixel. The remaining three pixels required for interpolation are determined by the remainder after rounding (x,y), i.e., the remainder after dividing (i,j) by 8 determines the data selection method. The first address generator generates the first address of the feature point based on the number of rearrangements and interpolation. The interval generator calculates the address difference between each adjacent element during descriptor convolution. The 4*4 feature region address is obtained by adding the first address and the interval address.

[0123] The descriptor generation structure reads the feature points from the shared data cache memory using the above address, thus completing the selection of feature points calculated by the descriptor generator.

[0124] Based on the above design, the SoftMax&NMS module can be further improved to reduce the amount of computation and improve computing efficiency.

[0125] like Figure 7 The diagram shown is a structural schematic of the SoftMax & NMS module in one embodiment of this application, which mainly includes the SoftMax submodule and the NMS submodule.

[0126] The SoftMax submodule includes a sub-region partitioning module (not shown in the figure) and an approximation calculation module (i.e., appro_2). αThe module includes an accumulation register, a first comparison and controller, a position information register, and a divider; the sub-region division module is used to divide the confidence feature map into multiple sub-regions, and the approximate calculation module is used to perform exponential operations on the feature data α of each pixel in the same sub-region to obtain the calculation result 2. α The accumulator register is used to sum the results of all operations within the same sub-region to obtain the accumulated result ∑2. α The comparison and controller are used to determine the feature data α with the largest computational result in the same sub-region. max The position information register is used to store α max Location information E max The divider is used to perform division operations α. max ÷∑2 α The location information E of the corresponding sub-region is obtained. max Valid value.

[0127] Specifically, the size of the confidence feature map is the same as the size of the input image. For example, if the input image is a 640×480 VGA image, the encoding convolution processor will perform downsampling convolution processing to generate a shared feature map of 80 (length) × 60 (width) × 256 (channels). The feature point convolution processor will then convolve the shared feature map to obtain a feature map of 80 (length) × 60 (width) × 64 (channels). After channel flattening, the confidence feature map is restored to 640×480.

[0128] In the SoftMax submodule, the 640×480 confidence feature map is first divided into multiple subregions using a subregion partitioning module. For example, each subregion is 8×8 in size, resulting in 4800 subregions. Each subregion then undergoes an approximation calculation module, an accumulation register, a first comparison and control unit, and a divider to obtain the effective value for that subregion. The expression for this effective value is: Therefore, it is a number less than 1.

[0129] In the specific design, a maximum value index register and a maximum value register can be added. The maximum value index register stores the index of the pixel with the largest exponential result, and the maximum value register stores the largest exponential result. The comparison and control unit is used to compare the current exponential result with the exponential result in the maximum value register whenever the approximation calculation module outputs an exponential result. If the current result is greater than the latter, the current exponential result is stored in the maximum value register as the updated maximum exponential result, and the index of the current pixel is also stored in the maximum value index register as the index of the pixel with the largest exponential result in the updated exponential result. After comparing all pixels in the current sub-region, the position information of the corresponding pixel is stored in the position information register according to the pixel index in the maximum value index register.

[0130] Continuing with the example of an 8×8 sub-region, each of the 64 elements contained in the 8×8 sub-region is received one by one, and each element is processed by 2... α The approximate calculation, this process is performed by appro_2 α The approximation calculation module is implemented by using bit shifting to perform exponential calculations on the feature map data. The accumulator register is used for appro_2. α The module's calculation results are stored and accumulated in each cycle. The final accumulated value is used as the divisor in the CORDIC division operation. The Softmax submodule uses a successive comparison method, that is, for each input feature map data... α The result of the index calculation will be compared with the current 2 α Compare the maximum values; if the current input is 2... α If the calculated exponent is greater than the maximum recorded value, then the currently input 2 α The result of the exponent calculation is the current new maximum value, and 2 α The index is stored in the maximum index register. Thus, the 2^64 elements... α The maximum value in the approximate calculation result is selected and used as the dividend in the CORDIC division operation. The output position information register stores the position index of the maximum feature point data. The result of the CORDIC division operation is the output of the Softmax function.

[0131] In one embodiment, a buffer controller and a buffer can be set behind the SoftMax submodule. The valid value output by the SoftMax submodule is saved to the buffer through the buffer controller for the NMS submodule to read.

[0132] The NMS submodule is used to sequentially obtain the sub-regions that are not set to 0 and use them as the center region. It compares the current center region with the adjacent valid values ​​within a set distance from the current center region's valid value that have not been compared with the current center region's valid value. If the valid values ​​of the center region are all greater than the adjacent valid values, the pixel at the location of the current center region's valid value is identified as a feature point and the adjacent valid values ​​are set to 0. Otherwise, there are no feature points in the current center region and the adjacent valid values ​​remain unchanged.

[0133] like Figure 8The diagram shows the traversal direction of the central region in one embodiment of the present invention. Each square in the diagram represents a sub-region. First, the first sub-region is obtained as the central region. Its adjacent regions are a sub-region on the right and two sub-regions on the bottom. The valid values ​​of the three adjacent sub-regions are the adjacent valid values. After the determination, if the second sub-region is not set to 0, the second sub-region is used as the central region and compared with its adjacent valid values. If the second sub-region is set to 0, the second sub-region is skipped and the third sub-region is used as the next central region, and so on. This can reduce the amount of data processing, save hardware resources, and improve energy efficiency.

[0134] Understandably, the set range can be determined based on experience or experimental comparison. For example, the set range can be the area covered by a rectangular window centered on the location of the current valid value in the central region. The length and width of the rectangular window are both greater than the length and width of the sub-region and both less than twice the length and width of the sub-region. Figure 9 The diagram shown is a comparison diagram of the central region and adjacent sub-regions in one embodiment of the present invention. Taking sub-region (5) as the central region, (1), (2), (3), (4), (6), (7), (8), and (9) are all adjacent sub-regions. Since (1), (2), (3), and (4) have all been compared with (5) when they are respectively taken as the central region, it is not necessary to compare with (1), (2), (3), and (4) when taking sub-region (5) as the central region. It is only necessary to compare with sub-regions (6), (7), (8), and (9). In this embodiment, the set range is a 9×9 window centered on the effective value position of sub-region (5). After comparison, only sub-region (6) is within the set range. Therefore, it is only necessary to compare the effective values ​​of sub-region (5) and sub-region (6). If the effective value of sub-region (5) is larger, the pixel at the effective value of sub-region (5) is output as the feature point, and the effective value of sub-region (6) is set to 0. Otherwise, the presence of feature points is determined by taking sub-region (6) as the center area.

[0135] In its specific design, the NMS submodule includes a first selector, a second selector, and a second comparison and controller. The first selector is used to obtain the position information of the valid value of the current central region and the position information of the valid values ​​of adjacent sub-regions that have not been compared with the valid value of the current central region, and sends them to the second comparison and controller. The second comparison and controller is used to first calculate the positional distance between the valid value of the current central region and the valid values ​​of the adjacent sub-regions and determine whether the positional distance between the two valid values ​​is within a set range. If so, the adjacent sub-region is considered a valid sub-region. The second selector is used to obtain the valid value of the current central region and the valid values ​​of the valid sub-regions and send them to the second comparison and controller. The second comparison and controller is also used to compare the size of the valid value of the current central region and the valid values ​​of each valid sub-region. If the valid value of the current central region is greater than the valid values ​​of each valid sub-region, the pixel at the position of the valid value of the current central region is output as a feature point, and the adjacent valid values ​​are set to 0 and the valid values ​​of the valid sub-regions are set to 0. Otherwise, it is considered that there are no feature points in the current central region and the valid values ​​remain unchanged.

[0136] Specifically, the first selector reads the position information corresponding to the valid values ​​of the central sub-region and its adjacent sub-regions from the position information of the buffer. The adjacent sub-regions read here are filtered out from those already compared with the current central sub-region. These are then sent to the second comparison controller for comparison. The second comparison controller first determines the distance between the position of the valid value of the adjacent sub-region and the position of the valid value of the central sub-region. If the distance between the positions of the valid values ​​of the central sub-region and the adjacent sub-regions is within a 9×9 window, the adjacent sub-regions are considered valid and their valid values ​​can be compared. Otherwise, they are invalid sub-regions and no comparison is needed. The second selector selects the valid values ​​of the central sub-region and the valid sub-regions, retrieves these data, and compares their values. If the valid values ​​of the central sub-region are all greater than the valid values ​​of the adjacent sub-regions, the pixel corresponding to the valid value of the central sub-region is considered a feature point. The NMS submodule outputs the coordinates of the feature points and sets the valid values ​​of the valid sub-regions to 0.

[0137] In this embodiment, the SoftMax & NMS module is improved. The SoftMax submodule design divides the system into multiple sub-regions. Based on the characteristic that the maximum value output by the SoftMax function is most likely to be retained after the NMS operation, only the maximum value of the sub-region is retained for calculation. This simplifies the calculation process, reduces the number of outputs from the SoftMax submodule, and decreases the number of data comparisons in the NMS submodule. Furthermore, considering that the absolute value of the SoftMax result is not critical, this invention innovatively uses a base-2 exponential operation as an approximation while ensuring that the relative size of the SoftMax result remains unchanged, further simplifying the calculation. In the NMS submodule design, setting the value to 0 during comparison avoids repeated comparisons, thus allowing for repeated comparisons. By improving the SoftMax & NMS module, the SoftMax & NMS operation process is simplified, significantly reducing circuit resource overhead and system power consumption.

[0138] Accordingly, the present invention also relates to a hardware acceleration method based on the above hardware acceleration system, comprising:

[0139] Step S1: In the first stage of the pipeline, the input image enters the coding convolution processor for feature extraction, obtains a shared feature map, and stores it in the shared feature map cache;

[0140] Step S2: In the second stage of the pipeline, the feature data in the shared feature map buffer is sent to the feature point convolution processor for convolution to obtain the confidence feature map and store it in the pipeline register. At this time, the system has already obtained the feature point information before performing descriptor convolution.

[0141] Step S3: In the third stage of the pipeline, the confidence feature map in the pipeline register is sent to the SoftMax&NMS module to calculate the feature confidence of each pixel and perform non-maximum suppression to obtain the feature points;

[0142] Step S4: In the first half of the fourth stage of the pipeline, the pixel region where the feature point is located is selectively generated by the feature map pixel selector based on the currently valid feature point position information.

[0143] Step S5: In the latter half of the fourth stage of the pipeline, the feature data located in the pixel region is read from the shared feature map by the selective descriptor convolution processor and descriptor convolution is performed to generate descriptors of effective feature points.

[0144] Step S6: In the fifth stage of the pipeline, the descriptors of effective feature points are interpolated and normalized by interpolation and normalization structure to obtain the descriptor feature map;

[0145] Among them, the encoding convolution processor, the feature point convolution processor, and the SoftMax&NMS module all complete their corresponding pipeline tasks within M cycles, while the selective descriptor convolution processor completes the generation of a valid feature point descriptor within N*M cycles, where N>1.

[0146] Specifically, the design of each structural unit in the hardware acceleration system can be referred to the above introduction, and will not be repeated here.

[0147] In this invention, the SuperPoint algorithm is implemented using a dedicated end-to-end hardware acceleration system. This avoids the frequent off-chip memory accesses caused by embedding a CPU to handle frequent DPU IP convolution instructions and inter-layer convolution operations, and also avoids post-processing operations via the CPU, significantly improving energy efficiency. Furthermore, improvements to each hardware module simplify the computation process, greatly reducing circuit resource overhead and system power consumption.

[0148] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. It should be noted that the terms "in one embodiment," "for example," and "again" are intended to illustrate the present invention and are not intended to limit the present invention.

[0149] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A hardware acceleration system based on neural network feature extraction, characterized in that, This includes a pipelined encoding convolution processor, a feature point convolution processor, a SoftMax & NMS module, a feature map pixel selector, a selective descriptor convolution processor, and interpolation and normalization modules. The coded convolutional processor is used to extract features from the input image to obtain a shared feature map; The feature point convolution processor is used to convolve the shared feature map to obtain a confidence feature map; The SoftMax&NMS module is used to extract candidate feature points from the confidence feature map; The feature map pixel selector is used to selectively generate the pixel region where the feature point is located based on the currently valid feature point location information; The selective descriptor convolution processor is used to read feature data located in the pixel region from the shared feature map and perform descriptor convolution to generate descriptors for the effective feature points; The interpolation and normalization structure is used to interpolate and normalize the descriptors of the effective feature points to obtain a descriptor feature map. The encoding convolution processor, the feature point convolution processor, and the SoftMax&NMS module all complete their corresponding pipeline tasks within M cycles, while the selective descriptor convolution processor completes its tasks within N cycles. A valid feature point descriptor is generated within M cycles, where N > 1; The SoftMax&NMS module includes: The SoftMax submodule includes a sub-region partitioning module, an approximation calculation module, an accumulation register, a first comparison and controller, a position information register, and a divider. The sub-region partitioning module is used to divide the confidence feature map into multiple sub-regions, and the approximation calculation module is used to process the feature data of each pixel in the same sub-region. The result is obtained by performing exponentiation. The accumulator register is used to accumulate all the operation results of the same sub-region to obtain the accumulated result. The first comparison and controller is used to determine the feature data with the largest exponential operation result in the same sub-region. The location information register is used to store Location information The divider is used to perform division operations. The location information of the corresponding sub-region is obtained. The effective value; The NMS submodule is used to sequentially obtain the sub-regions whose valid values ​​are not set to 0 and use them as the center region. It compares the current center region with the adjacent valid values ​​within a set distance from the current center region's valid value that have not been compared with the current center region's valid value. If the valid values ​​of the center region are all greater than the adjacent valid values, then the pixel at the location of the current center region's valid value is identified as a feature point and the adjacent valid values ​​are set to 0. Otherwise, there are no feature points in the current center region and the adjacent valid values ​​remain unchanged.

2. The hardware acceleration system based on neural network feature extraction as described in claim 1, characterized in that, The time required for the coded convolutional processor to calculate the feature of one output pixel is M cycles, and the pipeline method includes: The coded convolutional processor sequentially obtains the output pixel features every M cycles; The feature point convolution processor completes the calculation of the feature of one output pixel every M cycles to obtain the first calculation result; The SoftMax&NMS module completes the calculation of a first calculation result every M cycles to obtain a second calculation result; The selective descriptor convolution processor performs per N The calculation of a second calculation result is completed within M cycles, and the descriptor of the corresponding feature point is obtained. Where N is equal to the ratio of the sum of the areas of all feature point regions to the total area of ​​the overall shared feature map.

3. The hardware acceleration system based on neural network feature extraction as described in claim 1, characterized in that, The encoded convolution processor includes a pipelined intermediate buffer memory, convolution modules, and a max pooling module. Each convolution module includes a data selector, a ping-pong window register, a weight memory and a bias register, a multiplier, a product register, an adder, a demultiplexer, and an accumulation register. The weight memory is used to store the weights of each multiplier for multiplication operations; The bias register is used to store the bias for accumulation in the addition tree; The data selector is used to acquire the feature points to be calculated in the shared feature map and store them in the ping-pong window register; The multiplexer is used to extract the feature points to be calculated from the ping-pong window register and input them into multiple multipliers in parallel; Each multiplier is used to retrieve weights from the weight memory and perform a product operation with the feature points to be calculated, and the product result is stored in the product register. The addition tree is used to obtain the product result from the product register and to obtain the bias from the bias register and perform accumulation; The demultiplexer is used to store part of the accumulated result into the accumulated register and send part of the accumulated result back to the addition tree for the next round of addition operation; The result of the accumulation register is stored in the intermediate cache memory connected later.

4. The hardware acceleration system based on neural network feature extraction as described in claim 1, characterized in that, The process by which the feature map pixel selector generates the pixel region where the feature point is located includes: Determine the coordinates of the current valid feature point in the confidence feature map. ; coordinates Divide by respectively , obtain coordinates , , , This is the ratio of the side length of the confidence feature map to the side length of the shared feature map; Round the result of the division to obtain the center pixel. The three interpolated pixels are determined based on the remainder of the division result: like There is a remainder, and If there is a remainder, then the three interpolated pixels are respectively , and ; like There is a remainder, and If the integer is true, then the three interpolated pixels are respectively , and ; like It is an integer, and If there is a remainder, then the three interpolated pixels are respectively , and ; like It is an integer, and If the integer is true, then the three interpolated pixels are respectively , and ; Determine the pixel region of the current valid feature point, wherein the center pixel and three interpolated pixels are located in the central region of the pixel region.

5. The hardware acceleration system based on neural network feature extraction as described in claim 4, characterized in that, The hardware acceleration system also includes a shared feature map cache for storing the shared feature maps; The feature map pixel selector includes a FIFO module, a reordering module, an interpolation module, a first address generation module, an interval generation module, and a feature point address synthesis module. The FIFO module is used to cache the coordinates of valid feature points; The reordering module is used to obtain the row coordinates of the current valid feature points from the FIFO module, and determine the row index of the valid feature points in the shared feature map cache based on the row coordinates to obtain the reordering number; The interpolation module is used to obtain the coordinates of the current valid feature point from the FIFO module and determine the pixel region of the current valid feature point; The first address generation module is used to generate the first address of the pixel region in the shared feature map cache based on the rearrangement number and the position of the first pixel of the pixel region; The interval generation module is used to determine the address difference of each pixel based on the relative position of each pixel within the pixel region; The feature point address synthesis module is used to determine the address of each pixel in the pixel region in the shared feature map cache based on the starting address and the address difference.

6. The hardware acceleration system based on neural network feature extraction as described in claim 1, characterized in that, The SoftMax submodule further includes a maximum value index register and a maximum value register. The maximum value index register is used to store the index of the pixel with the largest exponential operation result, and the maximum value register is used to store the largest exponential operation result. The first comparison and controller is used to compare the current exponential operation result with the exponential operation result in the maximum value register whenever the approximate calculation module outputs an exponential operation result. If the current one is greater than the latter, the current exponential operation result is stored in the maximum value register as the updated maximum exponential operation result, and the index of the current pixel is stored in the maximum value index register as the index of the pixel with the largest updated exponential operation result. After comparing all pixels in the current sub-region, the position information of the corresponding pixel is stored in the position information register according to the index of the pixel in the maximum value index register.

7. The hardware acceleration system based on neural network feature extraction as described in claim 1, characterized in that, The NMS submodule includes a first selector, a second selector, and a second comparer and controller; The first selector is used to obtain the location information of the current central region's valid value and the location information of the valid values ​​of adjacent sub-regions that have not been compared with the current central region's valid value, and send them to the second comparison and controller; The second comparison and controller is used to first calculate the positional distance between the current effective value of the central region and the effective value of the adjacent sub-region and determine whether the positional distance between the two effective values ​​is within the set range. If so, the adjacent sub-region is considered to be a valid sub-region. The second selector is used to obtain the current valid value of the central region and the valid values ​​of the valid sub-regions and send them to the second comparison and controller; The second comparison and controller is also used to compare the effective value of the current central region with the effective values ​​of each effective sub-region. If the effective value of the current central region is greater than the effective values ​​of each effective sub-region, the pixel at the location of the effective value of the current central region is output as the feature point, the adjacent effective values ​​are set to 0, and the effective values ​​of the effective sub-regions are set to 0. Otherwise, it is considered that there are no feature points in the current central region and the effective values ​​are kept unchanged.

8. The hardware acceleration system based on neural network feature extraction as described in claim 1, characterized in that, The set range is the area covered by a rectangular window centered on the location of the current effective value in the central region. The length and width of the rectangular window are both greater than the length and width of the sub-region and both less than twice the length and width of the sub-region.

9. A hardware acceleration method for a hardware acceleration system based on neural network feature extraction as described in any one of claims 1 to 8, characterized in that, include: Step S1: In the first stage of the pipeline, the input image enters the coding convolution processor for feature extraction, obtains a shared feature map, and stores it in the shared feature map cache; Step S2: In the second stage of the pipeline, the feature data in the shared feature map buffer is sent to the feature point convolution processor for convolution to obtain the confidence feature map and store it in the pipeline register. At this time, before performing descriptor convolution, the system has already obtained the feature point information. Step S3: In the third stage of the pipeline, the confidence feature map in the pipeline register is sent to the SoftMax&NMS module to calculate the feature confidence of each pixel and perform non-maximum suppression to obtain the feature points; Step S4: In the first half of the fourth stage of the pipeline, the pixel region where the feature point is located is selectively generated by the feature map pixel selector based on the currently valid feature point position information. Step S5: In the latter half of the fourth stage of the pipeline, the feature data located in the pixel region is read from the shared feature map by the selective descriptor convolution processor and descriptor convolution is performed to generate the descriptor of the effective feature point; Step S6: In the fifth stage of the pipeline, the descriptors of the effective feature points are interpolated and normalized by interpolation and normalization structure to obtain the descriptor feature map; The encoding convolution processor, the feature point convolution processor, and the SoftMax&NMS module all complete their corresponding pipeline tasks within M cycles, while the selective descriptor convolution processor completes its tasks within N cycles. A valid feature point descriptor is generated within M cycles, where N > 1.

Images