High-parallelism-degree data prefetching implementation method for target tracking hardware accelerator

Abstract

The invention provides a high-parallelism-degree data prefetching implementation method for a target tracking hardware accelerator. A controller, an input cache module, a weight cache module, a matrix processing unit, an output cache module, a write-back module and an external memory are included. The controller controls the calculated data flow by controlling the input cache module, the matrix processing unit and the output cache module; the input cache module is responsible for completing pre-fetching and pre-processing operations of input data; the weight caching module is used for completing prefetching of weight data; the matrix processing unit is used for receiving output data of the input cache module and the weight cache module, carrying out parallel operation and outputting a result to the output cache module, and the output of the output cache module is written back to an external memory through the write-back module. The configurable accelerator architecture based on the vector processing unit (VPU) is combined with the optimized sliding window design and the multi-level parallel strategy, so that the calculation performance and the energy efficiency ratio are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular, to a target tracking hardware accelerator high parallelism data prefetching implementation method. BACKGROUND

[0002] The existing hardware accelerator faces the following problems in the target tracking task: the traditional convolution accelerator has insufficient flexibility and is difficult to adapt to multi-size sliding windows and dynamic up-sampling operations; the difference between matrix operation and convolution operation data flow leads to low hardware resource utilization and low data prefetching efficiency; zero value padding and pooling operation introduces additional storage and calculation overhead; the balance between parallelism and power consumption is difficult to achieve.

[0003] Patent document CN113313252A discloses a deep separable convolution implementation method based on systolic array, which adopts M rows and N columns of processing units (PE units) to form a systolic array structure: in the horizontal direction, adjacent PE units are connected to each other, and the left PE unit can pass data to the right PE unit; in the vertical direction, each PE unit has its own data input port and data output port. The data prefetching module provides feature map data and weight parameters for the calculation array. The addition tree accumulates the partial sum data output by each column of PE units in parallel. Each PE unit is mainly composed of a register, a data selector, an adder and a multiplier. The systolic array with the above structure is adopted in the present application, and the data prefetching module and the addition tree are used to realize different data flow and data reuse modes, so as to realize the acceleration calculation of standard convolution, point convolution and deep convolution.

[0004] However, patent document CN113313252A can only support convolution operation, and has poor support for matrix operation which requires higher input data throughput. Meanwhile, only one data selector is used as the functional part of data reuse when prefetching input activation data in convolution operation, which leads to low parallelism on the sliding window during convolution operation, and limits the overall calculation parallelism. Since the overall structure of the prior art patent does not support matrix multiplication and other related operations, it cannot be accelerated in the application scenario of target tracking. SUMMARY

[0005] In view of the defects in the prior art, the purpose of the present application is to provide a target tracking hardware accelerator high parallelism data prefetching implementation method.

[0006] According to the target tracking hardware accelerator high parallelism data prefetching implementation method provided by the present application, the controller, the input cache module, the weight cache module, the matrix processing unit, the output cache module, the write-back module and the external memory are included.

[0007] The controller controls the data flow of the calculation by controlling the input buffer module, the matrix processing unit and the output buffer module; the input buffer module is responsible for completing the prefetching and preprocessing operation of the input data; the weight buffer module completes the prefetching of the weight data; the matrix processing unit receives the output data of the input buffer module and the weight buffer module, performs parallelized operation and outputs the result to the output buffer module, and the output of the output buffer module is written back to the external memory through the write-back module.

[0008] Preferably, the input buffer module includes an up-sampling module, an input buffer, a sliding window module and a pooling module.

[0009] The preprocessing operation includes the pooling operation and the up-sampling processing of the feature map.

[0010] Preferably, the output buffer module integrates the activation and quantization functions, and is provided with a bypass buffer for supporting the convolution acceleration and matrix addition operation with a bypass branch.

[0011] The output buffer module includes an output buffer, a bypass buffer, an activation module, a quantization module and a dequantization module.

[0012] Preferably, the matrix processing unit includes a plurality of configurable vector processing units VPU.

[0013] Each vector processing unit VPU contains 4 basic processing units PE, i.e. vector PE, which supports the parallel multiply-accumulate operation of all elements in the vector within a single cycle.

[0014] Preferably, the top-level control unit supports dynamically shutting down the vector PE inside the vector processing unit VPU.

[0015] Preferably, the vector processing units VPU are configured in a two-dimensional array structure, wherein the vector processing units VPU in the row direction are responsible for calculating the results in the same row of the output matrix during matrix multiplication, the vector processing units VPU in different rows calculate different rows of the output matrix, and the VPU in the column direction is used for calculating the results in the same column.

[0016] Preferably, the sliding window module is a pipelined sliding window including 5 rows of registers and 4 row first-in-first-out buffers FIFO, which can realize the prefetching of input data with multiple parallelism in the form of pipeline.

[0017] A padding enable signal is introduced in the register of the sliding window array, which is used to control the value participating in the next step calculation; when the enable signal is at a low level, the register outputs the internal result, and if the enable signal is at a high level, the register outputs a zero value, so as to control the zero value padding through the dynamic shutdown and enable signal of the register.

[0018] Preferably, the pooling operation is based on column vectors;

[0019] When the pooling operation is performed, the column vector data obtained by the sliding window is sent into the pooling module for processing, including comparing the size or averaging, saving each column pooling result, and performing overall pooling on the pooling results of each column after the pooling results of all columns in the sliding window are calculated, to obtain the final data.

[0020] Preferably, the up-sampling module is used to write the expanded pixels into the input buffer for storage through address offset and sliding window index when up-sampling is needed.

[0021] Preferably, the input cache module adopts a double Bank ping-pong structure to support switching between convolution and matrix operation modes.

[0022] The pooling module is designed in a sliding window, in which a comparator and an addition tree are multiplexed.

[0023] Compared with the prior art, the present application has the following beneficial effects:

[0024] 1. The present application constructs a matrix processing unit (MPU) through a vector processing unit (VPU) to support multi-level parallel computing.

[0025] 2. The flow line sliding window module designed in the present application can obtain data of multiple convolution input windows in a flow line form, which reduces the number of registers by 50% compared with similar designs, and reduces storage overhead through zero value padding.

[0026] 3. The input / weight cache module of the present application adopts a ping-pong structure and a block strategy to adapt to matrix multiplication and convolution operation.

[0027] 4. The present application can adjust the row and column parallelism according to task requirements, and the peak computing efficiency reaches 1024 times of multiplication and accumulation per cycle. The accelerator is suitable for edge computing scenarios, and significantly improves the real-time performance and energy efficiency ratio of the target tracking algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0028] Other features, objects and advantages of the present application will become more apparent through reading the following detailed description of the non-limiting embodiments with reference to the accompanying drawings:

[0029] Figure 1 The figure is a schematic diagram of the structure of the vector PE in the vector processing unit in the present application;

[0030] Figure 2 The figure is a schematic diagram of the structure of the vector PE in the vector processing unit in the present application;

[0031] Figure 3 The figure is a schematic diagram of the structure of the vector PE in the vector processing unit in the present application;

[0032] Figure 4 Fig. 1 is a schematic diagram of a matrix processing unit architecture in the present application;

[0033] Figure 5 Fig. 2 is a schematic diagram of a multi-size sliding window mechanism in the present application;

[0034] Figure 6 Fig. 3 is a schematic diagram of a sliding window in a 3x3 window data fetching process with a step size of 1 in the present application;

[0035] Figure 7 Fig. 4 is a schematic diagram of a 3x3 window data fetching process with a step size of 2 in the present application;

[0036] Figure 8 Fig. 5 is a schematic diagram of a dynamic up-sampling module design structure in the present application;

[0037] Figure 9 Fig. 6 is a schematic diagram of a sliding window operation with a size of 3 and a step size of 1 for the first four times for a feature map with a size of 8x8 in the present application. DETAILED DESCRIPTION

[0038] The present application will be described in detail below with specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of changes and improvements can be made. These all belong to the protection scope of the present application.

[0039] The configurable target tracking algorithm hardware accelerator of the present application faces the edge computing scene, supports multiple convolution, pooling, matrix operation and dynamic up-sampling operation, and significantly improves the computing efficiency through the optimized parallel computing architecture and the high parallelism input data prefetching strategy.

[0040] According to the high parallelism data prefetching implementation method of the target tracking hardware accelerator provided by the present application, the method comprises the steps of Figure 1As shown, it includes a controller, an input buffer module, a weight buffer module, a matrix processing unit (MPU), an output buffer module, a write-back module, and an external memory. The controller controls the data flow of the calculation by controlling the input buffer module, the matrix processing unit, and the output buffer module. The input buffer module (regarded as a left multiplication matrix in matrix operation) is responsible for completing the prefetching and preprocessing operations of the input data, such as the pooling and up-sampling processing of feature maps. The weight buffer module (regarded as a left multiplication matrix in matrix operation) completes the prefetching of the weight data. The matrix processing unit receives the output data of the input buffer module and the weight buffer module, performs efficient parallelized operation, and finally outputs the result to the output buffer module. The output buffer module integrates the activation and quantization functions, and a bypass buffer is designed to support convolution acceleration with a bypass branch and matrix addition operation. The output of the output buffer module is written back to the external memory through the write-back module.

[0041] The matrix processing unit (MPU) includes a plurality of configurable vector processing units VPU. For example, it is composed of a 4-row x 16-column configurable vector processing unit VPU array. Each vector processing unit VPU contains 4 basic processing units PE, i.e., vector PE, which supports the parallel multiply-accumulate operation of the elements in the vector within a single period. The matrix processing unit MPU is constructed by a two-dimensional VPU array, supporting multi-level parallel computation in the row and column directions. As shown in Figure 2 As shown, the basic processing element (PE) in the array is called vector PE, which can perform parallel multiply-accumulate operation on all elements in the input vector. In contrast, the scalar PE takes elements as the operation unit and can only complete the multiply-accumulate operation of one element per clock cycle; while the vector PE can perform the multiply-accumulate calculation of all elements in the vector in parallel within one clock cycle. This parallelized design significantly reduces the overhead caused by frequent memory access and improves the calculation efficiency. Figure 2 The difference in operation capability between the scalar PE and the vector PE is intuitively shown in Figure 3 As shown in the parallel vector processing unit VPU structure diagram, each VPU contains 4 vector PEs, and the number of vector PEs in a single VPU is defined as P_vc in the present application. The VPU accumulates all PE results through an addition tree to generate the final result in structure, which significantly improves the overall parallelism of the calculation, and the vector calculation parallelism of a single VPU is improved to the highest P_simd x P_vc, where P_simd represents the number of multipliers contained in the vector PE. In addition, the top-level control unit supports dynamically shutting down the PEs inside the VPU. In the scenario where the calculation parallelism demand is low, this function can effectively reduce the power consumption.

[0042] AsFigure 4 As shown in the figure, the VPU is configured in a two-dimensional array structure, wherein the VPU in the row direction is responsible for calculating the results in the same row in the output matrix during matrix multiplication, the VPU in different rows calculates different rows of the output matrix, and the VPU in the column direction is used for calculating the results in the same column, so as to fully exert the parallel characteristics of matrix calculation. Figure 4 As shown in the figure.

[0043] The input cache module includes an up-sampling module, an input buffer, a sliding window module and a pooling module. The input cache module adopts a double Bank ping-pong structure and supports switching between convolution and matrix operation modes. The sliding window module is a flow type sliding window and is composed of 5 rows of registers and 4 row first-in-first-out buffers (FIFOs), wherein a D flip-flop (DFF) is a common register, and a Reg is a flip-flop with zero clearing, which is used to realize the function of zero value filling. The sliding window module can realize prefetching of input data with multiple parallel degrees in the form of a pipeline. Figure 5 As shown in the figure, a pipeline structure based on column vectors is adopted to support various step sizes of 3*3 convolution, 5*5 convolution and 2*2 average pooling and various sliding window operations. Zero value filling is controlled through a register dynamic shutdown and an enable signal to reduce storage overhead.

[0044] The matrix processing unit MPU calculation array is a matrix processing array with vectors as basic units, and the convolution layer operation needs to complete the conversion of input feature maps to matrices. In addition, in order to avoid calculation array pause, the idle waiting time of the calculation array should be reduced as much as possible for different step size operators to improve the data throughput rate. Therefore, the accelerator of the application designs a data prefetching mechanism supporting multiple sliding windows in the input cache module, so that the calculation array can process data of multiple sliding windows at the same time. The parallel sliding window structure with multiple sizes and supporting pooling and filling can improve the data reuse rate of the input feature map, and data can be extracted in units of vectors to reduce memory overhead and adapt to the vector processing unit.

[0045] The sliding window of the application discards the traditional design of simultaneously acquiring multiple sliding window data, and instead operates in a pipeline form in units of vectors. In the structure of the sliding window, the data flow between multiple sliding windows presents the characteristics of pipeline transmission (observed in units of column vectors). Therefore, while completing the sliding window data acquisition, the time data reuse rate is further improved, and the number of required registers in the sliding window is effectively reduced.

[0046] As Figure 6As shown in the figure, the sliding window takes data in a 4-step 3x3 sliding window, wherein each data is represented by the serial number of a column vector. As can be seen from the figure, at the 3rd cycle, the sliding window data jump is 2, and the first sliding window needs to jump by 4 steps, that is, the sliding window takes data from the 1st, 2nd and 3rd columns, and then takes data from the 5th, 6th and 7th columns. The results show that the structure can compress the number of register columns from 4 in the traditional design to 2, and the maximum data jump range is only 2. The time required for data jump in this design can be hidden in the data reading process of the current vector, thereby avoiding additional time overhead.

[0047] For the case of 2-step, the structure can complete the sliding window operation through 5 columns of registers. As shown in the figure, in the data taking process of a 4-step 3x3 sliding window, the maximum data jump is 4. This means that after 3 cycles required for reading the column vector, only one cycle of pause is needed to complete the data prefetch operation, which effectively reduces the 3-cycle pause time in the above vector parallel structure to 1 cycle, thereby optimizing the data transmission efficiency. Figure 7

[0048] Based on the characteristics of vector pipeline output, the number of registers required for sliding window operation of different sizes and steps can be obtained, as shown in Table 1

[0049] Table 1 Number of register columns required under different sliding windows

[0050]

[0051] Therefore, only 5 columns of registers are needed to support sliding window operation of all sizes. Considering that the maximum size of the sliding window is 5x5, the array designed by the application is composed of 5 rows of registers and 4 row buffers, as shown in the figure. Figure 9

[0052] The application introduces a fill enable signal in the register of the sliding window array, which is used to control the value participating in the next calculation. When the enable signal is at a low level, the register outputs the internal result, and if the enable signal is at a high level, the zero value is output. In this way, the fill control of the final result can be realized through the state of the enable signal without introducing additional clock cycles.

[0053] ​​The sliding window accesses data based on column vectors, that is, the same column of pixel values is accessed at the same time, and therefore the pooling operation in the sliding window is based on column vectors. When the pooling operation is performed, the column vector data obtained by the sliding window is fed into the pooling module for processing, including comparing sizes (max pooling) or averaging (average pooling). The pooling results of each column are saved, and after the pooling results of all columns in the sliding window are calculated, overall pooling is performed on the pooling results of each column to obtain the final data. The pooling module is designed in a pipelined sliding window, in which a multiplexed comparator and an addition tree are supported to support max pooling and average pooling.

[0054] To explain the working principle of the pipelined sliding window in this paper, an 8x8 feature map is taken as an example, and the first four sliding window operations with a size of 3 and a step of 1 are shown in FIG. 3. Figure 9 The filling state matrix controls the registers that need to be closed and samples the results of the sliding window, and stores the results in the sampling register. The fourth column sliding window performs a step of 2, which means that the sliding window samples after inputting two pixel points. By controlling the input signal of the sliding window and the sampling signal of the register, the column sliding window can work correctly and adapt to the calculation requirements of different steps. In the convolution operation, the results of the sampling register are output to the subsequent module one by one in three cycles; in the pooling operation, the sampling results directly enter the pooling module to calculate the pooling values of two column sliding windows. At the same time when the sampling register outputs data, the sliding window continues to input the data of the next pixel point to drive the column sliding window to move forward by one pixel point. When all the sampling values are output or the pooling calculation is completed, the sliding window re-samples the column sliding window.

[0055] The upsample module realizes 2x pixel expansion through address offset and sliding window index. Specifically, data is taken from external storage and input to the Input SRAM. The dynamic upsampling method of nearest neighbor interpolation expands the input image size to 2 times the original size, that is, a single pixel point is expanded to a 2x2 pixel block. In the accelerator, the nearest neighbor interpolation module is used to realize this function in cooperation with a 3x3 sliding window and an index, as shown in FIG. 4. Figure 8

[0056] ​When the layer does not need to be up-sampled, the result is directly written into the input buffer Input SRAM for storage. If it is an up-sampling layer, pixel expansion in the row direction is achieved by adding 1 to the original address, and pixel expansion in the column direction is achieved by adding the original feature map size to the original address. For dynamic up-sampling, the module will obtain the pixel values around the target pixel point according to the 3*3 sliding window, and use the index value to locate the data source to determine the position of the specific register from which the write data comes. In this way, the up-sampling result can be directly stored in the SRAM for the next calculation, reducing the overhead of re-reading the off-chip storage. At the same time, due to the large amount of repeated data in the expanded feature map, the waste of storage space caused by storage off-chip is avoided.

[0057] The application supports selective shutdown of VPU, and can effectively reduce power consumption in a low parallel scenario. In matrix operation, different data flow mapping methods can be adopted according to different sizes of the matrix size to improve the efficiency of matrix operation, for example, additional operation form setting is performed on the small size matrix feature in target tracking. The application balances the calculation efficiency and power consumption by configuring the row and column parallel degree (P row ,P col ) and vector internal parallel degree (P simd ) of the MPU. Wherein, P row represents the row parallel degree, such as the number of VPU array rows in the MPU shown in Figure 4 ; P col represents the column parallel degree, such as the number of VPU array columns in the MPU shown in Figure 4 ; and P simd represents the vector internal parallel degree, the number of multipliers contained in the VPU internal vector PE.

[0058] The application aims to solve the technical problems of efficiently calculating a plurality of operators (convolution, pooling, matrix multiplication) uniformly, reducing memory bandwidth pressure through data reuse, and dynamically adjusting parallel degree to adapt to different computing scenarios.

[0059] A two-dimensional matrix processing unit (MPU) composed of a vector processing unit (VPU) supports parallel computation in the row direction and the column direction; a configurable sliding window module based on a column vector pipeline structure adapts 2x2, 3x3, 5x5 convolution data prefetching of multiple parallel degrees and corresponding maximum or average pooling operations; an input / weight double Bank cache module block-stores data to support matrix multiplication and convolution operation mode switching; software can dynamically control parallel degrees to optimize computing efficiency and power consumption by adjusting row parallel degrees, column parallel degrees and vector internal parallel degrees. The sliding window module can simultaneously obtain the required data of input sliding windows of multiple parallel degrees in the form of a pipeline, and can output register values as internal data or zero values by filling a state matrix, and supports pooling operations. An output cache module integrates a bypass cache and a quantization unit to support residual connection and multi-channel accumulation, and the output cache module includes an output buffer, a bypass buffer, an activation module, a quantization module and a dequantization module. A dynamic upsampling module realizes 2-pixel expansion through address offset and sliding window index.

[0060] Those skilled in the art know that, in addition to implementing the system provided by the present application and each device, module and unit thereof in a pure computer readable program code manner, the system provided by the present application and each device, module and unit thereof can also be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. by logically programming the method steps to achieve the same functions. Therefore, the system provided by the present application and each device, module and unit thereof can be considered as a hardware component, and the devices, modules and units included therein for achieving various functions can also be considered as structures within the hardware component; the devices, modules and units for achieving various functions can also be considered as both software modules implementing methods and structures within the hardware component.

[0061] The specific embodiments of the present application are described above. It needs to be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the present application. The embodiments of the present application and the features in the embodiments can be combined with each other in any manner without conflict.

Claims

1. A method for high parallelism data prefetching implementation of a target tracking hardware accelerator, the method comprising: The application relates to a controller, an input cache module, a weight cache module, a matrix processing unit, an output cache module, a write-back module and an external memory. The controller controls the data flow of calculation by controlling the input cache module, the matrix processing unit and the output cache module; the input cache module is responsible for the pre-fetching and preprocessing operation of input data; the weight cache module is responsible for the pre-fetching of weight data; the matrix processing unit receives the output data of the input cache module and the weight cache module, performs parallel operation and outputs the result to the output cache module, and the output of the output cache module is written back to the external memory through the write-back module. The input cache module comprises an up-sampling module, an input buffer, a sliding window module and a pooling module.

2. The method of claim 1, wherein, The preprocessing operation comprises a pooling operation and up-sampling processing of a feature map. The output cache module integrates activation and quantization functions, and is provided with a bypass buffer for supporting convolution acceleration and matrix addition operation with a bypass branch.

3. The method of claim 1, wherein, The output cache module comprises an output buffer, a bypass buffer, an activation module, a quantization module and a dequantization module. The matrix processing unit comprises a plurality of configurable vector processing units VPU.

4. The method of claim 1, wherein, Each vector processing unit VPU comprises four basic processing units PE, namely vector PEs, which support parallel multiplication and accumulation operation of all elements in a vector within a single cycle. A top-level control unit supports dynamically turning off the vector PEs in the vector processing unit VPU.

5. The method of claim 4, wherein, The vector processing units VPU are arranged in a two-dimensional array structure, wherein the vector processing units VPU in the row direction are responsible for calculating the results in the same row of the output matrix, the vector processing units VPU in different rows are responsible for calculating different rows of the output matrix, and the VPU in the column direction is responsible for calculating the results in the same column.

6. The method of claim 4, wherein, The sliding window module is a pipelined sliding window, which comprises five rows of registers and four row first-in-first-out buffers FIFO, and can realize the pre-fetching of input data with multiple parallel degrees in a pipeline mode.

7. The method of claim 2, wherein the target tracking hardware accelerator high parallelism data prefetch implementation is characterized by, A filling enable signal is introduced in the register of the sliding window array, which is used for controlling the values participating in the next calculation; when the enable signal is at a low level, the register outputs the internal result, and when the enable signal is at a high level, the register outputs zero value, so that the zero value filling is controlled by the dynamic turning off and enabling of the register. The pooling operation is based on column vectors.

8. The method of claim 2, wherein the target tracking hardware accelerator high parallelism data prefetch implementation is characterized by, When the pooling operation is performed, the column vector data obtained through the sliding window is sent into the pooling module for processing, including comparing sizes or averaging, saving the pooling results of each column, and performing overall pooling on the pooling results of each column after the pooling results of all columns in the sliding window are calculated, to obtain the final data. The up-sampling module is used for writing and storing the pixel expansion after the pixel expansion into the input buffer through address offset and sliding window index when up-sampling is needed.

9. The method of claim 2, wherein the target tracking hardware accelerator high parallelism data prefetch implementation is characterized by, The input cache module adopts a double-Bank ping-pong structure and supports switching between convolution and matrix operation modes.

10. The method of claim 2, wherein the target tracking hardware accelerator high parallelism data prefetch implementation is characterized by, The pooling module is designed in the sliding window, and a comparator and an addition tree are multiplexed. ​

Images