A hardware acceleration device and method

Abstract

The application discloses a hardware acceleration device and method, and relates to the field of neural network hardware acceleration. The hardware acceleration device comprises at least one processing unit, the processing unit comprises a register stack, a multiplexing array and a plurality of multiply-add units; the register stack caches input feature data required for generating an output block to be calculated; the multiplexing array selects an input activation sub-window corresponding to a current non-zero weight from the input feature data according to a position index of the current non-zero weight, and outputs a plurality of input activation values in the input activation sub-window to the plurality of multiply-add units respectively; and the plurality of multiply-add units respectively perform multiply-add operations according to the current non-zero weight and the corresponding input activation values. According to the position index of the current non-zero weight, the application determines the input activation values corresponding to the plurality of multiply-add units at one time, does not need to independently determine input data for each multiply-add unit, and can reduce the hardware logic resources required for input data selection during unstructured sparse convolution parallel calculation.

Description

Technical Field

[0001] This invention relates to the field of neural network hardware acceleration, and in particular to a hardware acceleration device and method. Background Technology

[0002] Convolutional neural networks (CNNs) are widely used in tasks such as image classification, object detection, and semantic segmentation. As the network size increases, the number of weights and multiplication-accumulation operations involved in convolution computation becomes significant. To reduce the computational cost of neural networks, pruning can be used to reduce some weights to zero, resulting in a sparse weight matrix. In unstructured sparse weights, the zero-value weights and non-zero weights are typically irregularly distributed.

[0003] When performing hardware-accelerated computation on unstructured sparse weights, the corresponding input feature data needs to be determined based on the position of the non-zero weights in the convolution kernel. When multiple multiply-accumulate units perform convolution computation in parallel, the input feature data corresponding to each multiply-accumulate unit needs to be determined separately, and the process of selecting input data consumes a significant amount of hardware logic resources. Summary of the Invention

[0004] The purpose of this invention is to provide a hardware acceleration device and method to solve the problem that multiple multiply-accumulate units determine the corresponding input feature data in parallel computation of unstructured sparse convolution, which causes the selection of input data to occupy a lot of hardware logic resources.

[0005] In a first aspect, this application provides a hardware acceleration device, including at least one processing unit. The processing unit includes a register file, a multiplexer array, and multiple multiply-accumulate units. The register file is used to cache the input feature data required to generate the output patch to be computed. The multiplexer array is connected between the register file and the multiple multiply-accumulate units, and is used to select an input activation sub-window corresponding to the current non-zero weight from the input feature data cached in the register file according to the position index corresponding to the current non-zero weight. The position index is used to represent the position of the current non-zero weight in the convolution kernel. The input activation sub-window includes multiple input activation values ​​corresponding to multiple output positions in the output patch to be computed, and the multiple input activation values ​​are output to the multiple multiply-accumulate units respectively. The multiple multiply-accumulate units are used to perform multiply-accumulate operations according to the current non-zero weight and the corresponding input activation value respectively.

[0006] Secondly, this application provides a hardware acceleration method applied to the aforementioned processing unit. The hardware acceleration method includes: caching the input feature data required to generate the output patch to be calculated into a register file; obtaining the current non-zero weight and the position index corresponding to the current non-zero weight, the position index being used to characterize the position of the current non-zero weight in the convolution kernel; selecting an input activation sub-window corresponding to the current non-zero weight from the input feature data cached in the register file according to the position index via a multiplexing array, the input activation sub-window including multiple input activation values ​​corresponding to multiple output positions in the output patch to be calculated; outputting the multiple input activation values ​​to multiple multiply-accumulate units respectively, and performing multiply-accumulate operations according to the current non-zero weight and the corresponding input activation value through the multiple multiply-accumulate units respectively.

[0007] This invention provides a hardware acceleration device and method, relating to the field of neural network hardware acceleration. The hardware acceleration device includes at least one processing unit, which comprises a register file, a multiplexer array, and multiple multiply-accumulate units. The register file caches the input feature data required to generate the output patch to be computed. The multiplexer array selects an input activation sub-window corresponding to the current non-zero weight from the input feature data based on the position index of the current non-zero weight, and outputs multiple input activation values ​​from the input activation sub-window to the multiple multiply-accumulate units respectively. The multiple multiply-accumulate units perform multiply-accumulate operations based on the current non-zero weight and the corresponding input activation value. This invention determines the input activation value corresponding to multiple multiply-accumulate units at once based on the position index of the current non-zero weight, eliminating the need to independently determine the input data for each multiply-accumulate unit, thus reducing the hardware logic resources required for input data selection during parallel computation of unstructured sparse convolution.

[0008] This application utilizes the position index of the current non-zero weight to control a multiplexing array, selecting input activation values ​​corresponding to multiple output positions in the output tile to be calculated at once, and providing these input activation values ​​to multiple multiply-accumulate units respectively. For the same current non-zero weight, multiple multiply-accumulate units do not need to set up independent position resolution processes, which can reduce the hardware logic resources required for input data selection. Attached Figure Description

[0009] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments are briefly described below. The drawings described below are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings.

[0010] Figure 1 This is a schematic diagram of a processing unit structure provided by the present invention; Figure 2 A schematic diagram of the overall structure of a hardware acceleration device provided by the present invention; Figure 3This is a schematic diagram of a row selection network structure provided by the present invention.

[0011] Figure 4 A flowchart of a hardware acceleration method provided by the present invention. Detailed Implementation

[0012] The core of this invention is to provide a hardware acceleration device and method for reducing the hardware logic resources used in input data selection during unstructured sparse convolution parallel computation.

[0013] To make the technical solutions of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described below with reference to the accompanying drawings. The described embodiments are some embodiments of the present invention. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are also within the scope of protection of the present invention.

[0014] In a first aspect, this application provides a hardware acceleration device, including at least one processing unit. The processing unit includes a register file 100, a multiplexer array 110, and multiple multiply-accumulate units 120. The register file 100 is used to cache the input feature data required to generate the output patch to be calculated. The multiplexer array 110 is connected between the register file and the multiple multiply-accumulate units 120, and is used to select an input activation sub-window corresponding to the current non-zero weight from the input feature data cached in the register file 100 according to the position index corresponding to the current non-zero weight. The position index is used to represent the position of the current non-zero weight in the convolution kernel. The input activation sub-window includes multiple input activation values ​​corresponding to multiple output positions in the output patch to be calculated, and the multiple input activation values ​​are output to the multiple multiply-accumulate units 120 respectively. The multiple multiply-accumulate units 120 are used to perform multiply-accumulate operations according to the current non-zero weight and the corresponding input activation value respectively.

[0015] In this embodiment, the hardware acceleration device is used to execute the convolution operation in the sparsely processed convolutional neural network. Sparse processing can be completed before hardware inference; its function is to reduce some weights in the original weight matrix to zero, resulting in a sparse weight matrix that contains both zero and non-zero weights. The specific algorithm for sparse processing can employ methods that yield an unstructured sparse weight matrix. During the hardware execution phase, calculations are performed on the remaining non-zero weights. A currently non-zero weight represents a non-zero weight that the processing unit is using in the current computation cycle or phase.

[0016] like Figure 1As shown, the register file 100 in the processing unit is used to temporarily store a block of input feature data required for the current computation within the processing unit. The output patch to be computed includes multiple output locations, such as multiple adjacent output pixel locations. The input feature data cached in the register file 100 covers the input regions that these output locations need to access under the current convolution kernel. The register file 100 can be a register array or an on-chip storage structure capable of providing multiple input data to the multiplexer array 110 within one or more clock cycles.

[0017] In this embodiment, the multiplexing array 110 is disposed between the register file 100 and multiple multiply-accumulate units 120. The position index of the non-zero weight reflects the spatial position of the non-zero weight in the convolution kernel. For a two-dimensional convolution kernel, the position index can be determined by both the row position and the column position; alternatively, different convolution kernel positions can be numbered sequentially, and the number can be used as the position index. After obtaining the current non-zero weight, the processing unit simultaneously obtains the position index corresponding to the non-zero weight and uses the position index for input selection of the multiplexing array 110.

[0018] The multiplexing array 110 does not select an input activation value for a single multiply-accumulate unit 120. Instead, it selects a set of input activation values ​​from the input feature data already cached in the register file 100 based on the position index of the current non-zero weight. This set of input activation values ​​forms an input activation sub-window, with each input activation value in the sub-window corresponding to a different output position in the output tile to be calculated. Therefore, when the output tile to be calculated includes M output positions, the multiplexing array 110 can provide M input activation values ​​for the corresponding M multiply-accumulate units 120, with each input activation value corresponding to the input data covered by the current non-zero weight at different output positions. Figure 1 In one optional embodiment, the position index Index[3:0] includes Index[1:0] and Index[3:2], where Index[1:0] controls the selection of input data in the row direction, and Index[3:2] controls the selection of input data in the column direction. The row offset and column offset corresponding to the current non-zero weight are determined using Index[1:0] and Index[3:2], respectively, to obtain the corresponding input activation sub-window from the input feature data cached in the register file.

[0019] After receiving multiple input activation values ​​from the input activation sub-window, multiple multiply-accumulate units 120 perform multiplication calculations using the current non-zero weights, and accumulate the multiplication results to the partial sums corresponding to each output position. That is, the same current non-zero weight can be used by multiple multiply-accumulate units 120 in the same calculation phase, but each multiply-accumulate unit 120 receives different input activation values. After completing the multiplication and accumulation of the current non-zero weight, the process can switch to the next non-zero weight, and the input activation sub-window is redefined based on the position index of the next non-zero weight, continuing to accumulate the same set of output positions.

[0020] In some embodiments, the multiple multiply-accumulate units 120 can be arranged in a matrix, with the number of rows and columns of each multiply-accumulate unit 120 corresponding to the number of rows and columns of the output patch to be computed, respectively. In other embodiments, the multiple multiply-accumulate units 120 can also be arranged in a one-dimensional array or other form capable of mapping multiple output positions, as long as the multiplexing array 110 can provide the input activation values ​​corresponding to different output positions to the respective multiply-accumulate units 120. The size of the output patch to be computed can be set according to the convolutional layer size, the number of on-chip resources, and the scale of parallel computing.

[0021] In some embodiments, the connection paths in the multiplexer array 110 can be pre-fixed, and the position index is used to select the corresponding branch in the fixed connection path. Thus, when the current non-zero weight changes, it is not necessary to re-change the physical connections of the multiplexer array 110; only the selection state corresponding to the position index needs to be changed. For the same position index, the input activation value corresponding to multiple output positions can be determined by that position index, which can reduce the logic resources used for position resolution and input selection for multiple multiply-accumulate units 120 respectively.

[0022] When a processing unit executes a calculation of an output patch, it can repeatedly perform the processes of weight acquisition, position index reading, input activation sub-window selection, and multiply-accumulate in a predetermined order of non-zero weights. The same block of input feature data in register file 100 remains accessible across multiple non-zero weights. When the current non-zero weight changes, the multiplexing array 110 switches the output path based on the new position index, and the multiply-accumulate unit 120 continues to accumulate the newly generated multiplication result into the partial sum corresponding to the original output position. After all non-zero weights required for the current output patch have been processed, the process switches to the next output patch or the next convolution task. This execution process ensures that the local caching of input feature data, weight position selection, and parallel computation of multiple output positions are consistent throughout.

[0023] In one exemplary embodiment, the multiplexing array 110 includes a row selection network and a column selection network; the row selection network is used to select consecutive rows of input feature data corresponding to the number of rows of the output patch to be calculated from the input feature data cached in the register file 100 according to the row position of the current non-zero weights in the convolution kernel; the column selection network is used to select consecutive columns of input feature data corresponding to the number of columns of the output patch to be calculated from the input feature data output by the row selection network according to the column position of the current non-zero weights in the convolution kernel, so as to obtain an input activation sub-window.

[0024] In this embodiment, the position index can be divided into row selection information representing the row position of the convolution kernel and column selection information representing the column position of the convolution kernel. The multiplexing array 110 first narrows down the range of candidate input data in one direction, and then continues to select in another direction, finally obtaining the input activation sub-window corresponding to the size of the output patch to be calculated. The row selection network and the column selection network can be composed of multiple two-to-one multiplexers in a multi-level connection manner, and the specific number of levels and quantities are set according to the size of the candidate input data and the number of positions where the convolution kernel can appear.

[0025] With a convolution stride of 1, if the size of the output patch to be computed is Th×Tw and the size of the convolution kernel is Rh×Rw, then the input region required to cover the output patch can be (Th+Rh-1)×(Tw+Rw-1). The register file 100 can store this input region. The multiplexing array 110 selects a Th×Tw input activation sub-window from this input region based on the position of the current non-zero weights in the Rh×Rw convolution kernel. The size relationship here is used to illustrate the correspondence between the input region and the output patch; the actual size of the processing unit can be set according to the specific convolutional layer parameters.

[0026] Taking a 3×3 convolution kernel, a stride of 1, and an 8×8 output patch to be computed as an example, register file 100 can cache 10×10 rows of input feature data. When the current non-zero weight is in the first row of the 3×3 convolution kernel, the row selection network selects rows 1 to 8 of the input region. When the current non-zero weight is in the second row, rows 2 to 9 are selected. When the current non-zero weight is in the third row, rows 3 to 10 are selected. After completing the row selection, the 10×10 input region corresponds to an 8×10 candidate data region.

[0027] In the example above, the column selection network further selects columns based on the current non-zero weight's column position. If the current non-zero weight is in the first column of the convolution kernel, columns 1 to 8 of the candidate data region are selected. If it's in the second column, columns 2 to 9 are selected; and if it's in the third column, columns 3 to 10 are selected. After column selection, an 8×8 input activation sub-window is obtained, with 64 input activation values ​​in the window corresponding to the 64 output positions in the 8×8 output patch to be calculated.

[0028] Figure 3 A partial implementation of a row selection network is shown. Multiple 2-to-1 multiplexers are connected in a multi-stage manner. The selection signal in the position index, which represents the row position, controls the corresponding multiplexer, so that the output terminal obtains input data at consecutive positions. Figure 3 The selection signals in the diagram are only used to indicate the control relationship between the position index and the multi-level multiplexer, and it is not required that all embodiments use the same number of multiplexers or the same bit width of selection signals. For different convolution kernel sizes, selection states that can cover each row offset and column offset can be set accordingly.

[0029] In an optional embodiment, the selection control signal of the multiplexer array 110 can be determined based on the position index corresponding to the current non-zero weight. The position index is used to represent the row and column positions of the current non-zero weight in the convolution kernel. The control signal sel can include multiple control bits for controlling row selection and column selection, respectively. Taking a 3×3 convolution kernel as an example, the control signal sel can use a 4-bit control signal sel[3:0]. Different control signal values ​​correspond to different positions of the current non-zero weight in the 3×3 convolution kernel. Among them, a part of the control bits is used to control the row selection network to determine the consecutive multiple rows of data selected from the input feature data, and another part of the control bits is used to control the column selection network to determine the consecutive multiple columns of data selected from the row-selected data. The multiple multiplexers in the row selection network and the column selection network are jointly controlled by the corresponding control bits, so that the position index corresponding to the current non-zero weight can determine the selection result of multiple input activation values ​​in the input activation sub-window at one time, and provide the multiple input activation values ​​to the multiple multiply-accumulate units 120 respectively. Specifically, the position index is decoded to generate the control signal sel, in Figure 3 In the two-level selection structure shown, sel[3] is used to control the first-level two-to-one multiplexer, and sel[2] is used to control the second-level two-to-one multiplexer so that the corresponding input data is transmitted along the selected path. Out0 and Out1 are the output terminals of this local row selection network, used to output the input data after two-level selection, and as part of the input data of the subsequent column selection network.

[0030] In other embodiments, column direction selection can be performed first, followed by row direction selection, as long as the corresponding input activation sub-window can ultimately be obtained from the candidate input region based on the current non-zero weight position index. For hardware implementation, the order of the two selection stages can be determined based on wiring length, timing requirements, or the data arrangement of register file 100.

[0031] The bit width of the position index can be determined based on the number of selectable positions within the convolutional kernel. For example, a 3×3 convolutional kernel with 9 spatial positions can employ an encoding capable of distinguishing these 9 positions, from which row and column position selection information can be obtained. The control terminals of the row and column selection networks can share different fields within the same position index, or the corresponding row and column selection signals can be obtained first through simple decoding logic. Regardless of the encoding method used, the position index corresponding to the same current non-zero weight is used to determine the entire input activation sub-window, rather than configuring independent weight position encodings for each multiply-accumulate unit 120.

[0032] In one optional embodiment, the encoding information corresponding to the current non-zero weight includes a position index and an input channel increment; the input channel increment is used to characterize the address difference between the current non-zero weight and the previous non-zero weight in the input channel dimension, and is used to update the input channel access address based on the address difference.

[0033] In this embodiment, the input channel is the original data dimension of the input feature map of the current convolutional layer. Each input channel has a corresponding channel identifier and input feature data. Whether a certain input channel is processed by a certain processing unit depends on the allocation result of the current computing task and does not change the correspondence between the weights and the original input channels. The position index is used to represent the position of the current non-zero weights in the convolutional kernel space within an input channel, and the input channel increment is used to represent the address change of two adjacent non-zero weights to be processed in the input channel dimension. These two types of encoded information are used for data localization in two different dimensions.

[0034] During weight encoding, non-zero weights can be organized according to the hardware execution order, and a position index and input channel increment can be recorded for each non-zero weight. The position index can directly record the row and column coordinates, or it can record the position number obtained by converting the row and column coordinates. The input channel increment can record the difference between the input channel address corresponding to the current non-zero weight and the input channel address corresponding to the previous non-zero weight. After the processing unit or the corresponding address control logic reads this difference, it adds the input channel increment to the previous input channel access address to obtain the current input channel access address.

[0035] For example, if the previous non-zero weight to be processed corresponds to the second input channel, and the current non-zero weight to be processed corresponds to the fifth input channel, then the input channel increment can represent the address change from the second input channel to the fifth input channel. When the hardware updates the access address based on this input channel increment, it can directly locate the fifth input channel without sequentially performing accesses on intermediate input channels that are unrelated to the current non-zero weight. When two adjacent non-zero weights to be processed still belong to adjacent input channels or the same input channel, the input channel increment should accordingly use a value that can represent this address relationship.

[0036] The position index and input channel increment can be combined with the non-zero weights to form a single encoded record, or they can be organized separately into data sequences with corresponding orders. The former facilitates reading the non-zero weights and their control information at once, while the latter allows for setting independent data reading paths for different control information. Regardless of the storage format, the non-zero weights, position index, and input channel increment maintain a correspondence, allowing the position index to determine the spatial location of the convolution kernel and the input channel increment to update the input channel access address.

[0037] Through the above encoding, the processing unit can determine the input activation sub-window using the position index in the kernel space dimension, and change the access address using the input channel increment in the input channel dimension. For unstructured sparse weights, the distribution of zero-value weights in both the kernel space and the input channel dimension may be irregular. This embodiment enables the hardware to sequentially obtain the non-zero weights to be processed and their corresponding input data based on the encoded information.

[0038] In one exemplary embodiment, there are multiple processing units, and the hardware acceleration device further includes a controller; the controller is used to count the number of non-zero weights corresponding to the multiple input channels of the current convolutional layer, sort the multiple input channels according to the number of non-zero weights, divide the sorted multiple input channels into multiple input channel groups, and assign the multiple input channel groups to multiple processing units for calculation.

[0039] In this embodiment, multiple processing units are used to execute the convolution task in the current convolutional layer in parallel. Since the number of non-zero weights corresponding to different input channels may differ after unstructured sparsity processing, if tasks are directly allocated according to the original input channel order, the effective number of multiply-accumulate operations required by different processing units may vary significantly. Therefore, before starting the parallel computation of the current convolutional layer, the controller first obtains the number of non-zero weights corresponding to each input channel and uses the number of non-zero weights as a representation of the workload of the input channel.

[0040] The controller's statistical objects are the original multiple input channels of the current convolutional layer. Each input channel retains its original channel identifier; the sorting only changes the order in which input channels are used for task allocation, without altering the corresponding input feature data and weight data. When a processing unit obtains a sorted input channel, it still accesses the corresponding input feature data according to the original channel identifier of that input channel. Therefore, the sorting will not cause incorrect data correspondence between different input channels.

[0041] In some embodiments, the controller can arrange multiple input channels in descending order of the number of non-zero weights, placing input channels with more non-zero weights at the beginning of the sorting sequence and input channels with fewer non-zero weights at the end of the sorting sequence. When two or more input channels have the same number of non-zero weights, the relative positions of these input channels can be determined according to the original input channel identifier or a pre-set fixed order, so as to obtain a determined task allocation result each time the same convolutional layer is executed.

[0042] After obtaining the sorting results, the controller divides the multiple input channels into multiple input channel groups suitable for parallel computing, and assigns different input channel groups to multiple processing units. An input channel group can contain one or more input channels, and the processing units perform calculations sequentially according to the non-zero weights corresponding to the input channels within the group. After each processing unit completes the calculation of its respective input channel group, it can continue to accumulate the partial sums corresponding to different input channels to obtain the convolution result at the corresponding output position.

[0043] In other embodiments, the number of non-zero weights can be counted and grouped before each convolutional layer begins computation. Alternatively, the number of non-zero weights in each convolutional layer can be counted in advance and the corresponding sorting results saved during model deployment. During inference, the controller directly reads the saved sorting information. For convolutional layers whose weights do not change during inference, pre-counting can reduce control operations during inference. For scenarios where weights may be updated, the count can be recalculated after the weights are updated.

[0044] In one optional embodiment, the sorted input channels are divided into multiple input channel groups, including: sequentially selecting a group of input channels from the sorted input channels with the same number as the multiple input channel groups; sequentially allocating the group of input channels to the multiple input channel groups along a first allocation direction; sequentially allocating the next group of input channels to the multiple input channel groups along a second allocation direction opposite to the first allocation direction; and alternately allocating subsequent groups of input channels according to the first allocation direction and the second allocation direction.

[0045] In this embodiment, the first allocation direction and the second allocation direction correspond to the opposite traversal order among multiple input channel groups. Assuming there are N input channel groups, the order from the 1st input channel group to the Nth input channel group can be defined as the first allocation direction, and the order from the Nth input channel group to the 1st input channel group can be defined as the second allocation direction. Each time, N input channels are sequentially taken from the sorted sequence and placed into N input channel groups according to the current allocation direction. Then, the allocation direction is switched to continue processing the next group of N input channels.

[0046] Taking N=4 as an example, with multiple input channels already sorted from most to least number according to the number of non-zero weights, input channels 1 to 4 can sequentially enter group 1, group 2, group 3, and group 4 along the first allocation direction; input channels 5 to 8 can sequentially enter group 4, group 3, group 2, and group 1 along the second allocation direction; input channels 9 to 12 are allocated again along the first allocation direction. Following this method, input channels at positions 1, 8, 9, 16, etc., enter group 1; input channels at positions 2, 7, 10, 15, etc., enter group 2; and the remaining input channels enter their corresponding input channel groups according to the same pattern.

[0047] Because the number of non-zero weights in the initial input channels of the sorting sequence is relatively large, while the number of non-zero weights in the subsequent input channels gradually decreases, alternating forward and reverse allocation allows for the distribution of larger and relatively smaller task loads within the same round to different input channel groups. After grouping, the controller can calculate or directly use the non-zero weight set corresponding to each input channel group to distribute tasks. Each processing unit reads the input feature data according to the original channel identifier of the input channel in its respective input channel group.

[0048] When the number of input channels is not divisible by the number of input channel groups, the remaining input channels can continue to be allocated sequentially according to the current allocation direction until all input channels are in their respective input channel groups. This process does not require changing the previously completed groupings, nor does it require each input channel group to contain exactly the same number of input channels. Its focus is on making the total non-zero weights of each group relatively close through alternating allocation directions.

[0049] In one optional embodiment, after the division of multiple input channel groups is completed, each processing unit performs calculations according to the currently allocated input channel group. Each processing unit can sequentially obtain input channels from the corresponding input channel group in a preset number and process the non-zero weights corresponding to the obtained input channels; after completing the calculation of the current batch of input channels, it obtains the subsequent input channels until the input channels in the corresponding input channel group have been processed. Since the total amount of non-zero weights corresponding to multiple input channel groups is relatively close, the effective multiply-accumulate calculations required by each processing unit are also relatively close.

[0050] In one exemplary embodiment, there are multiple processing units, and the hardware acceleration device further includes a controller. The controller is used to determine the data flow mode corresponding to the multiple processing units based on the convolution parameters of the current convolutional layer and the preset correspondence between the convolution parameters and the data flow mode. The data flow mode includes at least one of a cross-input / output channel parallel mode, a cross-output feature map block parallel mode, and a cross-output channel parallel mode. The cross-input / output channel parallel mode is used to perform parallel computation simultaneously in the dimensions of the input channel and the output channel. The cross-output feature map block parallel mode is used to perform parallel computation between different spatial blocks of the output feature map. The cross-output channel parallel mode is used to perform parallel computation between different output channels.

[0051] In this embodiment, the convolution parameters read by the controller may include one or more of the following: input feature map height, input feature map width, number of input channels, number of output channels, kernel size, stride, and output feature map size. These parameters are system configuration information used by the hardware acceleration device when running the current convolutional layer. In addition to weight encoding information, they may also include the number of processing units, tile size, cache access parameters, and data transfer parameters. The system configuration information can be written during model deployment or loaded by the higher-level control module before executing different convolutional layers.

[0052] The correspondence between preset convolutional parameters and data flow patterns can be represented by parameter ranges, parameter combination conditions, or configuration tables. For example, several parameter regions can be defined based on the number of input channels, the number of output channels, and the size of the output feature map, with each parameter region recording a data flow pattern. When processing the current convolutional layer, the controller compares the current convolutional parameters with preset conditions to determine the corresponding data flow pattern for the current convolutional layer. Specific thresholds can be preset based on the number of processing units, on-chip storage capacity, and the neural network to be deployed, or they can be configured by the user during the deployment phase.

[0053] In the parallel mode across input and output channels, different processing units can process different groups of input channels and different groups of output channels separately, allowing convolutional tasks in both the input and output channel dimensions to be executed simultaneously by multiple processing units. This mode is suitable for convolutional layers where both the number of input and output channels can provide a large number of parallel tasks. For the same output position, the computation results corresponding to different input channels are partially summed and accumulated in subsequent stages.

[0054] In parallel mode across output feature maps, multiple processing units can process the same or corresponding input and output channels, but process different spatial patches in the output feature map. For example, when the output feature map size is large, it can be divided according to the size of the output patch that each processing unit can compute in one operation, assigning different complete patches to different processing units. Each processing unit can use the same weight data when computing these patches, only needing to obtain the input feature data corresponding to each spatial patch.

[0055] In parallel mode across output channels, multiple processing units process different output channels corresponding to the same input channel range. When the number of input channels in the current convolutional layer is small, and dividing the input channels cannot enable multiple processing units to all undertake computational tasks, different output channels can be assigned to different processing units, allowing multiple processing units to generate feature data for different output channels simultaneously.

[0056] For a single convolutional layer, different data flow modes can be combined according to preset configurations. For example, the output feature map can first be divided into multiple complete patches, and a cross-output feature map patch parallel mode can be used for the complete patches; for the remaining regions at the edges that cannot form complete patches, a cross-input / output channel parallel mode or a cross-output channel parallel mode can be used according to preset configurations. For example, when using a 56×56 output feature map, regions that can be divided according to the predetermined patch size can be processed first, and then another data flow mode can be switched for the remaining regions. The controller can complete the switching based on the preset region range and mode identifier.

[0057] The pre-defined correspondence can be saved layer by layer for each convolutional layer, or the parameters of multiple convolutional layers can be divided into several common parameter regions. For example, when deploying the network, a data flow pattern identifier can be written for each layer; alternatively, only several thresholds and comparison rules can be written, and the controller determines the pattern based on the parameter regions into which the current layer's input channel number, output channel number, and output feature map size fall. For convolutional layers using combined patterns, the first computation region, the second computation region, and their corresponding data flow patterns can also be recorded in the configuration, allowing the controller to read the next pattern identifier after completing the first computation region and continue processing the remaining regions. This allows the task division method among multiple processing units to be changed based on the parameters of different convolutional layers without altering the internal convolutional computation structure of the processing unit.

[0058] In one alternative embodiment, the hardware acceleration device further includes a shift network and multiple input buffers; in a parallel mode across input / output channels, the shift network is used to change the connection relationship between multiple processing units and multiple input buffers, so that each processing unit can obtain input feature data corresponding to different input channels.

[0059] In this embodiment, multiple input buffers can respectively buffer input feature data corresponding to different input channel groups. Since in the parallel mode across input and output channels, a processing unit may need to obtain data from other input channel groups after completing the calculation of the current input channel group, a shift network is set between the input buffers and multiple processing units to change the data transmission path according to the connection configuration given by the controller.

[0060] In some embodiments, the shift network can operate in a cyclic shift manner. Initially, the first processing unit is connected to the first input buffer, the second processing unit is connected to the second input buffer, and so on. After completing the current round of calculation, each connection is shifted one position in a fixed direction, allowing the first processing unit to access the data in the next input buffer. After several rounds of shifting, each processing unit can sequentially access the multiple input channel groups required for the current convolution calculation.

[0061] In other embodiments, the shift network can directly establish the correspondence between processing units and input buffers based on the connection selection information output by the controller, without requiring a single position to be moved each time. The controller can generate a connection order based on the current input channel group allocation result of the convolutional layer, so that each processing unit obtains the corresponding input buffer in that order. Regardless of whether cyclic shifting or configuration selection is used, the shift network changes the current data access relationship; the original channel identifiers and input feature data content of the input channels themselves remain unchanged.

[0062] Multiple input buffers can be stored in independent areas or in multiple partitions of the same on-chip memory. Each input buffer can prefetch the input feature data needed for the next round during the processing unit's computation, reducing the processing unit's waiting time for data loading. The shift network output is connected to the input data path of the processing unit. After the processing unit obtains the corresponding input feature data, it still controls the multiplexer array 110 to complete the selection of the input activation sub-window according to the current position index.

[0063] In one exemplary embodiment, the hardware acceleration device further includes a weight buffer, an increment buffer, and a partial sum buffer; the weight buffer is used to store the non-zero weights of the current convolutional layer and their corresponding encoding information, the encoding information including the position index and the input channel increment; the increment buffer is used to store the input channel increment and generate an update control signal for the input channel access address based on the input channel increment to indicate the jump position of the current non-zero weights in the input channel dimension; the partial sum buffer is used to cache the partial sum results output by multiple multiply-accumulate units 120 and to accumulate or write back the partial sum results after the current output tile calculation is completed.

[0064] like Figure 2As shown, in a specific hardware architecture, the hardware acceleration device can be connected to off-chip memory and includes a data loading module, input buffer, increment buffer, weight buffer, shift network, processing unit array, partial sum buffer, nonlinear computation unit, output buffer, data storage module, and controller. The data loading module reads the input feature data used by the current convolutional layer, the encoded non-zero weights, and system configuration information from off-chip memory, and writes the different data into the corresponding on-chip buffers. The system configuration information is used to configure the entire hardware acceleration device and includes not only data related to weight encoding but also convolutional layer parameters, number of processing units, tile size, buffer access method, and data flow mode configuration.

[0065] The weight buffer stores the non-zero weights to be calculated in the current convolutional layer and their encoding information. When the processing unit reads a current non-zero weight, it can simultaneously obtain the corresponding position index and control the multiplexing array 110 based on the position index. The input channel increment can be stored in the weight buffer as part of the encoding record, and simultaneously extracted and written to the increment buffer according to the execution order, for independent reading by the input channel address control path. In this way, the weight data path provides the processing unit with non-zero weights and position indices, while the increment data path provides the address control logic with input channel address change information.

[0066] The increment buffer can store multiple input channel increments in the order of execution of non-zero weights. After the current non-zero weight is processed, the address control logic reads the next input channel increment, updates the current input channel access address based on the increment, and generates a corresponding update control signal. The updated input channel access address is used to determine the input feature data that needs to be retrieved from the corresponding location in the input buffer or off-chip memory. For input channels without any non-zero weights to be processed, the address can be directly incremented to the next input channel that needs to be calculated.

[0067] The partial sum buffer is connected to the outputs of multiple multiply-accumulate units 120. Each multiply-accumulate unit 120 performs a multiplication and addition of the current non-zero weight and the corresponding input activation value at an output position, and the calculation result is written into the partial sum at the corresponding output position. When subsequent non-zero weights arrive, the partial sum is accumulated again based on the existing partial sum. After completing all input channels or the predetermined calculation range corresponding to the current output patch, the partial sum result can continue to be accumulated with the results generated by other processing units, or it can be output as the convolution result of the current output patch to the subsequent calculation module.

[0068] In one optional embodiment, the on-chip cache can be divided into multiple groups according to the number of processing units, with each group corresponding to one processing unit. Each group of on-chip caches may include an input cache, a weight buffer, and a partial sum buffer. Different processing units access the input cache, weight buffer, and partial sum buffer within their respective groups to reduce access conflicts caused by multiple processing units accessing the same cache space simultaneously. The increment buffer can be set independently of the above-mentioned multiple groups of on-chip caches and is used to store the input channel increment; the output cache is used to receive the output feature data processed by the nonlinear calculation unit, and the data storage module writes the output feature data back to the off-chip memory.

[0069] exist Figure 2 In the implementation shown, the convolution result can also be fed into a nonlinear computation unit to perform activation function or pooling processing, then written to the output buffer, and finally written back to off-chip memory by the data storage module. Input buffers, weight buffers, partial sum buffers, or output buffers can be configured using a ping-pong buffering method. For example, while one set of buffers provides the current computation data to the processing unit, another set of buffers loads the next batch of data to be computed; after the current computation is completed, the two sets of buffers exchange their working states, allowing data loading and processing unit computation to partially overlap in time.

[0070] Figure 2 The controller in the system coordinates data loading, cache access, processing unit computation, shifting network connection changes, and result output based on the current convolutional layer configuration information. Different functional modules can connect to the controller through independent control signals or carry the control information required by different modules through a unified instruction field. For specific convolutional layers that do not require a particular data flow pattern, the corresponding module can maintain its current connection state, and the processing unit can perform computation according to the already determined data access path.

[0071] In a complete data processing cycle, the data loading module first obtains the input feature data, non-zero weights, and system configuration information required by the current convolutional layer. The input feature data is written to the input buffer, the non-zero weights and their position indices are written to the weight buffer, and the input channel increments are written to the increment buffer. After the controller determines the task and data access path of the current processing unit based on the system configuration information, the processing unit reads data from the corresponding buffer and performs convolution calculations. The partial sum buffer continuously receives and saves the calculation results. After completing the convolution accumulation of the current convolutional layer, the result can be processed by the nonlinear calculation unit and written to the output buffer, and then written back to off-chip memory by the data storage module. The output of the previous convolutional layer can also be used as the input of the next convolutional layer and loaded into the input buffer.

[0072] like Figure 4 As shown, in a second aspect, this application provides a hardware acceleration method applied to the above-mentioned processing unit, the hardware acceleration method comprising: S11. Cache the input feature data required to generate the output block to be calculated into register file 100.

[0073] This step is executed by the data loading path of the processing unit. After determining the current output patch to be computed, the processing unit determines the input region to be accessed based on the output position covered by the output patch and the current convolution kernel size, and reads the corresponding input feature data from the input buffer and writes it into the register file 100. The input feature data in the register file 100 serves as a candidate input for the subsequent multiplexing array 110 and can be reused by multiple non-zero weights during the processing of the current output patch.

[0074] In some embodiments, register file 100 loads the input region corresponding to the current tile once when processing a new output tile begins. In other embodiments, overlapping input data may be retained between adjacent output tiles, and only the additional input data of the new tile compared to the previous tile may be loaded. Regardless of the loading method used, after this step, the data range stored in register file 100 can provide the input activation value that the current output tile to be calculated needs to select in the subsequent non-zero weight calculation process.

[0075] S12. Obtain the current non-zero weight and the position index corresponding to the current non-zero weight. The position index is used to represent the position of the current non-zero weight in the convolution kernel.

[0076] This step reads the weights sequentially according to the encoded non-zero weights. Unstructured sparse processing makes some weights in the convolution kernel zero, so the hardware can treat the remaining non-zero weights as objects to be processed, with the current position index corresponding to the current non-zero weight.

[0077] The position index can include information that uniquely determines the spatial location of the current non-zero weight within the convolution kernel. For example, for a two-dimensional convolution kernel, the row and column positions of the current non-zero weight can be read directly; alternatively, the position number can be read, and the corresponding position can be obtained based on a preset mapping between the number and the convolution kernel position. The current non-zero weight obtained in this step is used for multiplication calculations in multiple subsequent multiply-accumulate units 120, and the position index is used to control the selection of input data in subsequent steps.

[0078] S13. Based on the position index, select the input activation sub-window corresponding to the current non-zero weight from the input feature data cached in the register file 100 through the multiplexing array 110. The input activation sub-window includes multiple input activation values ​​corresponding to multiple output positions in the output block to be calculated.

[0079] In this step, after receiving the position index, the multiplexer array 110 changes the corresponding selection path, so that multiple candidate input data in the register file 100 are mapped to multiple output terminals according to the spatial offset of the current non-zero weight.

[0080] Different output positions in the output patch correspond to different convolution sliding positions. For the same current non-zero weight, the input activation values ​​required for each output position are also located at different input positions. This step obtains an input activation sub-window that includes these input activation values, so that one input activation value in the sub-window corresponds to one output position. The output of this step is directly used as the input of multiple multiply-accumulate units 120 in the next step.

[0081] S14. Output multiple input activation values ​​to multiple multiply-accumulate units 120 respectively, and perform multiply-accumulate operations through multiple multiply-accumulate units 120 according to the current non-zero weight and the corresponding input activation value respectively.

[0082] In this step, each multiply-accumulate unit 120 receives the current non-zero weight and an input activation value corresponding to its own output position, performs multiplication, and adds the multiplication result to the partial sum at that output position. Multiple multiply-accumulate units 120 can work in parallel in the same computation cycle or in adjacent pipeline stages.

[0083] After completing the calculation in this step, if there are still other non-zero weights that need to be processed in the current output patch, return to S12 to read the next non-zero weight and its position index, and re-execute S13 and S14 based on the new position index. After the non-zero weights corresponding to the output patch are processed, the convolutional parts and / or convolutional results corresponding to multiple output positions are obtained, and then the subsequent output, activation, or pooling processing is performed. Through the cooperation of the above four steps, the position index of a non-zero weight is used to determine the input activation values ​​required for multiple output positions at once, reducing the repetitive input selection logic generated when multiple multiply-accumulate units 120 parse the weight positions separately.

[0084] During hardware execution, after the current non-zero weight enters the multiply-accumulate phase, the next non-zero weight and its position index can begin to be read; when the data-related conditions are met, the multiplexer array 110 can prepare the corresponding input activation sub-window for the next non-zero weight. The registers or handshake signals used between each step can be set according to the clock frequency and memory access latency.

[0085] In one exemplary embodiment, selecting an input activation sub-window corresponding to the current non-zero weight from the input feature data cached in the register file 100 according to the position index via the multiplexing array 110 includes: selecting consecutive rows of input feature data corresponding to the number of rows of the output patch to be calculated from the input feature data cached in the register file 100 according to the row position of the current non-zero weight in the convolution kernel; and selecting consecutive columns of input feature data corresponding to the number of columns of the output patch to be calculated from the selected consecutive rows of input feature data according to the column position of the current non-zero weight in the convolution kernel, so as to obtain the input activation sub-window.

[0086] This embodiment further explains the input selection process in the above embodiments. The position index includes the row position and column position of the current non-zero weight. The multiplexing array 110 first uses one position dimension to select continuous data corresponding to the output patch to be calculated from the candidate input data in the register file 100, and then uses the other position dimension to complete the second selection. When row position is selected first, the number of rows output in the first stage is consistent with the number of rows of the output patch to be calculated, and the column direction still retains multiple candidate positions generated by the sliding of the convolution kernel; in the second stage, candidate columns that do not correspond to the current non-zero weight are removed according to the column position.

[0087] Taking a convolution stride of 1, a 3×3 convolution kernel, and an 8×8 output patch to be computed as an example, register file 100 can store a 10×10 input region. When the current non-zero weight is located in the first, second, or third row of the convolution kernel, rows 1 to 8, 2 to 9, or 3 to 10 are selected from the 10 candidate data rows, respectively, to obtain an 8×10 data region. Subsequently, depending on whether the current non-zero weight is located in the first, second, or third column of the convolution kernel, columns 1 to 8, 2 to 9, or 3 to 10 are selected from the 10 candidate data columns, respectively, to finally obtain an 8×8 input activation sub-window.

[0088] exist Figure 3 In one implementation shown, the row selection stage can be implemented using multi-level two-to-one multiplexers. The row position selection signal in the position index controls different levels of multiplexers, allowing data from corresponding consecutive input positions to be passed to the output. The column selection stage can use the same or a similar multiplexing structure as the row selection stage, except that its input consists of data from different columns. For convolution kernels larger than 3×3, selection states that can represent more row and column offsets can be added, and the number of outputs can be set according to the size of the output tile to be calculated.

[0089] In other embodiments, the order of the two selection phases can be interchanged: first, multiple consecutive columns of input feature data are selected based on column position, and then multiple consecutive rows of input feature data are obtained based on row position. Regardless of the selection order, each input activation value in the final input activation sub-window corresponds one-to-one with each output position of the output patch to be calculated, and is multiplied by the same current non-zero weight in subsequent multiplication and addition steps.

[0090] For further details on hardware acceleration methods, please refer to the embodiments of the hardware acceleration device described above; these will not be repeated here.

[0091] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements may include not only the elements expressly listed, but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0092] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Those skilled in the art can make corresponding adjustments or substitutions to the above embodiments without departing from the technical solution of the present invention. Therefore, the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A hardware acceleration device, characterized in that, It includes at least one processing unit, which includes a register file, a multiplexer array, and multiple multiply-accumulate units; The register file is used to cache the input feature data required to generate the output tile to be calculated; The multiplexing array is connected between the register file and the plurality of multiply-accumulate units, and is used to select an input activation sub-window corresponding to the current non-zero weight from the input feature data cached in the register file according to the position index corresponding to the current non-zero weight; wherein, the position index is used to represent the position of the current non-zero weight in the convolution kernel; the input activation sub-window includes a plurality of input activation values ​​corresponding to a plurality of output positions in the output patch to be calculated, and the plurality of input activation values ​​are respectively output to the plurality of multiply-accumulate units; The plurality of multiply-accumulate units are used to perform multiply-accumulate operations based on the current non-zero weight and the corresponding input activation value, respectively.

2. The hardware acceleration device according to claim 1, characterized in that, The multiplexing array includes a row selection network and a column selection network; The row selection network is used to select multiple consecutive rows of input feature data from the input feature data cached in the register file, based on the row position of the current non-zero weight in the convolution kernel, corresponding to the row number of the output patch to be calculated; The column selection network is used to select multiple consecutive columns of input feature data corresponding to the number of columns of the output patch to be calculated from the input feature data output by the row selection network according to the column position of the current non-zero weight in the convolution kernel, so as to obtain the input activation sub-window.

3. The hardware acceleration device according to claim 1, characterized in that, The encoding information corresponding to the current non-zero weight includes the position index and the input channel increment; The input channel increment is used to characterize the address difference between the current non-zero weight and the previous non-zero weight in the input channel dimension, and is used to update the input channel access address based on the address difference.

4. The hardware acceleration device according to claim 1, characterized in that, The number of processing units is multiple, and the hardware acceleration device also includes a controller; The controller is used to count the number of non-zero weights corresponding to the multiple input channels of the current convolutional layer, sort the multiple input channels according to the number of non-zero weights, divide the sorted multiple input channels into multiple input channel groups, and assign the multiple input channel groups to the multiple processing units for calculation.

5. The hardware acceleration device according to claim 4, characterized in that, The step of dividing the sorted input channels into multiple input channel groups includes: Select a group of input channels from the sorted input channels, the same number of which is the same as the number of input channel groups. Along the first allocation direction, a group of input channels is sequentially allocated to the plurality of input channel groups; Along a second allocation direction opposite to the first allocation direction, the next set of input channels is sequentially allocated to the plurality of input channel groups; The subsequent input channels are alternately allocated according to the first allocation direction and the second allocation direction.

6. The hardware acceleration device according to claim 1, characterized in that, The number of processing units is multiple, and the hardware acceleration device also includes a controller; The controller is used to determine the data flow mode corresponding to the plurality of processing units based on the convolution parameters of the current convolutional layer and the preset correspondence between the convolution parameters and the data flow mode. The data stream mode includes at least one of the following: cross-input / output channel parallel mode, cross-output feature map parallel mode, and cross-output channel parallel mode. Among them, the cross-input / output channel parallel mode is used to perform parallel computation simultaneously in the dimensions of input and output channels; the cross-output feature map block parallel mode is used to perform parallel computation between different spatial blocks of the output feature map; and the cross-output channel parallel mode is used to perform parallel computation between different output channels.

7. The hardware acceleration device according to claim 6, characterized in that, It also includes a shift network and multiple input buffers; In the parallel mode across input / output channels, the shift network is used to change the connection relationship between the multiple processing units and the multiple input buffers, so that each processing unit can obtain input feature data corresponding to different input channels.

8. The hardware acceleration device according to any one of claims 1-7, characterized in that, The hardware acceleration device also includes a weight buffer, an increment buffer, and a partial sum buffer; The weight buffer is used to store the non-zero weights of the current convolutional layer and their corresponding encoding information, which includes the position index and the input channel increment. The increment buffer is used to store the input channel increment and generate an update control signal for the input channel access address based on the input channel increment, so as to indicate the jump position of the current non-zero weight in the input channel dimension. The portion and buffer are used to cache the portions and results of multiple multiply-accumulate units, and to accumulate or write back the portions and results after the calculation of the current output tile is completed.

9. A hardware acceleration method, characterized in that, Applied to the processing unit as described in any one of claims 1-8, the hardware acceleration method includes: The input feature data required to generate the output patch to be calculated is cached in the register file; Obtain the current non-zero weight and the position index corresponding to the current non-zero weight, wherein the position index is used to represent the position of the current non-zero weight in the convolution kernel; Based on the position index, an input activation sub-window corresponding to the current non-zero weight is selected from the input feature data cached in the register file through the multiplexing array. The input activation sub-window includes multiple input activation values ​​corresponding to multiple output positions in the output patch to be calculated. The multiple input activation values ​​are output to the multiple multiply-accumulate units respectively, and the multiple multiply-accumulate units perform multiply-accumulate operations according to the current non-zero weight and the corresponding input activation value respectively.

10. The hardware acceleration method according to claim 9, characterized in that, The step of selecting the input activation sub-window corresponding to the current non-zero weight from the input feature data cached in the register file according to the position index via the multiplexing array includes: Based on the row position of the current non-zero weight in the convolution kernel, select consecutive rows of input feature data from the input feature data cached in the register file that correspond to the row number of the output patch to be calculated; Based on the column position of the current non-zero weight in the convolution kernel, select consecutive columns of input feature data corresponding to the column number of the output patch to be calculated from the selected consecutive rows of input feature data to obtain the input activation sub-window.

Images